Lab: Qlik Replicate with

Google BigQuery
TABLE OF CONTENTS
Introduction ............................................................................................................................................... 3
Prerequisites .............................................................................................................................................. 3
Accessing Replicate Environment ............................................................................................................. 4
Oracle Source Configuration ..................................................................................................................... 5
Google BigQuery Target Configuration ..................................................................................................... 9
Configure Replication Task ...................................................................................................................... 11
Run Task ................................................................................................................................................... 18
Summary .................................................................................................................................................. 22
Introduction
The objective of this lab is to provide the hands-on knowledge to participants on the Qlik Replicate
capability of migrating the data from Oracle database to Google Cloud BigQuery messaging application at
real time. By the end of this lab participants should be able to perform the data migration from database to
Google Cloud BigQuery. The areas covered include:

o Configuring Source Endpoint

o Configuring Target Endpoint

o Creating Replicate Tasks

Prerequisites
• Network connectivity should be good while executing the lab.

Google BigQuery | Introduction 3

Accessing Replicate Environment
Steps
1. Ensure that the URL for the Replicate Server is available, and access granted.
- This will be provided by your Systems Administrator.

2. Open browser** and enter the URL of the Replication Server.

The format is: https://replicate.attunitydemo.com:3552/attunityreplicate/7.0.0.267/#!/tasks

- This will prompt you to log into Replicate.

3. Enter your Username and Password.

Credentials would be provided by your System Administrator.
- This will take you to the Replicate Console.

Google BigQuery | AccessingGoogle BigQuery

Replicate in Qlik Replicate Environment
Environment 4
4
Oracle Source Configuration
The first thing we need to do is create a source endpoint. We do this by clicking the Manage Endpoint
Connections button at the top of the screen.

4. Select Manage Endpoint Connections.

The following window will appear:

Google
Google BigQuery | Oracle Source BigQuery in Qlik Replicate Environment
Configuration 5
5
 5. Select + New Endpoint Connection.

6. Enter a meaningful Endpoint Name and Description for the Endpoint Connector.

7. With Source button selected, select dropdown arrow to select Oracle.

You will notice as we proceed that the content of the configuration window is context sensitive.

Google
Google BigQuery | Oracle Source BigQuery in Qlik Replicate Environment
Configuration 6
6
 8. Fill in both Source, Target Servers credentials as provided by your Systems Administrator.

• Server:
• Port:
• User:
• Password:
• Security/SSL Mode:

9. Select Test Connection.

This tells whether your configuration is correct.

Google
Google BigQuery | Oracle Source BigQuery in Qlik Replicate Environment
Configuration 7
7
 Look for the “Test Connection succeeded” message. Any other message means something may be
incorrect with your Server/Database definitions, or the Server/Database is unavailable.

This is how the configuration will appear.

10. Select Save.

The configuration of your Oracle source endpoint is complete.

11. Click Close.

Google
Google BigQuery | Oracle Source BigQuery in Qlik Replicate Environment
Configuration 8
8
Google BigQuery Target Configuration
Next, we need to configure our Google BigQuery target endpoint. The process is much the same as you saw with other
endpoints, and once again you will note that the configuration process is context sensitive as we move along.
As before, the first step in the configuration process is to tell Replicate that we want to create a new endpoint.
1. In the Replicate Console, Select Manage Endpoint Connections.

2. Select + New Endpoint Connection.

3. Enter a meaningful Endpoint Name and Description for the Endpoint Connector.

4. Select Target button selected in Role.

5. Enter Systems credentials as listed below:

Note: The entire contents of the .json file listed above must be copied into the Service Account Key.

6. Select Test Connection.

GoogleTarget
Google BigQuery | Google BigQuery BigQuery in Qlik Replicate Environment
Configuration 9
9
 Again, look for the “Test Connection succeeded” message. Any other message means something may be
incorrect with your Server/Database definitions, or the Server/Database is unavailable.

Your screen should look like the following, indicating that your connection succeeded:

7. Select Save.

8. Select Close.

GoogleTarget
Google BigQuery | Google BigQuery BigQuery in Qlik Replicate Environment
Configuration 10
10
Configure Replication Task
Now that we have configured our Oracle source and Google BigQuery target endpoints, we need to tie them together
in what we call a Replicate task. In short, a task defines the following:

• A source endpoint

• A target endpoint

• The list of tables that we want to capture.

• Any transformations we want to make on the data.

We start by doing the following:

1. In the Replication Console, Select +New Task to create a New Task.

The following window like this will pop up:

2. Enter a meaningful Task Name.

Give this task a meaningful name like Oracle to BigQuery. For this task we will take the defaults:
Name: Oracle to BigQuery.

3. Select Unidirectional radio button.

Google BigQuery | ConfigureGoogle BigQuery

Replication Task in Qlik Replicate Environment 11
11
 - This indicates data flows from Source to Target – with no writebacks to source system.

4. Ensure that Full Load and Apply Changes are shaded.

You should have the following filled in:

• Unidirectional
• Full Load: enabled (Blue highlight is enabled; Select to enable / disable)
• Apply Changes: enabled (Blue highlight is enabled; Select to enable / disable)
• Store Changes: disabled (Blue highlight is enabled; Select to enable / disable)
5. Select OK.
- This closes the New Task dialog box.
Once completed, the following window will appear:

Qlik Replicate is all about ease of use. The interface is point-and-click, drag-and-drop. To configure our task,
we need to select a source endpoint (Oracle) and a target endpoint (Google Cloud BigQuery). You can either
drag the Oracle Source endpoint from the box on the left of the screen and drop it into the circle that
says Drop source endpoint here, or you can click on the arrow that appears just to the right of the endpoint
when you highlight it.

Google BigQuery | ConfigureGoogle BigQuery

Replication Task in Qlik Replicate Environment 12
12
 1. On the left of the Replicate Console panel, Select Source.

2. Locate the Source Endpoint created above or one which meets your Source definitions.

3. Drag and drop to Source Endpoint on the right, as indicated in diagram.

4. On the left of the Replicate Console panel, Select Target.

5. Locate the Target Endpoint created above or one which meets your Target definitions.

6. Drag and drop to Target Endpoint on the right, as indicated in diagram.

Google BigQuery | ConfigureGoogle BigQuery

Replication Task in Qlik Replicate Environment 13
13
 7. Our next step is to select the tables we want to replicate from Oracle into Google BigQuery. Click on
the Table Selection… button in the top center of your browser.

8. Select HR Schema.

9. Enter % in Table.

Google BigQuery | ConfigureGoogle BigQuery

Replication Task in Qlik Replicate Environment 14
14
 10. Press the Search button.
This will retrieve a list of all the tables in the HR schema.
Note: entering % is not strictly required. By default, Qlik Replicate will search for all tables (%) if you
do not limit the search.

11. Select a few tables:

• HR. Departments
• HR. Employee

Select each table from the Results list and press the > button to move them into the Selected
Tables list. Note that multi-select is enabled. You can select the tables all at once or move them
individually.

At this point we could define transformations on the selected tables if we wanted, but we will keep it simple
for this part of the testing and move the data as is instead.

Google BigQuery | ConfigureGoogle BigQuery

Replication Task in Qlik Replicate Environment 15
15
 12. Select OK.

That completes configuration of the task. We are now ready to save our task and run it.

13. Click on the Task Setting.

Select Metadata.

14. Enter Table Target Schema (Because these are file spaces, use your username or as provided by your
Systems Admin).

15. Press Save.

Google BigQuery | ConfigureGoogle BigQuery

Replication Task in Qlik Replicate Environment 16
16
 16. Select Run.

Google BigQuery | ConfigureGoogle BigQuery

Replication Task in Qlik Replicate Environment 17
17
Run Task

When you press Run, Replicate will automatically switch from Designer mode to Monitor mode. You will be able to
watch the status of the full load as it occurs, and then switch to monitoring change data capture as well.

1. Select down arrow beside Run.

2. Select Start Processing.

- If this is not the first time this Task is being used to extract data, Reload Target must be used – Reload
Target will not be an option.

3. Select Yes to reload Data Target.

Google BigQuery | Run Task Google BigQuery in Qlik Replicate Environment 18

18
 After Full Load is complete, select on the Completed bar to display the tables.

4. Select on the Change Processing tab to see it in action.

Note: Changes to the tables occur somewhat randomly in the background. You will have to wait for
the scheduling [polling] time (a few minutes) before you will see any changes appearing in the tables
that we selected**.

* Please note, this is only available for Delta capable Extractors

Google BigQuery | Run Task Google BigQuery in Qlik Replicate Environment 19

19
 5. Just open DBeaver and test Google BigQuery connection to validate the records.

6. Click on the Test connection to test the connectivity.

Google BigQuery | Run Task Google BigQuery in Qlik Replicate Environment 20

20
 7. Connect to Google BigQuery and check if Provided Target Schema name <it should be your
username> is present in the BigQuery and both the tables Employees and Departments should be
available in that schema.

8. When completed, press Stop in the top left corner of the Replicate console to end the task.
9. Confirm, by clicking Yes.
10. Close the Oracle to Google BigQuery tab or click on the TASKS tab to return to Replicate Console.

Google BigQuery | Run Task Google BigQuery in Qlik Replicate Environment 21

21
Summary
In this lab we have:

• Defined access and authentication into an Oracle source and a Google Cloud BigQuery target.

• Defined the source tables from which you want to create Google Cloud BigQuery messages.

• Configured the Oracle to Google Cloud BigQuery task.

• Captured initial data from the source without while maintaining business continuity (DML activity was going
on in the background to simulate users working on the source database).

• Captured all new transactions which were happening while the initial load was running.

• Turned all net new data into Google Cloud BigQuery messages.

• Observed change data being recorded as it is sent to and applied at the target.

Google BigQuery | SummaryGoogle BigQuery in Qlik Replicate Environment 22

22
About Qlik
Qlik’s vision is a data-literate world, where everyone can use data and analytics to improve decision-making and solve their most challenging
problems. Qlik provides an end-to-end, real-time data integration and analytics cloud platform to close the gaps between data, insights and action.
By transforming data into active intelligence, businesses can drive better decisions, improve revenue and profitability, and optimize customer
relationships. Qlik does business in more than 100 countries and serves over 50,000 customers around the world.

qlik.com
© 2021 QlikTech International AB. All rights reserved. All company and/or product names may be trade names, trademarks and/or registered trademarks of the respective owners with which they are
associated. CODE NEEDED FOR PUBLISHING

Google BigQuery | SummaryGoogle BigQuery in Qlik Replicate Environment 23

23