Lab: Qlik Replicate with Azure

Databricks as a Target
TABLE OF CONTENTS
Overview ..................................................................................................................................................... 3
Introduction ............................................................................................................................................... 3
Managing Endpoint Connections .............................................................................................................. 4
Creating Source Endpoint Connections 4
Creating Target Endpoint Connections 6

Creating Replicate Task ............................................................................................................................. 7

Assign Endpoint Connections .................................................................................................................... 9
Managing Global Transformation............................................................................................................ 10
Update Task Settings ............................................................................................................................... 11
Managing Data Extraction........................................................................................................................ 12
View Data in Databricks ........................................................................................................................... 15
 Overview
This Document covers the configuration of Endpoints for extracting data out of SAP Enterprise
Resource Planning (ERP) System via Qlik Replicate and storing it in Azure Databricks. The areas
covered include:

o Configuring Source Endpoint

o Configuring Target Endpoint

o Creating Replicate Tasks

Introduction
A Replicate Task is required to manage Source data change-data/capture from a Source System, into
the target database or file storage systems. A Replicate task can only manage one source and one
target system, but within a project, multiple Replicate Tasks can be used – one for each Source
System.

In this Lab, you will create a Qlik Replicate task with corresponding Source and Target Endpoints.
This task will be used to extract data from SAP ERP System, by way of SAP Extractors, and store in
Databricks in an Azure environment.

Lab: Qlik Replicate with Azure Databricks as a Target | Overview 3

 Managing Endpoint Connections
The Replicate task will use the connections you are about to create to perform a full loading of data, followed
by incremental log-based data replication from your source system to your target.

Creating Source Endpoint Connections

Qlik Replicate requires information to connect to the source and target endpoints that you want to use in a
task. In addition to providing a complete list of the source and target endpoints supported by Qlik Replicate,
there is also the ability to create new Endpoints.

Qlik Replicate delivers a number of Endpoints types for various Source and Target Systems.

Steps
1. In the Replication Console, Select Manage Endpoint Connections.

2. To create Endpoints, Select + New Endpoint Connections.

3. Enter a meaningful Endpoint Name and Description for the Endpoint Connector.

4. With Source button selected, select dropdown arrow to select appropriate Source Type.

Lab: Qlik Replicate withWorking with Databricks

Azure Databricks as a Target in Qlik Replicate
| Managing Environment
Endpoint Connections 4
4
 5. Enter Source Server information as provided by your System/Database Administrator.
- Note: In this example, SAP Extractor is the Source of the data.

Name:

Description:

Role: Source

Type: SAP Extractor

Connection Mode: Custom

Application Server

Server Name: (IP Address)

Instance Identifier: 0

Client identifier: (SAP Client

number)

Username:

Password:

Number Format:

6. Select Test Connection.

Look for the “Test Connection succeeded” message. Any other message means something may be
incorrect with your Server/Database definitions, or the Server/Database is unavailable.

7. Select Save.

Lab: Qlik Replicate withWorking with Databricks

Azure Databricks as a Target in Qlik Replicate
| Managing Environment
Endpoint Connections 5
5
 Creating Target Endpoint Connections

Steps
1. Select New Endpoint Connection.

2. Enter a meaningful Endpoint Name and Description for the Endpoint Connector.

3. Select Target button selected in Role.

4. Select dropdown arrow in type and select Microsoft Azure Databricks.

5. Enter the following Target Server credentials as provided by your System/Database Administrator.

Azure Storage
Storage Type:

Storage Account:

Azure Active Directory ID:

Azure Active Directory Application ID:

Azure Active Directory Application Key:

File System:

Target Folder:

Databricks ODBC Access

Host:

Storage Account:

Port:

Token:

HTTP Path:

Database:

Mount Path:

Lab: Qlik Replicate withWorking with Databricks

Azure Databricks as a Target in Qlik Replicate
| Managing Environment
Endpoint Connections 6
6
 6. Select Test Connection.

Again, look for the “Test Connection succeeded” message. Any other message means something
may be incorrect with your Server/Database definitions, or the Server/Database is unavailable.

7. Select Save.

8. Select Close.

Creating Replicate Task

Steps
1. Ensure that the URL for the Replicate Server is available, and access granted.
- This will be provided by your Systems Administrator.

2. Open browser** and enter the URL of the Replication Server.

The format for Replicate on Windows is https://<computer name>/attunityreplicate.

- This will prompt you to log into Replicate.

3. Enter your Username and Password.

- This will take you to the Replicate Console.

4. Select +New Task to create a New Task.

Lab: Qlik Replicate withWorking with Databricks

Azure Databricks as a Target in Qlik Replicate
| Creating ReplicateEnvironment
Task 7
7
 The following window will appear:

5. Enter a meaningful Task Name.

6. Select Unidirectional radio button.

- This indicates data flows from Source to Target – with no writebacks to source system.

7. Ensure that Full Load and Apply Changes are shaded.

The shade indicates the Task Options which are selected.

8. Select OK.
- This closes the New Task dialog box.
The Replicate Task is now ready for the creation of Endpoints.

Lab: Qlik Replicate withWorking with Databricks

Azure Databricks as a Target in Qlik Replicate
| Creating ReplicateEnvironment
Task 8
8
 Assign Endpoint Connections
At this point the Source and Target Endpoints created above will be assigned, in order for Replicate to extract
and load the data as defined.

Steps
1. On the left of the Replicate Console panel, Select Source.

2. Locate the Source Endpoint created above or one which meets your Source definitions – SAP
Source.

3. Drag and drop to Source Endpoint on the right, as indicated in diagram.

4. On the left of the Replicate Console panel, Select Target.

5. Locate the Target Endpoint created above or one which meets your Target definitions – Azure
Databricks.

6. Drag and drop to Target Endpoint on the right, as indicated in diagram.

7. Select Save.

Lab: Qlik Replicate withWorking with Databricks

Azure Databricks as a Target in Qlik Replicate
| Assign Environment
Endpoint Connections 9
9
 Managing Global Transformation
Steps
1. In Replicate Console, select Global Transformation.

2. Select New Global Transformation.

3. Enter meaningful Transformation description.
4. Select Rename Schema button.
5. Select Next.
6. Select Next.
7. Select Rename Schema button.
8. Enter the name of the Databricks Database.

9. Select Finish.
10. Select OK.

Lab: Qlik Replicate withWorking with Databricks

Azure Databricks as a Target in Qlik Replicate
| Managing Environment
Global Transformation 10
10
 Update Task Settings
Steps
1. In Replicate Console, Select Task Setting.

2. Enter Table Target Schema.

This is the schema in the target database where all extracted tables will be stored.

3. Select OK.

4. Select Save.

Lab: Qlik Replicate withWorking with Databricks

Azure Databricks as a Target in Qlik Replicate
| Update Environment
Task Settings 11
11
 Managing Data Extraction
Replicate is ready to extract the data into SQL Database.

Steps
1. In the Qlik Replicate Console, Select Table Selection.

2. Select Search.
A List of available files/tables will appear.

3. Select Files/tables to be transfer into the Qlik Replicate Console.

Lab: Qlik Replicate withWorking with Databricks

Azure Databricks as a Target in Qlik Replicate
| Managing Environment
Data Extraction 12
12
 4. Select > icon to transfer data to the Selected Tables column.

5. Select OK.

6. Select down arrow beside Run.

7. Select Reload Target.

- If this is the first time this Task is being used to extract data, Start Processing must be used –
Reload Target will not be an option.

Lab: Qlik Replicate withWorking with Databricks

Azure Databricks as a Target in Qlik Replicate
| Managing Environment
Data Extraction 13
13
 8. Select Yes to reload Data Target.

Replicate switches to Monitor mode to allow for monitoring of Data Loads.

Once data load is completed, log into the Target Microsoft Azure Databricks and validate data.

Lab: Qlik Replicate withWorking with Databricks

Azure Databricks as a Target in Qlik Replicate
| Managing Environment
Data Extraction 14
14
 View Data in Databricks
In this scenario, the data was extracted into Databricks. (If using QMI® ensure Databricks is enabled as a
Database before provisioning).

We will now go to view the data.

The tool I will use to browse the data is DBeaver.

1. Locate the Databricks Database used in the Target Endpoint.

2. Drilldown to locate the database Schema defined as part of the Replicate.

3. Identify table for which to view data.

4. Right-Mouse click on the table.

Lab: Qlik Replicate withWorking with Databricks

Azure Databricks as a Target in QlikData
| View Replicate Environment
in Databricks 15
15
 About Qlik
Qlik’s vision is a data-literate world, where everyone can use data and analytics to improve decision-making and solve their most challenging
problems. Qlik provides an end-to-end, real-time data integration and analytics cloud platform to close the gaps between data, insights and action.
By transforming data into active intelligence, businesses can drive better decisions, improve revenue and profitability, and optimize customer
relationships. Qlik does business in more than 100 countries and serves over 50,000 customers around the world.
qlik.com
© 2021 QlikTech International AB. All rights reserved. All company and/or product names may be trade names, trademarks and/or registered trademarks of the respective owners with which they are
associated. CODE NEEDED FOR PUBLISHING

Lab: Qlik Replicate withWorking with Databricks

Azure Databricks as a Target in QlikData
| View Replicate Environment
in Databricks 16
16