DATAIQ INSTALLATION

AND IMPLEMENTATION

PARTICIPANT GUIDE

PARTICIPANT GUIDE
 DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page i

Table of Contents

Requirements ............................................................................................................. 2
Installation Requirements for DataIQ v2.1.x ......................................................................... 3
Other Requirements............................................................................................................. 4
Target Volume ..................................................................................................................... 6
Sizing Guidelines ................................................................................................................. 7

DataIQ Installation...................................................................................................... 9
Installation Considerations ................................................................................................. 10
DataIQ Installer - Things to Know ...................................................................................... 11
Types of Installation - Virtual Host...................................................................................... 12
DataIQ Installation - Physical Host ..................................................................................... 15
Upgrade ............................................................................................................................. 16

DataIQ Plugins Installation ...................................................................................... 17

Plugin Installation ............................................................................................................... 18
Data Mover Plugin ............................................................................................................. 20
Data Mover Configuration .................................................................................................. 21
Data Mover Process Anatomy............................................................................................ 23
Data Mover Considerations ................................................................................................ 25

DataIQ Secure Remote Services ............................................................................. 26

Secure Remote Services Overview .................................................................................... 27
DataIQ SRS Integration ..................................................................................................... 28
Telemetry Data .................................................................................................................. 29
DataIQ SRS Integration Architecture.................................................................................. 30
DataIQ SRS Integration Installation ................................................................................... 31

DataIQ Installation and Implementation-SSP

Page ii © Copyright 2021 Dell Inc.

 Requirements

DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 1

Requirements

Requirements

DataIQ Installation and Implementation-SSP

Page 2 © Copyright 2021 Dell Inc.

 Requirements

Installation Requirements for DataIQ v2.1.x

Enterprises must meet software and security requirements in their environment,

which determines the platform on which DataIQ is installed. Install DataIQ either on
a standard system or a virtual appliance.

Component Requirement

DataIQ Host Operating Open Virtual Appliance (OVA): VMware ESXI

System versions 6.7.
Red Hat Enterprise Linux® or CentOS:
• 64-bit CentOS 7.6, 7.7, or 7.8
• 64-bit Red Hat Enterprise Linux (RHEL) Server,
v7.6, 7.7, or 7.8

DataIQ 2.1 lets users use Red Hat Package

Manager (RPM) to install DataIQ.

DataIQ Host CPU Performance (recommended): 12 CPUs

Minimum: 8 CPUs

DataIQ Host RAM 64 GB

DataIQ Host Capacity Minimum of 15% total drive free space + 40 GB free
space for installation

DataIQ SSD Minimum of 512 GB

At least 250 GB for every 1 billion files and folders to
scan
The table shows the installation requirements.

DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 3

Requirements

Other Requirements

Drive

A combination of both solid-state drives (SSD) and traditional hard disk drives
(HDD) is recommended. Set up drives as logical drives1 to support future
expansion.

Directory Description

Operating system Hosted on a traditional HDD or an SSD.

/opt/dataiq The /opt/dataiq directory stores the DataIQ

application, the storage monitoring database, the
keycloak database, and the claritynow configuration
database.

/var/lib/docker The /var/lib/docker directory stores the

Docker application and DataIQ pods.

1Logical volumes can be expanded later onto new drives when more storage is
needed.

DataIQ Installation and Implementation-SSP

Page 4 © Copyright 2021 Dell Inc.

 Requirements

/mnt/ssd In DataIQ v2.1, data management database no

longer requires a specific directory (/mnt/ssd).
This directory is still the default directory, but you
may now change its location during installation by
using the DataIQ --ssd_path <path> option. The
default backup path is
/mnt/ssd/data/claritynow/ssd/backup.
Unlike other directories, it is recommended to host
/mnt/ssd on an SSD.

Browser

You can use the following browsers with DataIQ:

• Google Chrome
• Microsoft Edge
• Safari
• Mozilla Firefox (standard edition)
• Internet Explorer 11+

Network

DataIQ does not require an Internet connection. However, some features such as
proactive support do require an Internet connection.

The DataIQ host and each worker node require a static IP address. All nodes can
access each other hence fast, and stable networking between nodes improves
performance and scalability.

Check the following before installing:

• Appropriate default gateway
• Appropriate DNS entries
• No IP address conflicts

DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 5

Requirements

Target Volume

DataIQ can manage storage across multiple types of unstructured storage

products, and can monitor storage on Dell EMC PowerScale and Isilon systems.

Data Management

DataIQ manages, scans, indexes, and reports on network-attached file systems

including NFS and SMB. DataIQ requires root access to clusters that it is directly
managing.

Examples of volumes you can connect:

• Dell EMC PowerScale and Isilon systems
• Amazon Web Services (AWS) S3
• Google Cloud Platform (GCP)
• Dell EMC ECS
• Dell EMC Unity

Storage Monitoring

DataIQ can manage and monitor storage on Dell EMC PowerScale and Isilon
systems using OneFS 8.0.0.0 and later, including Gen 4, Gen 5, Gen 6, and Gen
6.5 product models.

DataIQ can monitor storage on up to 70 Dell EMC PowerScale and Isilon clusters
up to 2,000 nodes.

DataIQ requires access to a user account with read permission to the clusters that
it is monitoring.

DataIQ Installation and Implementation-SSP

Page 6 © Copyright 2021 Dell Inc.

 Requirements

Sizing Guidelines

Sizing estimates depend on the complexity of the DataIQ solution and

requirements2. The general rule for sizing is to add additional disk resources to the
planned capacity.

Disk-space requirements depend on various factors that are used for data
management or storage monitoring.

Data management

• The data management portion of the software writes to an IndexDB which is

installed and hosted on a specific mount point3, such as /mnt/ssd.
• The underlying disk platform4 is recommended to be SSD.
• When sizing a DataIQ solution, consider the complexity of the customer storage
environment and the total number of files, folders, and objects per volume.
• Workloads can be distributed across the external data mover nodes in a scale-
out fashion.

2There is no one size that can fit all hardware resources that are planned for
DataIQ because every environment is different. The CPU, memory, and network
are shared resources for dataset management and storage monitoring.

3 This mount point (or subdirectory or folder) must exist before installation either in
the form of a simple folder or as a mounted partition.

4This can be in the form of a single large SSD VMDK for the entire operating
system and DataIQ application (such as the OVA by itself), or an SSD partition or
VMDK mounted to /mnt/ssd.

DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 7

Requirements

Storage monitoring

The general considerations for storage monitoring disk space are as follows:

• Load rate on the cluster is beyond the scope of DataIQ.

• DataIQ uses different data collection strategies for different dashboards.
• DataIQ provides a data retention feature and can only delete older monitoring
data.
• DataIQ provides a data backup feature for storage monitoring.
• Always add additional disk resources to the planned capacity to allow for
monitoring-data growth.
• Never undersize the estimated size of the disk requirements for monitoring.
• The disk-space requirement depends on various factors5.

Resource: For more details about sizing guidelines, refer to DataIQ

Best Practices Guide

5Various factors, such as active OneFS features, load rate, data backup strategy,
and data-retention settings. DataIQ regularly backs up data automatically, but the
backup can be disabled or defined as needed. The data-retention policy is disabled
by default, but it can be enabled and configured.

DataIQ Installation and Implementation-SSP

Page 8 © Copyright 2021 Dell Inc.

 DataIQ Installation

DataIQ Installation

DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 9

DataIQ Installation

Installation Considerations

Listed are areas to consider when installing DataIQ:

• DataIQ supports HTML5 and does not require Java.
• The DataIQ host should not have Docker, or Kubernetes preinstalled as the
DataIQ install program installs these components.
• DataIQ may need additional storage for systems using plugins, or if the DataIQ
scan environment has many files and folders. As a rule of thumb, at least 250
MB should be allotted for scanning every million files and folders.
• The administrator is given a temporary password for the DataIQ UI when the
installation completes.
• DataIQ works well on systems that have up to 20 billion files and folders.
Environments with more than 2000 nodes or 70 clusters may need multiple
DataIQ instances.

• It is recommended to use a dedicated server host for each instance of

DataIQ (server or data mover host) to avoid potential network port or
resource conflicts. There is no way to combine data from multiple DataIQ
instances.

DataIQ Installation and Implementation-SSP

Page 10 © Copyright 2021 Dell Inc.

 DataIQ Installation

DataIQ Installer - Things to Know

The DataIQ installer is the installation software bundle.

The details of the Installer are:

• Bundle size is about 4 GB.
• Bundle is self-extracting and executable.
• A single installation can take up to 10 minutes.
• Installer displays a successful message when install succeeds.
• Successful or failed installations are shown in log file.

DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 11

DataIQ Installation

Types of Installation - Virtual Host

The installation process has been simplified in DataIQ v2.1.

The tabs show the two ways to install DataIQ.

Install DataIQ onto a new VM created from an OVA

To install DataIQ deploy a new VM using an OVA installer6, use a virtualization

platform such as VMware vSphere.

Click each installation step to learn more.

1) Download installation and upgrade packages7
2) Prepare a VM to host DataIQ8
3) Create and mount logical volumes9

6The OVA installer prepares a VM with CentOS 7.6, DataIQ, Docker, and
Kubernetes installed, with many settings preconfigured.

7Download the installer and optionally, a DataIQ license file. A DataIQ license can
be added either during installation or after installation through the UI.

8You can host DataIQ and its features entirely on an SSD, or you can split it across
drives and use a less expensive HDD.

9 Prepare logical volumes using logical volume manager (LVM) for your storage
locations before installing DataIQ.

DataIQ Installation and Implementation-SSP

Page 12 © Copyright 2021 Dell Inc.

 DataIQ Installation

4) Set up networking10
5) Initialize DataIQ11

Install DataIQ on a server or existing VM

Install DataIQ onto a server or VM that already has CentOS or RHEL installed.

Click each installation step to learn more.

• Install and register the operating system12
• Download installation and upgrade packages
• Protect authenticity and integrity13
• Create and mount logical volumes
• Set up NetworkManager14
• Set up networking

10 Perform this step even if you are installing DataIQ on a host that is not connected
to the Internet.

11Run the DataIQ installer, receive initial passwords, and optionally set up Secure
Remote Services.

12You can install DataIQ on supported Red Hat Enterprise Linux or CentOS
systems. If you use a Red Hat Enterprise Linux, it must be registered.

13Files that have a .sign file extension can be verified using standard OpenSSL
tools.

14 Enable and start Linux NetworkManager tools.

DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 13

DataIQ Installation

• Extract files from the installer packages15

• Initialize DataIQ

Note: Use the uninstall.sh script to uninstall DataIQ from a host.

The uninstall script completely removes all the Docker, Kubernetes,
and nfs-utils packages. An uninstall can take up to 5 minutes to
complete.

15 Extract the DataIQ files from the installer and dependency packages.

DataIQ Installation and Implementation-SSP

Page 14 © Copyright 2021 Dell Inc.

 DataIQ Installation

DataIQ Installation - Physical Host

1:

Download the DataIQ installer package from Dell Support. If the DataIQ host has
Internet access, download the online package. If the DataIQ host does not have
Internet access, download the offline package to a client and then transfer it to the
DataIQ host.

2: Change the hostname from the default localhost to a unique name.

Example: hostnamect1 set-hostname dataiq-server where dataiq-server

is a unique name.

3: Run the installer ./dataiq_installer_<version>.sh

The installer prompts the user for entries such as the IP address and the IP
addresses of worker nodes. Upon completion, the user will have:

• The DataIQ admin URL or IP address

• A temporary password for the DataIQ UI administrator user

DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 15

DataIQ Installation

Upgrade

DataIQ supports upgrade to a higher version and getting new features and fixes.
But you should have a supported version of DataIQ installed to upgrade it to 2.x.

The host must meet all the requirements that are mentioned in the requirement
section of the course.

DataIQ version 2.x requires more CPUs, drive space, and memory (RAM) than
version 1.x. Hence ensure that sufficient resources are configured on the DataIQ
host.

The upgrade process completes within 20 minutes. During this time, users are
logged out and storage is not monitored.

After the upgrade completes, Data Management searches will not show any results
until after the first full volume scan is complete.

From/To DataIQ 1.0 DataIQ 2.0 DataIQ 2.1

DataIQ 1.0 - Supported Supported

DataIQ 2.0 - - Supported

DataIQ 2.1 - - -

Uninstall DataIQ and reinstall the original version to roll back to the original version
of DataIQ.

DataIQ Installation and Implementation-SSP

Page 16 © Copyright 2021 Dell Inc.

 DataIQ Plugins Installation

DataIQ Plugins Installation

DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 17

DataIQ Plugins Installation

Plugin Installation

The steps for installing a plug-in can vary slightly. Refer the DataIQ Admin Guide
for comprehensive details.

The high-level steps are:

1. Download the plug-in package from Dell Support.
• If available, get a copy of its proof of authenticity16.
2. Transfer the plugin tar.gz file to a folder on the DataIQ Linux host system.
You can use SCP or a shared mount point to transfer.
3. From the DataIQ host on the /root directory, run:
# cp /<package location>/<plugin package>-
installer.tar.gz /root
# cd /root
# tar -zxvf <plug-in package>-installer.tar.gz
# cd /root/<plug-in package>/
# python <plug-in package>.py install
4. Optional: Check the status by running kubectl get pods -ndataiq
5. Enable the plugin on the DataIQ WebUI > Settings > Data management
configuration page:

16Files that have a .sign file extension can be verified using standard OpenSSL
tools. The RPM uses a GPG signature for signing. The OVA can be verified by
using the Open Virtualization Format Tool (ovftool) utility that VMware provides.

DataIQ Installation and Implementation-SSP

Page 18 © Copyright 2021 Dell Inc.

 DataIQ Plugins Installation

Enable, disable, and manage plugin

Note: If you have previously installed a plugin, check to see if it still

exists on the system. If the plugin does exist, either remove or
upgrade17 the plugin.

17 Use the plugin installation package to update your plugins.

DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 19

DataIQ Plugins Installation

Data Mover Plugin

DataIQ Data Mover is a paid plugin that enables the movement of files and folders
more efficiently between file systems. Use Data Mover to transfer files and folders
between OneFS systems and external object storage such as Amazon Simple
Storage Service (S3), Google Cloud Platform (GCP), and ECS.

DataIQ logs file transfers to assist with audits and supports transfer policies to
allow or restrict the use of Data Mover based on user group and by source and
destination visibility. DataIQ is kept updated by reindexing files after transfers.

Data Mover enables you to choose a single file, or an entire folder, for a Copy-
Move operation from the folder tree UI.

The Data Mover has three components. Click each component for an overview:
• UI plugin18
• Service plugin19
• Workers plugin20

18 The user interface that is installed on the DataIQ host.

19The service plug-in is the job manager that is installed on the DataIQ host. This
service accepts requests from the Data Mover UI Plugin and services the work
assignment requests to/from Data Mover Workers.

20 The workers plug-in is the transfer agent service, responsible for transferring
files. Install the Data Mover Worker plug-in on dedicated transfer hosts or the
DataIQ host. Administrators can deploy multiple Data Mover Worker hosts. If on a
dedicated host, it requires RHEL/CentOS 7.6 or 7.7.

DataIQ Installation and Implementation-SSP

Page 20 © Copyright 2021 Dell Inc.

 DataIQ Plugins Installation

Data Mover Configuration

Configure Data Mover, including adding required volume information. If you use
Data Mover External Workers, modify the configuration on each External Worker to
match.

You can edit the Data Mover configuration file from the DataIQ UI. If there are no
active and paused transfer jobs, DataIQ applies changes and resets the Data
Mover service. The reset takes about one minute to complete.

By default, the Data Mover will refuse to transfer certain file paths and file types.
Data Movers for 2.1 have added support for these files21 in the plugin configuration.

21 1) support_extended_characters ––#1 & #2

2) convert_system_files_to_stubs ––#3

DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 21

DataIQ Plugins Installation

The graphic shows opening the Data Mover configuration file from the WebUI > Settings > Data
management configuration page.

DataIQ Installation and Implementation-SSP

Page 22 © Copyright 2021 Dell Inc.

 DataIQ Plugins Installation

Data Mover Process Anatomy

The graphic shows an overview of the end-to-end process for moving data from a
PowerScale cluster to a target. The target can be an S3 object target, ECS, Isilon,
or a supported third-party platform.

The image shows a Data Mover Worker on a separate node. Worker nodes do not
require a database disk or an SSD.

1. The DataIQ server scans and indexes a mounted volume. The Data Mover UI
plugin and the Data Mover service plugin is installed on the DataIQ host.

2. The network share is also mounted to the DataIQ Data Mover worker.

3. A file or folder is manually selected using the DataIQ WebUI.

4. The Data Mover service receives requests from the Data Mover UI and sends
instructions to the Data Mover worker.

5. After validating the file or folder path, the Data Mover heavy-worker begins the
copy job to the target. Only the data, not ACL and DACL information is sent. No file
stubs are used. The full file is either copied to target or copied with source deleted.
Light workers perform pre-transfer job validation and path preparation. Heavy
workers perform pre-allocations and data transfer. Administrators can configure the
number of workers in the
/usr/local/data_mover_workers/etc/workers.cfg file.

Allocating too many heavy workers may negatively impact the stability of the host.

DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 23

DataIQ Plugins Installation

6. If the target is S3, the Data Mover uses embedded credentials and the Data
Mover worker logs in to the cloud subscription service. For ECS, the service opens
an SSL tunnel to the recommended load balancer. Data copy-move operation is
complete.

DataIQ Installation and Implementation-SSP

Page 24 © Copyright 2021 Dell Inc.

 DataIQ Plugins Installation

Data Mover Considerations

Consider the following when planning to move data from a source to a target
volume.

• File to file data movement with a scale-out worker node architecture.

• Moving data from file to S3 with a scale-out worker node architecture preserves
data independence by leaving a standard S3 object in the destination.
Administrators can retrieve and consume objects independent of any DataIQ
component.
• Data Mover does not support moving data directly between two external object
storage (S3) buckets.
• Cannot move files based on policies. For example, data not accessed in the last
30 days.
• Does not copy Windows file permissions. Data Mover uses NFS protocol when
moving data.
• It is not tested with petabyte-scale data movement.
• Does not support automatic archiving, and does not have incremental archiving
capability.

DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 25

DataIQ Secure Remote Services

DataIQ Secure Remote Services

DataIQ Installation and Implementation-SSP

Page 26 © Copyright 2021 Dell Inc.

 DataIQ Secure Remote Services

Secure Remote Services Overview

The Secure Remote Services (SRS) application architecture is a secure,

asynchronous messaging system that is designed to support the functions of
secure encrypted file transfer, monitoring of device status, and remote execution of
diagnostic activities.

SRS delivers a secure, IP-based, distributed remote service support solution that
provides command, control, and visibility of remote services access.

Secure Remote Services Architecture.

DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 27

DataIQ Secure Remote Services

DataIQ SRS Integration

Dell EMC software products must be able to debug and resolve any issues with
minimal input from customers. DataIQ can communicate with Dell Support using
SRS. DataIQ can send logs, configurations, and managed device status to Dell
Support, either periodically for proactive support, or on-demand as needed.

SRS integration features are:

• SRS proactively and periodically sends telemetry data to Dell EMC after getting
confirmation from administrators.
• DataIQ launches a process that hides all sensitive information before sending
the telemetry data.
• Users can disable SRS at any time.
• SRS does not upload anything on disconnected sites (dark or on-premise sites)
although telemetry data collection continues.

With DataIQ v2, you can create an account for Dell Support to use a secure tunnel
into your system to investigate issues and support requests.

For more information about SRS, refer to your Secure Serviceability

Customer Letter.

DataIQ Installation and Implementation-SSP

Page 28 © Copyright 2021 Dell Inc.

 DataIQ Secure Remote Services

Telemetry Data

The table shows the DataIQ handling of telemetry data:

Proactively push debug information to engineering and support when necessary to achieve an adequate level of
Upload Telemetry application quality monitoring and instrumentation.
Data
Customer confirmation required (Yes/No, Default: No).

Gather Telemetry Telemetry data is always gathered without regard to the internet connection status.

Data Customer confirmation required (Yes/No, Default: No).

Hide Sensitive
Automatically hides sensitive information such as username, password, directory names, and so on.
Information

DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 29

DataIQ Secure Remote Services

DataIQ SRS Integration Architecture

DataIQ services communicate with SRS in customer network. SRS communicates

with Dell EMC backend.

Web Services

Access Servers
Certification Authority
Dell EMC Support Analyst

Resource: For assistance with setup and maintenance with SRS,

contact your Dell EMC Global Services representatives.

DataIQ Installation and Implementation-SSP

Page 30 © Copyright 2021 Dell Inc.

 DataIQ Secure Remote Services

DataIQ SRS Integration Installation

Set up SRS, and then configure it to work with DataIQ. SRS and DataIQ must be
installed on separate hosts or VMs. DataIQ can be attached to the external SRS
gateways based on Docker Edition (DE) or Virtual Edition (VE). SRS in DE can be
installed on DataIQ as embedded.

Prerequisite

Listed are the prerequisites for SRS installation:

• Specific assigned ports must be available.
• Docker supported Linux distribution (x64 bit).
• Docker Engine (Docker runtime).
• Disk space availability must be at least 64 GB.
• Valid IPv4 address.

Procedure

Listed is the procedure to perform SRS installation in DataIQ:

1. During the DataIQ installation, SRS installation is prompted with two options,
either to install SRS or not (default: No).
2. If user chooses to install SRS, then the installer requests password to be set for
the root account of SRS.
3. At the end of a successful SRS installation, the installer displays a message
stating that the SRS services are up and running and provides the URL to
continue to provision SRS in a web browser.

Configuration

The administrator must configure external network and/or firewalls to allow traffic
over specific ports and ensure that the default gateway is properly configured.

TCP Port or Protocol Description

Passive FTP 21 Sends Connect home messages

DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 31

DataIQ Secure Remote Services

SMTP 25 Processes Connect home files, sends email

notifications

HTTPS 443 Service notification, setup, and so on

HTTPS 443 AND 8443 Remote support22

HTTP 9443 Manage File Transfer (MFT)23

Note: SRS upgrade and uninstall are subprocesses of DataIQ

upgrade and uninstall processes. It is NOT recommended to upgrade
or uninstall SRS separately.

22The Dell EMC remote services strategy delivers immediate, secure response to
product event reports such as error alerts, which can greatly increase the
availability of your information infrastructure.

23 MFT is a bidirectional file transfer mechanism that is provided as part of SRSv3.

You can use MFT to send or receive large files, such as log files, microcode,
firmware, scripts, or large installation files between the managed device and Dell
EMC. A distribution "locker" is used for bidirectional file staging.

DataIQ Installation and Implementation-SSP

Page 32 © Copyright 2021 Dell Inc.

 DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 33