Real Time Data Strategy And

Architecture

Alan McSweeney
http://ie.linkedin.com/in/alanmcsweeney

Real Time Data Collection Strategy And Architecture

Approach

These notes are concerned with describing a generalised

approach to defining a strategy for collecting (near or
actual) real time, high volume data

This is data is generated by sensors that is transmitted to a

central location for processing, reporting, analysis and
ultimately action

Sensors can be regarded as logical or physical sources of

streams of measurement data

(Near) real time data can be termed Telemetry

March 8, 2016

Real Time Data Collection Strategy And Architecture

Approach

Approach adopted from TMForum Resource Domain

Frameworx eTOM (Enhanced Telecoms Operating Model)
Business Process Framework

Approach has been generalised for real-time data and

telemetry

March 8, 2016

Real Time Data Collection Strategy And Architecture

Collect data from range of data sources across the

organisations (internal and external) operating landscape

Approach can be applied to collection of measurement

data from multiple sources and of multiple types through
sensors:
Different entities interacting with the organisation and the
gathering of data on different actions and events

Approach can be applied to areas such as Telemetry, Big

Data, Smart Metering and Internet of Things
implementations and operations

March 8, 2016

Real Time Data Collection Strategy And Architecture

Real Time Data Strategy
Telemetry
Strategy

SmartX
Strategy

March 8, 2016

Big Data
Strategy

Digital
Strategy

Internet Of
Things
Strategy

Why Have A Real Time Data Collection Strategy And

Architecture?

Real time situational data gives rise to situational

awareness and understanding which in turn presents
opportunities for effective and rapid situational decisions
What is happening usage, performance
What can be improved
What are the optimisation and productivity opportunities

Real time situational data enables greater situational

visibility which means increased operational intelligence

March 8, 2016

Organisation Operating Landscape

March 8, 2016

Organisation Operating Landscape

The operating landscape of the organisation defines the

number and type of interactions outside the organisation

This operating landscape affects the decision on what and

how to measure

Not all interactions with all entities are measured

Need to be realistic about what can be collected and

processed

Need to understand the need for sensors to collect data

March 8, 2016

Data Sensors Combinations Of Options

Sensor Type
Logical

Direct
Sensor Ownership

Indirect

Physical

Web Site and App Activity

and Usage Data

Internet of Things Devices

Remote Real Time Unit
Smart Devices

Third-Party Web Site and

App Activity and Usage
Data

Third-Party Devices

Sensor Type Physical sensors are actual units such as RTUs

(Remote Telemetry Units) that measure and generate data
while Logical sensors are representations of data sources
Sensor Ownership Direct sensors are those installed and
maintained by the collecting organisation while Indirect
sensors are installed by a third-party
March 8, 2016

Measurement Data Sensors

March 8, 2016

10

Measurement Data Sensors

These can gather data for different measures using

multiple measurement techniques

These can be regarded as collectors of any data usage,

activity, performance, throughput

Each measurement type will have a unit or dimension

Logical representation of data collectors

Need to decide on what can be measured, what to

measure and how to measure it

March 8, 2016

11

Measurement Data Sensors Decide On What And

How To Measure

March 8, 2016

12

Real Time Data Architecture Complexity

Real Time Data Architectures
Tend To Focus On The Simplicity
Of A Possible Real Time Data
Collection Architecture

And Ignore The Complexity And

Difficulties Of Sensor Installation,
Operation, Maintenance, Errors,
Rework, Logistics, Service
Management, Data Volumes
March 8, 2016

13

Real Time Data Architecture Complexity

Structured approach is intended to ensure that complexity

is understood and can be appropriately addressed at an
early stage before problems become to embedded to be
solved

March 8, 2016

14

Real Time Data Architecture Complexity

Dont Let The Ignored But

Knowable And Addressable
Complexity Sink Your Real
Time Data
Programme/Initiative
March 8, 2016

15

Real Time Data Strategy And Architecture Issues

What business benefits will a real time data strategy yield?

How can the benefits be realised?
What is the business case for investment in a real time data
strategy?
What infrastructure, communications/connectivity, data and
application architectures are needed to support a real time data
strategy?
What integration is required with existing applications?
What skills, capabilities and changes does the organisation need to
adopt and exploit a real time data strategy?
How is the real time data strategy managed and serviced?
What solution and service providers and tools/platforms and
sourcing strategy should be selected?
What are the privacy and security issues and requirements?
March 8, 2016

16

Evolution Of Real Time Data Architecture

Sense and respond

Fly

Optimising

Processes, mechanisms and

metrics

Run

Managing

Making and implementing

effective investment
decisions

Communicating

Walk

Governing

Admitting
There must be a better way
March 8, 2016

Crawl

Establishing and
communicating business case

Leveraging
for Strategic
Outcomes

Real time data management, reporting and

analysis techniques are deployed for
strategic mission/business outcomes

Improving Real
Time Data
Processes

Process evaluation techniques focus on

improvement of performance and
management

Developing
Complete Real
Time Data
Portfolio

Comprehensive selection and control

processes with benefit and risk criteria
linked to strategy requirements

Building Real Time

Data Investment
Foundation

Implement investment controls and develop

key foundational capabilities

Create Awareness
for Real Time Data
Investment

Undefined/unarticulated/uncertain real time

data strategy. Real time data processes are
ad hoc, focussed on individual solution and
outcomes vary widely.
17

Developing A Real Time Data Strategy Generalised

High Level Steps
1
Real Time
Strategy And
Planning

2
Real Time
Capability
Delivery

Real Time
Provisioning

Real Time
Data
Collection
And
Distribution

March 8, 2016

3
Real Time
Development
And
Retirement

8
Real Time
Problem
Management

4
Real Time
Management
and
Operations
Support And
Readiness

9
Real Time
Performance
Management

5
Workforce
Management

10
Real Time
Data
Aggregation
and
Reporting

18

Developing A Real Time Data Strategy Generalised

High Level Steps

Comprehensive set of steps from definition of what is required

from real time data to commissioning of real time data
collection facilities to effective use of collected data
Not all steps are relevant to all real time data initiatives

For example, if installation, commissioning, operation and maintenance

of physical sensors is not applicable then related steps will not be
required

Represents an idealised organisation and process breakdown

across entire spectrum of real time data from strategy to
workforce management to device installation, data collection
and data usage and actioning
Provides a basis for developing a work breakdown and an
implementation plan
Represents a comprehensive structure that can be adapt to
meet its long-term real time data needs
March 8, 2016

19

Developing A Real Time Data Strategy Generalised

High Level Steps 1 2
Step

Scope

1. Real Time Strategy And

Planning

2. Real Time Capability

Delivery

March 8, 2016

Develop real time data strategy, policies and plans for the organisation governed by long-term
business, market, product and service needs directions
Perform research and analysis to determine real time targets and strategies to reach the defined
targets
Understand the real time data capabilities of the existing infrastructure
Build real time data model
Define approaches to real time data quality and real time data governance
Define and agree the infrastructure needs based on market, product and service strategies of the
organisation
Manage the capabilities of the suppliers and partners to develop and deliver new real time data
capabilities and detail the approach to the deployment of new and enhanced infrastructure
Define the real time data implementation standards sought, key real time data capabilities
required, real time data support levels and approaches required, real time data design elements
to be developed, and real time data cost parameters and targets.
Define the policies relating to technical real time data sensors and their implementation
Ensure that network, application and computing real time data facilities are deployed
Provide the physical real time data capabilities necessary for the ongoing operations and longterm well-being of the organisation and ensure the basis on which all real time data capabilities
and services will be constructed
Plan real time data resource supply logistics
Plan real time data sensor installation
Verify the real time data sensor installation
Handover real time data capabilities to operations
20

Developing A Real Time Data Strategy Generalised

High Level Steps 3 6
Step

Scope

3. Real Time
Development And
Retirement

4. Real Time

Management and
Operations Support And
Readiness
5. Workforce

Management

6. Real Time Provisioning

March 8, 2016

Develop new or enhance existing technologies and associated real time data types applying the
capability definition or requirements defined by the Real Time Strategy And Planning step
Decide on acquisition of real time data resources from third parties
Retire or remove technology and associated real time data resource types that are no longer
needed by the organisation
Manage the types of real time data resources and ensure that necessary application, computing
and network facilities are available and ready to implement and manage resource instances

Manage the direct and indirect personnel who perform work assignments or work orders relating
to real time data resources installation, commissioning and maintenance as well as managing the
actual activity being performed
Report and monitor activities
Establish, manage and allocate work assignments to direct and indirect personnel
Establish and manage priority and urgent assignment capabilities to allow for modification of
work assignments as required to meet urgent and high priority conditions
Allocate, install, configure, activate and test of real time data resources to meet the defined
requirements
Resolve real time resource capacity issues, availability issues or failure conditions
Configure and activate physical and/or logical real time resources
Update of real time resource register database to reflect that the specific real time resource has
been allocated, modified or recovered
21

Developing A Real Time Data Strategy Generalised

High Level Steps 7 8
Step

Scope

7. Real Time Data

Collection And
Distribution

8. Real Time Problem

Management

March 8, 2016

Collect and distribute management information and real time data between data sources and
service instances and other organisation functions and processes
Work with the real time data resource and service instances to collect usage, network and
technology events and other management information for distribution to other processes within
the organisation
Handle and process command, query and other management information for distribution to
resource and service instances
Process the data and management information through actions such as filtering, aggregation,
formatting, transformation and correlation of the information before presentation to other
processes, real time data instances or service instances
Perform usage reporting, fault and performance analysis, service quality management analysis,
resource performance analysis of resources and services
Manage real time data resource problems including security events
Detect, analyse, manage and report on resource alarm event notifications
Initiate and manage real time data resource problems reports
Perform real time data resource problem localization analysis and resolve problems
Reporting progress on resource trouble reports to other processes
Assign and track real time data resource problem testing and resolution activities
Managing real time data resource problem urgent conditions

22

Developing A Real Time Data Strategy Generalised

High Level Steps 9 10
Step

Scope

9. Real Time Performance

Management
10. Real Time Data
Aggregation and
Reporting

March 8, 2016

Manage, track, monitor, analyse and report on the performance of real time data resources
Identify real time data resource performance disruptions or a service performance disruptions
Manage real time data resource events by correlating and formatting them into a usable format
Report of real time data resource data

23

Sample Expansion Step 1 Real Time Strategy And

Planning Activities 1.1 1.7
Step

Scope

1.1 Gather And Analyse Real

Time Data Information
1.2 Manage Real Time Data
Research
1.3 Establish Real Time Data
Strategy And Architecture
1.4 Define Real Time Data
Support Strategies
1.5 Produce Real Time Data
Business Plans

Research and analyse customer, technology, competitor and marketing information to identify
new real time data requirements and industry real time data capabilities and availability.
Manage internally driven research investigations and activities which are used to provide detailed
technical assessment or investigation of new and emerging real time data capabilities.
Establish the real time data strategies based on market trends, future needs, technical
capabilities and addressing shortcomings in existing real time data support.
Define the principles, policies and performance standards for the operational organisation
providing real time data support.
Develop and deliver annual and multi-year real time data plans in support of services,
products and offers that include volume forecasts, negotiation for required levels of resources
and budgets.
Obtain real time data development and management as well as supply chain commitment
and executive approval for the plans.
Identify the impacts that new or modified real time data infrastructure will cause on the
installed infrastructure and workforce and establish the functions and benefits that new or
modified real time data will provide to users.
Identify the requirements for real time data capabilities to be sourced from partners or suppliers,
and any real time data capabilities to be delivered internally to the organisation.
Obtain organisation commitment to the resource strategy and business plans including all aspects
of identification of stakeholders and negotiation to gain stakeholder approval.

1.6 Develop Real Time Data

Partnership Requirements
1.7 Gain Enterprise
Commitment To Real Time
Data Plans
March 8, 2016

24

Sample Expansion Step 1 - Real Time Strategy And

Planning Activity 1.5 Tasks 1.5.1 1.5.5
Step

Scope

1.5.1 Develop And Deliver Develop and deliver annual/multi year real time data business plans focus on developing and
Annual/Multi Year Real
delivering annual and multi-year real time data in support of services, products and offers that
Time Data Business Plans include volume forecasts, negotiation for required levels of resources and budgets, gaining real time
data development and management as well as supply chain commitment and executive approval for
the plans.
1.5.2 Forecast High Level Forecast real time data demand and capture new opportunities processes ensures that budgets are
Real Time Data Demand assigned which allow the organisation to implement the real time data capabilities and capacity
And Capture New
necessary for the future needs of their customers and potential customers.
Opportunities
1.5.3 Assess Impact Of
Asses impact of real time data business plan processes assess the impacts that new or modified real
Real Time Data Business time data infrastructure will cause on the installed infrastructure and workforce, and establish the
Plans
functions and benefits that new or modified real time data will provide to users
1.5.4 Identify Timetables Identify timetables for new real time data capability introduction
For New Real Time Data
Capability Introduction
1.5.5 Identify Logistics For Identify logistics for new real time data capability introduction
New Real Time Data
Capability Introduction

March 8, 2016

25

Real Time Data Quality And Data Governance

Real-time data is inherently:

High Volume lots of sensors generating lots of data
Noisy lots of variation, statistical noise, inaccuracies, sensor drift, errors,
incorrect calibration
Changing data landscape subject to substantial change along the dimensions
of data sources, volumes, types
Inconsistent different sensor types measuring different values with different
units of measure and at different intervals
Heterogeneous very mixed data sources
Non-standardised multiple, emerging, overlapping standards and
approaches

Sophisticated approach to data quality and data governance must be

embedded in any real-time architecture
Traditional approach of data collection storage and analysis may
need to change to handle data volumes and quality data filtering,
quality, summarisation and transformation component
March 8, 2016

26

Real Time Data Principles

To manage and utilise real time information as a strategic asset

To implement processes, policies, infrastructure and solutions to govern, protect,

maintain and use real time information

To make relevant and correct real time information available in all business
processes and IT systems for the right people in the right context at the right time
with the appropriate security and with the right quality

To exploit real time information in business decisions, processes and relations

March 8, 2016

27

Real Time Data And Data Governance And Data

Quality

Data Quality - measure, assess, improve, and ensure the

fitness of data for use

Data Governance - authority and control over the

management of data assets

March 8, 2016

28

Real Time Data Governance

Core function of real time data management

Interacts with and influences each of the surrounding ten data
management functions
Data governance is the exercise of authority and control
(planning, monitoring, and enforcement) over the management
of data assets
Data governance function guides how all other data
management functions are performed
High-level, executive data stewardship
Data governance is not the same thing as IT governance
Data governance is focused exclusively on the management of
data assets
March 8, 2016

29

Real Time Data Governance Definition and Goals

Definition
The exercise of authority and control (planning, monitoring, and
enforcement) over the management of data assets

Goals
To define, approve, and communicate data strategies, policies,
standards, architecture, procedures, and metrics
To track and enforce regulatory compliance and conformance to
data policies, standards, architecture, and procedures
To sponsor, track, and oversee the delivery of data management
projects and services
To manage and resolve data related issues
To understand and promote the value of data assets
March 8, 2016

30

Real Time Data Governance Structure

Real Time Data
Governance Framework

Real Time Data

Architecture to Implement
Data Governance
Real Time Data
Infrastructure to
Implement Data
Architecture
Real Time Data
Operations to Manage
Data Infrastructure
March 8, 2016

31

Real Time Data Governance Activities

Real Time Data Governance
Real Time Data Management Planning

Real Time Data Management Control

Understand Strategic Enterprise Real Time Data

Needs

Supervise Real Time Data Professional Organisations

and Staff

Develop and Maintain the Real Time Data Strategy

Coordinate Real Time Data Governance Activities

Establish Real Time Data Professional Roles and

Organisations

Manage and Resolve Real Time Data Related Issues

Identify and Appoint Real Time Data Stewards

Monitor and Ensure Regulatory Compliance

Establish Real Time Data Governance and

Stewardship Organisations

Monitor and Enforce Conformance withReal Time

Data Policies, Standards and Architecture

Develop and Approve Real Time Data Policies,

Standards, and Procedures

Oversee Real Time Data Management Projects and

Services

Review and Approve Real Time Data Architecture

Communicate and Promote the Value of Real Time

Data Assets

Plan and Sponsor Real Time Data Management

Projects and Services
Estimate Real Time Data Asset Value and Associated
Costs
March 8, 2016

32

Real Time Data Governance Inputs And Outputs

Inputs

Primary Deliverables

Business Goals
Business Strategies
IT Objectives
IT Strategies
Data Needs
Data Issues
Regulatory Requirements

Suppliers

Data Policies
Data Standards
Resolved Issues
Data Management Projects and
Services
Quality Data and Information
Recognised Data Value

Real Time Data

Governance

Data Producers
Knowledge Workers
Managers and Executives
Data Professionals
Customers

Business Executives
IT Executives
Data Stewards
Regulatory Bodies

Participants
Executive Data Stewards
Coordinating Data Stewards
Business Data Stewards
Data Professionals
DM Executive
CIO

March 8, 2016

Consumers

Tools
Intranet Website
E-Mail
Metadata Tools
Metadata Repository
Issue Management Tools
Data Governance KPI
Dashboard

Metrics
Data Value
Data Management Cost
Achievement of Objectives
# of Decisions Made
Steward Representation /
Coverage
Data Professional Headcount
Data Management Process
Maturity
33

Real Time Data Quality Management

Critical support process in organisational change management

Data quality is synonymous with information quality since poor
data quality results in inaccurate information and poor
business performance
Data cleansing may result in short-term and costly
improvements that do not address the root causes of data
defects
More rigorous data quality program is necessary to provide an
economic solution to improved data quality and integrity
Institutionalising and operationalising processes for data
quality oversight, management, and improvement hinges on
identifying the business needs for quality data and determining
the best ways to measure, monitor, control, and report on the
quality of data
Continuous process for defining the parameters for specifying
acceptable levels of data quality to meet business needs, and
for ensuring that data quality meets these levels
March 8, 2016

34

Real Time Data Quality Management Definition

and Goals

Definition
Planning, implementation, and control activities that apply quality
management techniques to measure, assess, improve, and ensure
the fitness of data for use

Goals
To measurably improve the quality of data in relation to defined
business expectations
To define requirements and specifications for integrating data
quality control into the system development lifecycle
To provide defined processes for measuring, monitoring, and
reporting conformance to acceptable levels of data quality

March 8, 2016

35

Real Time Data Quality Management Inputs And

Outputs
Inputs

Primary Deliverables

Business Requirements
Data Requirements
Data Quality Expectations
Data Policies and Standards
Business metadata
Technical metadata
Data Sources and Data Stores

Suppliers
External Sources
Regulatory Bodies
Business Subject Matter Experts
Information Consumers
Data Producers
Data Architects
Data Modelers

Participants
Data Quality Analysts
Data Analysts
Database Administrators
Data Stewards
Other Data Professionals
DRM Director
Data Stewardship Council
March 8, 2016

Real Time Data

Quality
Management

Improved Quality Data

Data Management
Operational Analysis
Data Profiles
Data Quality Certification
Reports
Data Quality Service Level
Agreements

Consumers
Data Stewards
Data Professionals
Other IT Professionals
Knowledge Workers
Managers and Executives
Customers

Tools
Data Profiling Tools
Statistical Analysis Tools
Data Cleansing Tools
Data Integration Tools
Issue and Event Management
Tools

Metrics
Data Value Statistics
Errors / Requirement Violations
Conformance to Expectations
Conformance to Service Levels

36

Real Time Data Quality Plan Definition Activities

Real Time Data Quality Plan Definition

March 8, 2016

1. Develop and Promote Data Quality

Awareness

2. Define Data Quality Requirements

3. Profile, Analyse and Assess Data

Quality

4. Define Data Quality Metrics

5. Define Data Quality Business Rules

6. Test and Validate Data Quality

Requirements

7. Set and Evaluate Data Quality Service

Levels

8. Continuously Measure and Monitor

Data Quality

9. Manage Data Quality Issues

10. Clean and Correct Data Quality

Defects

11. Design and Implement Operational

Data Quality Management Procedures

12. Monitor Operational Data Quality

Management Procedures and
Performance
37

Developing A Real Time Data Strategy Generalised

High Level Steps

10 high level steps with activities and tasks

Over 180 detailed tasks for a complete view of work required
Comprehensive set of steps from definition of what is required from real time data to
commissioning of real time data collection facilities to effective use of collected data
Real Time Data
Strategy and
Implementation

1 Real Time
Data Strategy
And Planning

2 Real Time
Data
Capability
Delivery

3 Real Time
Data
Development
And
Retirement

4 Real Time
Data
Management
and
Operations
Support And
Readiness

5 Workforce
Management

6 Real Time
Data
Provisioning

7 Real Time
Data
Collection
And
Distribution

8 Real Time
Data Trouble
Management

9 Real Time
Data
Performance
Management

10 Real
Time Data
Aggregation
and
Reporting

Activities
Tasks
March 8, 2016

38

Real Time Data Strategy and Implementation

Organisation, Function And Process Structure Steps 1-10
Real Time Data
Strategy and
Implementation
1 Real Time Data
Strategy And
Planning

2 Real Time Data

Capability Delivery

3 Real Time Data

Development And
Retirement

4 Real Time Data

Management and
Operations
Support And
Readiness

5 Workforce
Management

6 Real Time Data

Provisioning

7 Real Time Data

Collection And
Distribution

8 Real Time Data

Trouble
Management

9 Real Time Data

Performance
Management

10 Real Time Data

Aggregation and
Reporting

1.1 Gather And

Analyse Real Time
Data Information

2.1 Map And

Analyse Real Time
Data
Requirements

3.1 Gather And

Analyse New Real
Time Data Ideas

4.1 Enable Real

Time Data
Provisioning

5.1 Manage
Schedules and
Appointments

6.1 Allocate And

Install Real Time
Data

7.1 Collect
Management
Information And
Data

8.1 Survey And

Analyse Real Time
Data Trouble

9.1 Monitor Real

Time Data
Performance

10.1 Aggregate
Real Time Data
Records

1.2 Manage Real

Time Data
Research

2.2 Capture Real

Time Data
Capability
Shortfalls

3.2 Assess
Performance Of
Existing Real Time
Data

4.2 Enable Real

Time Data
Performance
Management

5.2 Plan and

Forecast
Workforce
Management

6.2 Configure And

Activate Real Time
Data

7.2 Process
Management
Information And
Data

8.2 Localise Real

Time Data Trouble

9.2 Analyse Real

Time Data
Performance

10.2 Report Real

Time Data
Records

1.3 Establish Real

Time Data
Strategy And
Architecture

2.3 Gain Real Time

Data Capability
Investment
Approval

3.3 Develop New

Real Time Data
Business Proposal

4.3 Support Real

Time Data Trouble
Management

5.3 Administer
and Configure
Workforce
Management

6.3 Test Real Time

Data

7.3 Distribute
Management
Information And
Data

8.3 Correct And

Resolve Real Time
Data Trouble

9.3 Control Real

Time Data
Performance

1.4 Define Real

Time Data
Support Strategies

2.4 Design Real

Time Data
Capabilities

3.4 Develop
Detailed Real
Time Data
Specifications

4.4 Enable Real

Time Data
Collection And
Distribution

5.4 Report
Workforce
Management

6.4 Track And

Manage Real Time
Data Provisioning

7.4 Audit
Management And
Security Data
Collection And
Distribution

8.4 Track And

Manage Real Time
Data Trouble

9.4 Report Real

Time Data
Performance

1.5 Produce Real

Time Data
Business Plans

2.5 Enable Real

Time Data
Support And
Operations

3.5 Manage Real

Time Data
Development

4.5 Manage Real

Time Data
Inventory

5.5 Manage Work

Order Lifecycle

6.5 Report Real

Time Data
Provisioning

8.5 Report Real

Time Data Trouble

9.5 Create Real

Time Data
Performance
Degradation
Report

1.6 Develop Real

Time Data
Partnership
Requirements

2.6 Manage Real

Time Data
Capability Delivery

3.6 Manage Real

Time Data
Deployment

4.6 Manage
Logistics

6.6 Close Real

Time Data Order

8.6 Close Real

Time Data Trouble
Report Flow

9.6 Track And

Manage Real Time
Data Performance
Resolution

1.7 Gain
Enterprise
Commitment To
Real Time Data
Plans

2.7 Manage
Handover To Real
Time Data
Operations

3.7 Manage Real

Time Data Exit

6.7 Issue Real

Time Data Orders

8.7 Create Real

Time Data Trouble
Report

9.7 Close Real

Time Data
Performance
Degradation
Report

6.8 Recover Real

Time Data

March 8, 2016

39

Step 1 Real Time Data Strategy And Planning

Processes And Functions Details
1 Real Time Data
Strategy And
Planning

1.1 Gather And

Analyse Real Time
Data Information

1.2 Manage Real

Time Data Research

1.1.1 Gather Real

Time Data
Information

1.2.1 Manage Real

Time Data Research
Investigations

1.1.2 Analyse New

Real Time Data
Requirements

1.2.2 Manage
Administration Of
Real Time Data
Research

1.1.3 Analyse To
Develop
New/Enhance Real
Time Data
Requirements

1.2.3 Define Real

Time Data Research
Assessment
Methodologies

1.3 Establish Real

Time Data Strategy
And Architecture

1.5 Produce Real

Time Data Business
Plans

1.6 Develop Real

Time Data
Partnership
Requirements

1.7 Gain Enterprise

Commitment To
Real Time Data
Plans

1.3.1 Establish Real

Time Data Strategy

1.4.1 Define Real

Time Data Support
Principles

1.5.1 Develop And

Deliver
Annual/Multi Year
Real Time Data
Business Plans

1.6.1 Identify The

Requirements For
Real Time Data
Capabilities

1.7.1 Identify
Stakeholders To
Real Time Data
Strategy And Real
Time Data Plans

1.3.2 Develop Real

Time Data Strategy

1.4.2 Define Real

Time Data Support
Policies

1.5.2 Forecast High

Level Real Time
Data Demand And
Capture New
Opportunities

1.6.2 Recommend
Real Time Data
Partnership

1.7.2 Gain Real Time

Data Strategy And
Real Time Data
Plans Stakeholders
Approval

1.3.3 Establish Real

Time Data Delivery
Goals

1.4.3 Define Real

Time Data Support
Performance
Standards

1.5.3 Assess Impact

Of Real Time Data
Business Plans

1.6.3 Determine
Extent Of Real Time
Data Capabilities
Sourcing

1.7.3 Gain
Enterprise
Commitment To
Real Time Data
Strategy And Real
Time Data Plans

1.3.4 Establish Real

Time Data
Implementation
Policies

March 8, 2016

1.4 Define Real

Time Data Support
Strategies

1.5.4 Identify
Timetables For New
Real Time Data
Capability
Introduction
1.5.5 Identify
Logistics For New
Real Time Data
Capability
Introduction

40

Step 2 - Real Time Data Capability Delivery

Processes And Functions Details
2 Real Time Data
Capability Delivery

2.1 Map And

Analyse Real Time
Data Requirements

2.2 Capture Real

Time Data Capability
Shortfalls

2.3 Gain Real Time

Data Capability
Investment
Approval

2.4 Design Real Time

Data Capabilities

2.5 Enable Real Time

Data Support And
Operations

2.6 Manage Real

Time Data Capability
Delivery

2.7 Manage
Handover To Real
Time Data
Operations

2.1.1 Capture Real

Time Data Demand
And Performance
Requirements

2.2.1 Capture Real

Time Data Capacity
Shortfalls

2.3.1 Develop Real

Time Data Capability
Investment
Proposals

2.4.1 Define Real

Time Data Capability
Requirements

2.5.1 Design Real

Time Data
Operational Support
Process
Improvements

2.6.1 Co-Ordinate
Real Time Data
Capability Delivery

2.7.1 Co-Ordinate
Real Time Data
Operational
Handover

2.1.2 Agree Real

Time Data
Infrastructure
Requirements

2.2.2 Capture Real

Time Data
Performance
Shortfalls

2.3.2 Approve Real

Time Data Capability
Investment

2.4.2 Specify Real

Time Data Capability
Infrastructure

2.5.2 Identify Real

Time Data Support
Groups, Skills And
Training

2.6.2 Ensure Real

Time Data Capability
Quality

2.7.2 Validate Real

Time Data
Infrastructure
Design

2.4.3 Select Real

Time Data Capability
At Other Parties

2.5.3 Identify Real

Time Data Support
Requirements

2.6.3 Manage
Commissioning Of
New Real Time Data
Infrastructure

2.7.3 Ensure Real

Time Data Handover
Support

2.2.3 Capture Real

Time Data
Operational Support
Shortfalls

2.6.4 Establish Real

Time Data Capability
Sourcing

March 8, 2016

41

Step 3 - Real Time Data Development And

Retirement Processes And Functions Details
3 Real Time Data
Development And
Retirement

3.1 Gather And

Analyse New Real
Time Data Ideas

3.2 Assess
Performance Of
Existing Real Time
Data

3.3 Develop New

Real Time Data
Business Proposal

3.4 Develop
Detailed Real Time
Data Specifications

3.5 Manage Real

Time Data
Development

3.6 Manage Real

Time Data
Deployment

3.7 Manage Real

Time Data Exit

3.1.1 Gather Real

Time Data
Information

3.3.1 Develop Real

Time Data Business
Proposal

3.4.1 Develop
Detailed Real Time
Data Technical
Specifications

3.5.1 Identify
Required Processes
And Procedures For
Real Time Data

3.6.1 Manage Real

Time Data Process
And Procedure
Implementation

3.7.1 Identify
Unviable Real Time
Data

3.1.2 Analyse Real

Time Data Classes

3.3.2 Gain Real Time

Data Business
Proposal Approval

3.4.2 Develop
Detailed Real Time
Data Support
Specifications

3.5.2 Develop
Required Processes
And Procedures For
Real Time Data

3.6.2 Manage Real

Time Data
Operational Staff
Training

3.7.2 Identify
Impacted Real Time
Data Customers

3.4.3 Develop
Detailed Real Time
Data Operational
Specifications

3.5.3 Develop
Service And
Operational
Agreements For
Real Time Data

3.6.3 Develop Real

Time Data
Supplier/Partner
Operational Support

3.7.3 Develop Real

Time Data Transition
Strategies

3.4.4 Develop
Detailed Real Time
Data Manuals

3.5.4 Gain Service

And Operational
Agreements
Approval For Real
Time Data

3.6.4 Manage Real

Time Data
Acceptance Testing

3.7.4 Manage Real

Time Data Exit
Process

3.1.3 Develop Real

Time Data Classes

March 8, 2016

3.5.5 Produce
Supporting
Documentation And
Training Packages
For Real Time Data

42

Step 4 - Real Time Data Management and Operations

Support And Readiness Processes And Functions Details
4 Real Time Data Management
and Operations Support And
Readiness
4.1 Enable Real Time Data
Provisioning

4.2 Enable Real Time Data

Performance Management

4.3 Support Real Time Data

Trouble Management

4.4 Enable Real Time Data

Collection And Distribution

4.5 Manage Real Time Data

Inventory

4.6 Manage Logistics

4.1.1 Plan And Forecast Real

Time Data Infrastructure
Requirements And Manage
Capacity Planning

4.2.1 Monitor And Manage

Regulatory Issues

4.3.1 Manage Real Time Data

Trouble And Performance Data
Collection

4.4.1 Manage And Administer

Real Time Data Collection And
Distribution

4.5.1 Manage Real Time Data

Inventory Database And
Processes

4.6.1 Manage Warehousing

4.1.2 Establish, Manage, And

Develop Organisetion, Tools And
Processes

4.2.2 Establish And Maintain

Performance Threshold
Standards

4.3.2 Manage Real Time Data

Infrastructure, Provisioning And
Preventive Maintenance
Schedules

4.4.2 Manage Real Time Data

Storage Facilities And Associated
Processes

4.5.2 Track And Monitor Real

Time Data Repository
Capabilities

4.6.2 Manage Orders

4.1.3 Develop And Implement

Capacity And Operational Rules
And Procedures

4.2.3 Undertake Performance

Trend Analysis

4.3.3 Create Report

4.4.3 Track, Monitor And Report

Real Time Data Collection
Processes And Capabilities

4.5.3 Identify Repository Issues

And Provide Reports And
Warnings

4.6.3 Track And Monitor

Logistics And Manage Real Time
Data Inventory

4.1.4 Perform Acceptance Test

And Address And Monitor The
Change

4.2.4 Monitor And Analyse Real

Time Data Performance Reports,
And Identify Issues

4.3.4 Establish Warehouse And

Manage Spares Including Other
Parties

4.4.4 Identify Data Collection

Issues And Report

4.1.5 Track And Supervise The

Rollout Of New And/Or
Modified Infrastructure

4.2.5 Correlate The Performance

Problem Reports And Manage
Inventory Repository

4.3.5 Track, Monitor And

Manage Real Time Data Trouble
Management Processes
Including Other Parties

4.1.6 Monitor, Report And

Release Mgmt. Of Real Time
Data Infrastructure And Capacity
Utilisation

4.2.6 Manage Real Time Data

Performance Data Collection

4.3.6 Provide Support For Real

Time Data Trouble Management
And Support Service Problem
Management Processes

4.1.7 Optimise Existing Real

Time Data Infrastructure
Utilisation

4.2.7 Establish, Maintain And

Manage The Support Plans

4.1.8 Track, Monitor And Report

Real Time Data Provisioning

4.2.8 Assess And Report Real

Time Data Performance
Management Processes

4.1.9 Update Inventory Record

4.2.9 Provide Supporting

Procedures And Quality
Management Support

March 8, 2016

4.6.4 Identify Logistic Issues And

Provide Reports

43

Step 5 - Workforce Management Processes And

Functions Details
5 Workforce Management
5.1 Manage Schedules And
Appointments

5.2 Plan And Forecast

Workforce Management

5.3 Administer And

Configure Workforce
Management

5.4 Report Workforce

Management

5.5 Manage Work Order

Lifecycle

5.1.1 Workforce
Management Schedule

5.2.1 Forecast Demand

5.3.1 Configure Work

Catalog

5.5.1 Issue Work Order

5.1.2 Determine Work

Schedule

5.2.2 Forecast Workforce

Availability

5.3.2 Administer Human

Real Time Data Catalog

5.5.2 Analyse And

Decompose Work Order

5.1.3 Manage Reservations

5.2.3 Adjust Durations

5.3.3 Administer
Organisation's Catalog

5.5.3 Assign Task

5.3.4 Administer Tools And

Materials Catalog

5.5.4 Dispatch Task

5.3.5 Configure Skill Catalog

5.5.5 Track And Manage

Work Order

5.3.6 Configure Schedules

5.5.6 Close Work Order

5.3.7 Administer
Registration And Access

5.5.7 Report On Work Order

5.1.4 Manage Appointments

5.3.8 Configure Logging And

Audit
March 8, 2016

44

Step 6 - Real Time Data Provisioning Processes And

Functions Details
6 Real Time Data
Provisioning

6.1 Allocate And

Install Real Time
Data
6.1.1 Determine
Real Time Data
Availability

6.2 Configure
And Activate
Real Time Data

6.2.1 Configure
Real Time Data

6.3 Test Real

Time Data

6.4 Track And

Manage Real
Time Data
Provisioning

6.7 Issue Real

Time Data
Orders

6.8 Recover Real

Time Data

6.4.1 Coordinate
Real Time Data
Provisioning
Activity

6.5.1 Monitor
Real Time Data
Order Status

6.7.1 Assess
Real Time Data
Request

6.5.2 Distribute
Real Time Data
Order
Notification

6.7.2 Create
Real Time Data
Orders

6.5.3 Distribute
Real Time Data
Provisioning
Reports

6.7.3 Mark Real

Time Data Order
For Special
Handling

6.1.2 Reserve
Real Time Data

6.2.2 Implement
Real Time Data

6.3.2 Develop
Test Plans

6.1.3 Release
Real Time Data

6.2.3 Activate
Real Time Data

6.3.3 Capture
Test Results

6.4.3 Manage
Real Time Data
Provisioning
Activity

6.1.5 Install And

Commission
Real Time Data

6.6 Close Real

Time Data Order

6.3.1 Test
Specific Real
Time Data

6.4.2 Track Real

Time Data
Provisioning
Activity

6.1.4 Allocate
Real Time Data

6.5 Report Real

Time Data
Provisioning

6.4.4 Update
Real Time Data
Repository

6.8.1 Develop
Real Time Data
Recovery Plan
6.8.2 Provide
Real Time Data
Recovery
Proposal
Notification
6.8.3 Request
Real Time Data
Recovery
Authorisation
6.8.4 Commence
Real Time Data
Recovery
6.8.5 Complete
Real Time Data
Recovery
6.8.6 Recover
Specific Real
Time Data

March 8, 2016

45

Step 7 - Real Time Data Collection And Distribution

Processes And Functions Details
7 Real Time Data
Collection And
Distribution

7.1 Collect
Management
Information And Data

7.2 Process
Management
Information And Data

7.3 Distribute
Management
Information And Data

7.4 Audit Management

And Security Data
Collection And
Distribution

7.1.1 Intercept
Events/Information

7.2.1 Determine
Recipients For
Information/Data

7.3.1 Distribute
Information/Data

7.1.2 Deliver
Management
Information

7.2.2 Filter
Information/Data

7.3.2 Manage
Distribution

7.2.3 Aggregate
Information/Data

7.3.3 Confirm
Distribution And CleanUp

7.2.4 Format
Information/Data
March 8, 2016

46

Step 8 - Real Time Data Trouble Management

Processes And Functions Details

8 Real Time Data

Trouble
Management

8.1 Survey And

Analyse Real
Time Data
Trouble

March 8, 2016

8.2 Localise Real

Time Data
Trouble

8.3 Correct And

Resolve Real Time
Data Trouble

8.4 Track And

Manage Real
Time Data
Trouble

8.5 Report Real

Time Data
Trouble

8.6 Close Real

Time Data
Trouble Report
Flow

8.7 Create Real

Time Data
Trouble Report

47

Step 9 - Real Time Data Performance Management

Processes And Functions Details
9 Real Time Data
Performance
Management

9.1 Monitor Real Time

Data Performance

9.2 Analyse Real Time

Data Performance

9.3 Control Real Time

Data Performance

9.4 Report Real Time

Data Performance

9.5 Create Real Time

Data Performance
Degradation Report

9.6 Track And Manage

Real Time Data
Performance Resolution

9.7 Close Real Time Data

Performance
Degradation Report

9.1.1 Manage Real Time

Data Performance Data

9.2.1 Perform Specific

Real Time Data
Performance Diagnostics

9.3.1 Instantiate Real

Time Data Performance
Controls

9.4.1 Monitor Real Time

Data Performance
Degradation Report

9.5.1 Generate Real Time

Data Performance
Degradation Problem

9.6.1 Coordinate Real

Time Data Performance

9.1.2 Record Real Time

Data Performance Data

9.2.2 Manage Real Time

Data Performance Data
Collection Schedules

9.3.2 Instantiate Real

Time Data Trouble
Controls

9.4.2 Distribute Real

Time Data Quality
Management Reports
And Summaries

9.5.2 Convert Report To

Real Time Data
Performance
Degradation Report
Format

9.6.2 Request Other

Parties Performance
Degradation Report
Creation And Update

9.1.3 Correlate Real Time

Data Performance Event
Notifications

9.6.3 Update First In Real

Time Data Testing
Results
9.6.4 Cancel Real Time
Data Performance
Degradation Report
9.6.5 Escalate/End Real
Time Data Performance
Degradation Report
9.6.6 Clear Real Time
Data Performance
Degradation Report
Status
9.6.7 Engage External
Party Real Time Data

March 8, 2016

48

Step 10 - Real Time Data Aggregation And Reporting

Processes And Functions Details
10 Real Time Data
Aggregation And
Reporting
10.1 Aggregation
Real Time Data
Records

10.2 Report Real

Time Data Records

10.1.1 Validate
Real Time Data
Records
10.1.2 Normalise
Real Time Data
Records
10.1.3 Convert
Real Time Data
Records
10.1.4 Correlate
Real Time Data
Records

10.1.5 Remove
Duplicate Real
Time Data Records
March 8, 2016

49

Using Real Time Data Strategy and Implementation

Approach
Timeline

Activity
Real Time Data Strategy and Implementation Plan
1 Real Time Data Strategy And Planning
1.1 Gather And Analyse Real Time Data Information
1.1.1 Gather Real Time Data Information
1.1.2 Analyse New Real Time Data
Requirements
1.1.3 Analyse To Develop New/Enhance Real
Time Data Requirements
1.2 Manage Real Time Data Research
1.2.1 Manage Real Time Data Research
Investigations
1.2.2 Manage Administration Of Real Time
Data Research
1.2.3 Define Real Time Data Research
Assessment Methodologies
1.3 Establish Real Time Data Strategy And
Architecture
1.3.1 Establish Real Time Data Strategy
1.3.2 Develop Real Time Data Strategy
1.3.3 Establish Real Time Data Delivery Goals
1.3.4 Establish Real Time Data
Implementation Policies
1.4 Define Real Time Data Support Strategies
1.4.1 Define Real Time Data Support
Principles
1.4.2 Define Real Time Data Support Policies
1.4.3 Define Real Time Data Support
Performance Standards

March 8, 2016

50

Using Real Time Data Strategy and Implementation

Approach

Use the proposed work breakdown to produce a detailed

plan

March 8, 2016

51

Real Time Architecture High-Level Components

Data Sources

Data Collection
And Data Source
Management

Communications
And Security

Data Quality/
Summary/
Filter/
Transformation

Data Storage

Data Storage
Infrastructure

Data Reporting
and Analysis

External
Systems (Asset
Management,
Workforce
Management)

System
Management,
Administration
and Control

March 8, 2016

Data
Integration

52

Real Time Architecture High-Level Components

Component

Description

Data Sources

These are the data collection/generation sources/sensors installed in the data collection landscape and new
sensor devices and signal data sources. Over time the approach to data collection may be standardised and old
equipment replaced.
This logical component acts as a local front-end to existing signal data collection/generation sources/sensors. It
eliminates the need to replace existing devices. It offers a standard interface. It manages the sensor
infrastructure and landscape

Data Collection And

Data Source
Management
Communications And
Security
Data Integration
Data Quality/
Summary/ Filter/
Transformation
Data Storage
Data Storage
Infrastructure

This is the communications infrastructure used to securely transmit remotely collected data from local data
sources/sensors and data collection units to the central facility.
This component manages the receipt of multiple data types in multiple formats from multiple data sources.
This component applies data quality algorithms to intrinsically noisy real time data to make it more usable.
Data may be filtered, summarised and transformed prior to storage
This is the software component for storing in a structured manner and providing access to real time data.
This is the underlying data storage infrastructure. The volumes of real time data are potentially very substantial
hundreds of millions of data points per day. The data storage requirements could amount to thousands of
Terabytes over several years. This components includes facilities for backup, recovery, archiving and deletion.
This component provides facilities to manage, administer and control the overall real time system.

System Management,
Administration and
Control
This provides reporting and analysis facilities to meet a wide variety of business requirements.
Data Reporting and
Analysis
Real time data can be merged with other data such as asset to provide a usage dimension to static asset data
External Systems
and to integrate with workforce management to manage sensor infrastructure.
March 8, 2016

53

Possible Approaches To Real Time Data Architecture

You can expend a great deal of time, resources and money

on defining the requirements of an idealised real time data
architecture before embarking on any procurement and
implementation

OR

You can research the viable products and their

functionality and what other water utilities have
implemented and define requirements in terms of what is
realistically achievable, reducing costs and time and
delivering results and learning more quickly

March 8, 2016

54

Getting Real Time Data Right

Avoid false starts. Balance implementation urgency and pace with

organisational readiness and maturity. Success requires change
management
Cannot do it all at once
Real time data is an enabler and not an end in itself
Embed use of real time data data in the organisation
Don't turn it into a "finding the right tool" decision
Recognise the interaction of governance, processes and tools that enables
the organisation optimise its real time data investment
Focus on value, risks and prioritisation with active engagement of key
stakeholders
Avoid using it solely for one-time annual budget decisions. It is about
continuous alignment, tracking and benefits realisation of existing real
time data systems and new projects
Clarify governance and put a process in place that uses portfolio
management approach to consider, make and enforce decisions
March 8, 2016

55

Summary

Real time data includes Telemetry, Big Data, Smart Metering

and Internet of Things
Represents an idealised organisation and process breakdown
across entire spectrum of real time data from strategy to
workforce management to device installation, data collection
and data usage and actioning
Provides a basis for developing a work breakdown and an
implementation plan
Real time data infrastructure and systems without organisation
and processes will yield few benefits
Represents a comprehensive structure that an organisation can
adapt to meet its long-term real time data needs
Enables application and use of real time data to be embedded
in the organisation
March 8, 2016

56