Extract Interface Load
Staging Interface
Tables Tables
(Source ®
Extract Transform Load Destination ®
Instance) Instance
Validation
Standard API’s and
Reconciliation
Abstract:
• The Data Migration process is about,
1) Extracting the Data from Source Instance
2) Validating the Data at Destination Instance
3) Loading the Data at Destination Instance.
Scenario:
• Data Migration from 11.5.5 to 11.5.10.2 instance.
• Huge Volume of Data.
• Production Go-live instance availability - Time limitation.
• Loaded data in terms of quality and quantity
• Others – Execution sequence, strategy, reporting etc.
Best Practice:
In terms of learning's gathered during the data migration process which can provide
basics for a smooth execution of a DM project.
1. DM Strategy to be dependent on the number of records to be migrated against each
components. E.g.: If we have around 50 records to be migrated, it’s better to get that
migrated manually for a component.
2. A Separate USER to migrate the data. This will help identifying the records easily, which
are migrated.
3. The development/test instances should have enough capacity to mock the data
migration activity for identifying and estimating the timings required for the data
migration process.
4. The sequence of execution of components needs to be identified. Separate investigation
to be performed to identify the components that can run in parallel. Note: When planning
to run the components in parallel, take care that the instance is capable enough to
support the parallel activity. Else it might produce negative results. Static and Dynamic
component wise segregation will provide better picture.
5. Identify the components which needs to be migrated during early phases of data
migration process. Its always better to have each component tested at least twice along
with all the other components. Say for e.g.: if 6 test cycles are there, make sure to get
all the components identified, developed and tested at least by 4th test cycle.
Best Practice:
6. For better control, Extraction of records from Source instance (or OU etc.) , Validation
and Loading to be planned separately. The validation of extracted records especially by
business would provide better confidence.
7. If any kind of validation programs are run against the extracted records, especially when
validated against the set ups, its always advisable to send out the reports to the set up
team who might have missed some set ups which can be done, before loading the
records. Note: this will allow to get a better number/percentage of load. Make sure
that this process is in place as part of the data migration activity.
8. Always design the validation programs in such a way that they can be re executed
multiple times without much effort (doing any back end updates. Performing any kind of
back end updates during go live is too tough and DBA’s won’t be willing to do that.
9. Performance is a critical area which needs focus for a dynamic go live activity. Take care
specifically when we got large volume of data to be migrated. Each component needs to
be tuned to produce best results.
10. Identify the Data Base Triggers (standard as well as custom) which are defined against
the base tables where we populate the data. This helps us to disable the triggers which
are not required while during the Data migration activity. Take good care to Enable them
back once DM activity is finished.
Best Practice:
11. Take care to cross verify data pertaining to the specific data migration especially when using
standard interface tables. Note: standard interface tables may have some other data also
which is already there existing. In the sense it should not be a surprise to the DM team, to note
that some other data was also present there in the interface tables along with the Migration
data which might also get loaded along. This might create difficulty in generating DVR (Data
Validation reports) reports especially when reports make use of created_by user as Data
Migration User. Plans should be in place for what exactly has to be done. Plans can be there
to back up these other records to a back up table and remove this from interface table and
move it back to interface once DM activity is over.
12. Cross component training needs to be done within team members – Execution perspective.
This allows arrangement in executing the components in case of personal emergencies. Note:
DM is a very dynamic activity where we should not loose execution time in between, once the
dependent tasks are executed.
13. When large volume of data needs to be migrated for a component, it’s better to do data
migration in parts for that component as different sets of data, and make sure that each cycle
is complete. Eg: When large volume of data is there, plan to do it in different cycles which
involves same set of process with different (logical) sets of data. And make sure to cross verify
each cycles data in the interface tables, error tables, staging tables etc to be in better control.
Note: Some times it happens like, although we ran a standard import, it may not even pick up
the records or it might change the statuses of records to some strange values which we
may not have ever seen.
14. When large volume of data is involved, plan to take back ups of Standard interface tables,
error tables and clean these tables so that the further runs would be faster. And also can think
of
re analyzing/gather stats the interface tables each time which improves performance.
Best Practice:
15. Test cycles for data migration should be meant to test the set ups, instance, data quality,
load against extracts, the time taken for executing each component and ultimately the
time taken for the whole migration process. Each test cycle should provide inputs to further
improvement of data quality and better load timings and better number/percentage of data
load. This may also include the learning’s for parallel executions of components which can be
run independently.
16. The more test cycles we do the better we achieve.
17. Reporting is a very important activity related to data migration. Define the reporting in such a
way that, the report shows up no ID fields, until and unless asked for. For e.g.: rather than
Org_id use Org_name. The success/failure counts should match the extract counts.
18. If huge volume of data is spooled in for reports, think of alternatives like having data reporting
as different sets. For e.g.: data reporting in terms of OU wise, use sql*plus to spool the data etc.
19. Another identification is that data migration projects are the ones which well suited for six
sigma, because DM projects play around with numbers(Extracted, loaded etc).
20. Execution team should be structured in such a way to have back up resources for executing
the same component. This is very much required for components with huge volume of data and
which are time consuming to load. (Load includes loading to base tables as well as reporting.
Any thing which takes more than 12 hours should have an additional back up resource to
execute the same component. Plan the back up resource so that the primary execution is done
by the critical resource and the back up resource is just to carry the load forward).
Take Away / Benefits and Business Value Generated
With these we could achieve,
(1) Very Good percentage of loaded data – Over all 99%.
(2) Very Good Quality of data loaded – Which was very much
appreciated by the client.
(3) Very Good Load Time for a Dynamic Go Live.
