Professional Documents
Culture Documents
Table of Contents
1 Introduction.................................................................................................................................. 4
1.1 Background.......................................................................................................................... 4
1.2 Overview.............................................................................................................................. 4
3 Restoration process................................................................................................................... 15
3.1 Checkpoints for Restoration process..................................................................................23
3.1.1 HUB/HUB_REM/IFAI/HMI: restoring data in Reprocessing tables...................................24
3.1.2 OHC/CARM/IFAI/HMI/BASEL/External/Parameter/HUB_REM/XXXXX.Net: Restoring
process ......................................................................................................................................... 25
4 Recovery process...................................................................................................................... 27
4.1 HUB.................................................................................................................................... 28
4.2 CARM................................................................................................................................. 28
4.3 OHC................................................................................................................................... 28
4.4 BASEL................................................................................................................................ 29
4.5 External.............................................................................................................................. 29
4.6 XXXXXnet.......................................................................................................................... 29
4.7 Parameter Files.................................................................................................................. 29
4.8 HMI, HUB_REM, XXXXXnet.............................................................................................. 30
Table of Figures
Figure 1: Process flow diagram for backup and archive process.......................5
Table of Tables
Table 1: Jobs not to be run during Recovery process......................................23
1 Introduction
This document provides an overview of the design related to Change Data Capture (CDC) Daily Data
Backup, Storage and Restoration for all the lookup and reprocessing tables under One XXXXX
Business Intelligence (OCBB) Integrated Customer Relationship Analysis (ICRA).
1.1 Background
As per the functional requirement FR5.4, a process to backup and restore the reprocessing and/or
lookup tables needs to be designed and implemented so that the OCBB environment can be restored
to normal processing easily whenever data issues arise.
The objective of the design is to take the backup of all the source tables used in Daily Cycle into
backup folder and restore it based on the business request.
Daily cycle data is stored for 32 days.
1.2 Overview
The scope of this design document is to meet the functional requirement FR5.4 from Global Analytics
Detailed Requirements document. The following are the details of the requirement:
1) Retain copies of the Source System and CDC extracts as follows:
Daily Load data should be retained for 32 days (If parameter is raised, then it can be defined by
the user)
2) The OCBB environment and all current data should be available within 24 hours of a system or
process failure
3) The system should also support purging of data based on the aforementioned data retention
parameters, or additional parameters established at the local or regional level.
This backup and restore process is mainly to re-run the cycles from the required date. This result and
data file is mainly for the Support Team and not for the business users.
This activity is not applicable for restoring the initial cycle data. If there
is any issue with the initial cycle data, it has to be started from the
beginning of the cycle.
Backup and Archive process is implemented with the help of Unix scripts.
2.2 System flow chart
The following figure illustrates the system flow chart diagram for backup and archive process:
Figure 2: System flow chart for backup process
Process date: Process date is nothing but load date (Parm Date) from
the cycle.
2.2.2.12 *.msgs Files (Reprocessing table backup data message files) Archiving
Refer to the following points related to *.msgs Files Archiving process:
1) Retrieves all the files with *.msgs extension at the backup landing path($prmDirLandingBkp)
2) Combines all the *.msgs files into a single file named (OCBB_Daily_del_msgs.<Process
date>.tar) using the unix tar command and saves at the path $prmDirdatabkp (parameter
from parameter file)
3) Compresses the tar file using the Unix gzip (OCBB_Daily_del_msgs.<Process date>.tar.gz)
command
4) Removes all the *.msgs files from original location.
All the gzip files that are older than the Retention Period (32 days) are removed.
2.3 Pre-job requirements
Refer to the following prerequisites before the backup and archive scripts run.
2) Please run the script, OCBB_Archive_Process.sh to take the backup of the all the source
files that are landed on Cycle Hold Day 1
Script name: <Script_path>/<country_cde>/ OCBB_Archive_Process.sh
Command line:
For example:
sh OCBB_Archive_Process.sh /data/ds/dpr_bir2_dev/scripts bn
bir2_wh.param
This script should be run everyday to take the backup of the source
files that fall under Cycle Hold Days.
This script should be run with proper prmDate and landing directories.
For update prmDate and landing directories, the following script should be run before the
aforementioned script:
Script name: <Script_path>/<country_cde>/OCBB_Recovery_PrmFileUpdate.sh
Command line:
For example:
a) Assume 1st May 2010 to 5th May 2010 data was corrupted or bad data was loaded;
therefore data needs to recover back through the recovery process
b) Assume 3 Man Days has consumed to recover the 1st May 2010 to 5th May 2010 data; so
cycle has hold on for 3 days. That is the cycle is held up to 8th May 2010. Therefore, the 3
days regular Daily cycle which is on hold should run from Recovery process directory
(/data/ds/dpr_bir2_dev/landing/recovery).
Regular Daily Control-M draft can be used to run the three days data
mentioned here.
c) Assume 1 and half days have been consumed to run the 3 days data (point 2). That is
Cycle is held up to 10th May 2010 noon.
d) Hence, again 1 cycle data should run from Recovery process directories
(/data/ds/dpr_bir2_dev/landing/recovery).
Regular Daily Control-M draft can be used to run the one day data
mentioned here.
e) 10th May 2010 noon onwards we can run actual Regular Daily Cycle from original landing
directory (/data/ds/dpr_bir2_dev/landing).
3) OCBB_Recovery_nickname.ksh script should be run before recovery process. To run this
script DBA privileges is required.
Script name: <Script_path>/<country_cde>/OCBB_Recovery_nickname.ksh
Command line:
sh OCBB_Recovery_nickname.ksh
sh OCBB_Recovery_stg_load.ksh
Where,
prmDirLanding = /data/ds/dpr_gacpt_dev/landing/bkp
prmDirXml = /data/ds/dpr_gacpt_dev/landing/bkp
prmDate = YYYY-MM-DD
startdt = YYYY-MM-DD
enddt = YYYY-MM-DD
After updating the inputs; please validate the aforementioned file.
Script name: <Script_path>/<country_cde>/OCBB_Recovery_PrmFileUpdate.sh
The following script should be run before the Recovery process to update the parameter files:
Command line:
You should use the letter R at the end of the command line.
R indicates Recovery process.
The following script should be run at the end of the Recovery process to delete all files [Source
files, .del files [Source table backup data], and Temporary files] that are restored from backup
files:
Command line:
You should use the letter D at the end of the command line to delete
the files that were recovered on a particular day.
It is recommended to delete files everyday end of recovery cycle.
D indicates Delete.
The following script should be run end of the Recovery process to update the parameter files to
Resume the Regular Daily Run:
Command line:
You should use the letter O at the end of the command line.
O indicates Original (Regular Daily Cycle) process.
6) Untar all or required files to get the sources files into the backup directory based on
requirement. This can be done with the following script:
Script name: <Script_path>/<country_cde>/OCBB_CDC_Untar_Process.sh
For example:
sh OCBB_CDC_Untar_Process.sh /data/ds/dpr_gacpt_dev/scripts bn
bir2_wh.param
ALL/HUB/OHC/CARM/XXXXX_NET/IFAI/HMI/HUB_REM/GPS/EXTERNAL/PARAMETER/BAS
EL
sh OCBB_CDC_InsertMode_tables.sh /data/ds/dpr_gacpt_dev/scripts bn
bir2_wh.param InputFile_OCBB_CDC_InsertMode_tables
8) Clean up all the Reprocessing Source tables where REJECT_IND is set to N and whose
LOAD_DT falls within the recovery date range. This should be done for all the Source Systems
(HUB/HUB_REM/HMI/IFAI) or the required Source System based on the system(s) to be
recovered. This can be done with the help of the following script:
Script name: <script_path>/<country_cde>/OCBB_CDC_Rep_Table_RejectRecord_Del.sh
For example:
sh OCBB_CDC_Rep_Table_RejectRecord_Del.sh
/data/ds/dpr_gacpt_dev/scripts bn bir2_wh.param
InputFile_OCBB_CDC_Rep_Table_RejectRecord_Del ALL/HUB/HUB_REM/HMI/IFAI
9) Once the aforementioned points are taken care, start importing the *.del files to the respective
tables based on the requirement. This can be done with help of the following script:
Script name: <script_path>/<country_cde>/ OCBB_CDC_Recovery_Process.sh
For example:
sh OCBB_CDC_Recovery_Process.sh /data/ds/dpr_gacpt_dev/scripts bn
bir2_wh.param InputFile_OCBB_CDC_BKP_Process ALL/HUB/HUB_REM/IFAI
10) Make sure you have copied all the required data from Source System Day1 cycle and placed
into recovery directory, $prmDirdatabkp (/data/ds/dpr_gacpt_dev/landing/bkp)
a) Restore process again based on the impact; it should validate and process based on the
current situation
b) Further, based on the affected tables and Source System to be recovered, first restore the
corresponding Source System files from the backup directory and run all the required
source to interface jobs to populate Day 1 worth of data with proper LOAD_DT to
corresponding Interface tables.
11) Validate required Source System data OHC, CARM, IFAI, HMI and XXXXX.net Source System
Day 1 input files and location. Please find the here
12) You can run multiple days of data but it is required to execute till source to Interface table with
one day data to get correct LOAD_DT for all the records
13) After successful execution of Source to Interface job with single day data; you need to run
OCBB_Recovery_PrmFileUpdate.sh
This script should be run at the end of the Recovery process to delete all files [Source files, .del
files (Source table backup data), and Temporary files] that are restored from backup files.
Command line:
You should use the letter D at the end of the command line to delete
the files that were recovered on a particular day.
It is recommended to delete files everyday at the end of the recovery
cycle.
D indicates Delete.
14) After successful execution of source to interface job with single day data, start run CDC
Backup/Archive script process manually and take that particular days data backup
15) Repeat the same process for the remaining days
The following scripts should be run on only Day1; please do not run these scripts on the
remaining days.
a) OCBB_Recovery_nickname.ksh
b) OCBB_Recovery_stg_load.ksh
c) OCBB_CDC_InsertMode_tables.sh
d) OCBB_CDC_Rep_Table_RejectRecord_Del.sh.
16) Please do not run the following jobs while the Recovery process is in progress:
Job names
Group No. 1
djpHewr2XFM_CDC_PRODUCT_UPS
djpHewr2XFM_CDC_INVOLVED_PARTY_UPS_1
djpHewr2XFM_CDC_ACCOUNTING_UNIT_UPS_1
djpHewr2XFM_CDC_CHG_TABLES
djpHewr2XFM_CDC_LOCATION_UPS
djpHewr2XFM_CDC_ARR_CHG_TABLES_1
djpHewr2XFM_CDC_ARRANGEMENT_UPS_1
djpHewr2XFM_CDC_ARRANGEMENT_UPS_2
djpHewr2XFM_CDC_ARRANGEMENT_UPS3
djpHewr2XFM_CDC_EVENT_UPS
djpHewr2XFM_CDC_ARR_CHG_TABLES_2
djpHewr2XFM_CDC_ARR_CHG_TABLES_3
djpHewr2XFM_CDC_ARR_CHG_TABLES_4
djpHewr2XFM_CDC_INVOLVED_PARTY_UPS_2
djpHewr2XFM_CDC_INVOLVED_PARTY_UPS_4
Job names
djpHewr2XFM_CDC_ACCOUNTING_UNIT_UPS_2
djpHewr2XFM_CDC_ARRANGEMENT_UPS_3
djpHewr2XFM_CDC_ARRANGEMENT_UPS_4
djpHewr2XFM_CDC_ARRANGEMENT_UPS_5
djpHewr2XFM_CDC_ARRANGEMENT_UPS_6
djpHewr2XFM_CDC_ARRANGEMENT_UPS_7
djpHewr2XFM_CDC_ARRANGEMENT_UPS_8
djpHewr2XFM_CDC_ARRANGEMENT_UPS_9
djpHewr2XFM_CDC_ARRANGEMENT_UPS_10
djpHewr2XFM_CDC_ARRANGEMENT_UPS_11
djpHewr2XFM_CDC_ARRANGEMENT_UPS_16
djpHewr2XFM_CDC_ARRANGEMENT_UPS_17
djpHewr2XFM_CDC_ARR_CHG_TABLES_7
djpHewr2XFM_CDC_ARR_CHG_TABLES_8
djpHewr2XFM_CDC_EVENT_ODBC_HUBODS
djpHewr2XFM_CDC_IP_ODBC_HUBODS
djpHewr2XFM_CDC_EVENT_UPS_SITEHUBFP
djpHewr2XFM_CDC_ARRANGEMENT_UPS_10_SITEHUBFP
djpHewr2XFM_CDC_ARRANGEMENT_UPS3_ATMFP
djpHewr2XFM_CDC_IFAI_ARRANGEMENT_UPS
djpHewr2XFM_CDC_ARRANGEMENT_UPS_18
djpHewr2XFM_CDC_INVOLVED_PARTY_UPS_3
djpHewr2XFM_CDC_PCM_CHG
djpHewr2XFM_CDC_PCM_UPS
Group No. 3
djpHewr2XFM_VIEW_IP
djpHewr2XFM_VIEW_EVENT
djpHewr2XFM_VIEW_ARR
djpHewr2XFM_CHG_HIE
djpHewr2XFM_CDC_REMIT_ODBC
djpHewr2XFM_CDC_PRODUCT_ODBC
Job names
djpHewr2XFM_CDC_LOCATION_ODBC
djpHewr2XFM_CDC_IP_ODBC
djpHewr2XFM_CDC_IFAI_ARRANGEMENT_CHG
djpHewr2XFM_CDC_EVENT_ODBC_SITEHUBFP
djpHewr2XFM_CDC_EVENT_ODBC
djpHewr2XFM_CDC_AU_ODBC
djpHewr2XFM_CDC_ARR_ODBC_SITEHUBFP
djpHewr2XFM_CDC_ARR_ODBC_ATMFP
djpHewr2XFM_CDC_ARR_ODBC
Group # 5
djpHewr2XFM_CDC_EVENT_DEL
djpHewr2XFM_CDC_ARRANGEMENT_DEL3
djpHewr2XFM_CDC_ARRANGEMENT_DEL_17
djpHewr2XFM_CDC_ARRANGEMENT_DEL_16
djpHewr2XFM_CDC_ARRANGEMENT_DEL_15
djpHewr2XFM_CDC_ARRANGEMENT_DEL_14
djpHewr2XFM_CDC_ARRANGEMENT_DEL_13
djpHewr2XFM_CDC_ARRANGEMENT_DEL_5
djpHewr2XFM_CDC_ARRANGEMENT_DEL_4
djpHewr2XFM_CDC_ARRANGEMENT_DEL_12
djpHewr2XFM_CDC_ARRANGEMENT_DEL_11
djpHewr2XFM_CDC_ARRANGEMENT_DEL_3
djpHewr2XFM_CDC_ARRANGEMENT_DEL_2
djpHewr2XFM_CDC_ARRANGEMENT_DEL_7
djpHewr2XFM_CDC_ARRANGEMENT_DEL_6
djpHewr2XFM_CDC_ACCOUNTING_UNIT_DEL
djpHewr2XFM_CDC_ARRANGEMENT_DEL_1
djpHewr2XFM_CDC_LOCATION_DEL
djpHewr2XFM_CDC_INVOLVED_PARTY_DEL_2
djpHewr2XFM_CDC_PRODUCT_DEL
djpHewr2XFM_CDC_EVENT_DEL_SITEHUBFP
Job names
djpHewr2XFM_CDC_ARRANGEMENT_DEL_10_SITEHUBFP
djpHewr2XFM_CDC_IFAI_ARRANGEMENT_DEL
djpHewr2XFM_CDC_ARRANGEMENT_DEL_18
djpHewr2XFM_CDC_INVOLVED_PARTY_DEL
djpHewr2XFM_CDC_PCM_DEL
Table 1: Jobs not to be run during Recovery process
17) Once all the recovery days of data is available in the interface tables, run all the WH jobs and
populate data into WH tables
18) Validate the records in the target tables and continue the cycle
19) The same approach cannot be implemented for IFAI Source System, which processes CDC
data population with manual process instead through Attunity. For those tables only latest
records are populated instead of all the records.
20) After successful completion of restore and recovery process, the following script updates the
parameter files. Script mentioned here should be run at the end of the Recovery process to
update the parameter files to resume the Regular Daily Run
Command line:
You should use the letter O at the end of the command line.
O indicates Original (Regular Daily Cycle) process.
21) After successful completion of restore and recovery process, make sure to validate the
corrected data and continue the ongoing Daily Cycles.
sh OCBB_CDC_Untar_Process.sh /data/ds/dpr_gacpt_dev/scripts bn
OCBB_initial_src_int.param ALL/HUB/OHC/CARM/XXXXX_NET/IFAI/HMI/
HUB_REM/GPS/EXTERNAL/PARAMETER/BASEL
3) Once the Unzip and Untar process is done, the files are available in the backup directory; the
aforementioned script also removes the tar files from landing directory
4) The *.del files that have been untarred by the Untar script are used for importing data into the
reprocessing tables based on the Source System required are to be restored. This is done with
the help of the following script:
Script name: <Script_path>/<country_cde>/OCBB_CDC_Recovery_Process.sh
For example:
sh OCBB_CDC_Recovery_Process.sh /data/ds/dpr_gacpt_dev/scripts bn
OCBB_initial_src_int.param InputFile_OCBB_CDC_BKP_Process
ALL/HUB/HUB_REM/HMI/IFAI
5) Once data is imported into the required Source System tables, execute the required jobs. Run
the corresponding Source Jobs and/or Groups and Force OK the remaining source Jobs and
Groups.
6) If there is any data from External Source System data files or any other files available, continue
the corresponding Combine Jobs and/or Groups and Force OK the remaining Source System
Combine Jobs and Groups if there is no data file.
7) After validating recovery data into Interface tables (required), continue the corresponding WH
Jobs/Groups and Force OK the remaining Source System WH Jobs/Groups.
8) Validate the recovery data and repeat this process till completion of recovery and latest data in
the target date.
3.1.2 OHC/CARM/IFAI/HMI/BASEL/External/Parameter/HUB_REM/XXXXX.N
et: Restoring process
Refer to the following points for restoring data in the Reprocessing tables:
1) Take backup from the same tables with existing data before importing the restored data into
interface tables and delete all the rejected records from the previous cycles from Reprocessing
and Interface tables (this can help us to restore the latest image if recovery process is not
completed successfully as these rejects may occur again with backup data).
2) Unzip and untar the OCBB_Daily_<SRCE>.<process_date>.tar.gz files that have been
archived based on the Source System to be recovered. This could be for a single or multiple
systems. The following script is used for this process:
Script name: <Script_path>/<country_cde>/ OCBB_CDC_Untar_Process.sh
For example:
sh OCBB_CDC_Untar_Process.sh /data/ds/dpr_gacpt_dev/scripts bn
OCBB_initial_src_int.param ALL/HUB/OHC/CARM/XXXXX_NET/IFAI/HMI/
HUB_REM/GPS/EXTERNAL/PARAMETER/BASEL
3) Once the Unzip and Untar process is done, the files are available in the backup directory; the
aforementioned script also removes the tar files from the landing directory
a) Calculate the number of days for the recovery process based on impact analysis
b) List the number of days/date file for Recovery process
c) Pack identified required files from the backup folder that were processed daily with Process
Date (LOAD_DT) and stored back in the backup folder with Tar/zip format during backup
process
d) Start run the recovery process. Each day of data recovery process is an independent
process and cycle for the other days.
e) One execution should have one day data only and LOAD_DT is required to be taken from
Backup Tar file name itself. This way we can ensure particular days of data with the
corresponding process date. As BASEL does not have the facility to identify the particular
days of data, this process is very important for further activity.
For example:
If recover process is to be done from 5th to 10th day of data on a particular month, take 5th
of Backup Tar file from backup folder and place in the required BASEL landing directory;
run all the corresponding Combine jobs. However, ensure LOAD_DT, it should update
prm_date(LOAD_DT) value from Tar file name itself and it is required to update Combine
job parameter file to get correct LOAD_DT in the corresponding Interface table for each
process.
f) After successful completion of 5th day data till interface level, again the 6th day data is
processed with 6th day process date (LOAD_DT) same as 5th day process and it is
repeated till the 10th day of month data.
g) Ensure interface table level process is repeated till 10th day process and continued for the
remaining job execution to populate data into warehouse table in single time execution
h) Source file to Interface table process is a multiple process based on number of days.
However, Interface table to WH process is a single process and processes latest records at
the end.
i) Ensure the process date matches with the required LOAD_DT(Prm_date) during recovery
j) Remove all the Source files from original location after successful completion of each
execution
k) Validate the records in interface table and WH tables based on impact analysis
l) Execute all the required jobs. Run corresponding Source Jobs/Groups and the remaining
Jobs and Groups
m) After validation, recover data into Interface tables(required) and continue the
corresponding WH jobs based on the dependency with HUB data due to Customer and
some Product Details derived from HUB Source System and Force OK the remaining
Source System WH Jobs/Groups.
n) Validate data recovery; repeat this process till completion of recovery and latest data in the
target date.
During the original run, all the external files are read from following
directory Server Path:
\<project>\landing\<CountryCode>\
However, during recovery process all the files are available in the
recovery folder. Therefore, it is necessary to update all the parameters
required to read the data from the following folder to be revoked after
the recovery process.
Server Path: \<project>\landing\<CountryCode>\ <Recovery>
Before starting this process; you should read Restoration process point 2.
As per prerequisites, during recovery and restoration process all the Daily cycles should be on hold till
completion of recovery process. After the completion of the recovery cycle the hold data would be
processed as follows. The following scripts do the Attunity export and import which helps in processing
the back log data:
<Script_path>/<Country_cde>/OCBB_GA_SP_EXPORT.sh
<Script_path>/<Country_cde>/OCBB_GA_SP_IMPORT.sh
For example:
Before running the aforementioned scripts, run the following script to update the prmDate and some
directory paths
Command line:
You should use the letter O at the end of the command line.
O indicates Original (Regular Daily Cycle) process.
Please validate all the parameter files for prmDate and directory paths.
4.1 HUB
Refer to the following steps pertaining to the recovery process for HUB:
1) Reload the Attunity Stream Position table from the first day of backup files with help of
OCBB_GA_SP_EXPORT.sh
OCBB_GA_SP_EXPORT.sh - Taking backup from Attunity
OCBB_GA_SP_IMPORT - Reloading data from backup file to Attunity
2) Validate the Attunity STREAM POSTION tables after loading from backup files
3) Retrieve PARM_DATE from STREAM_POSTION table backup filename
4) Do not run HUB_DATE_CTRL jobs during back log
5) Update the parm_dt to the parameter file to get correct LOAD_DT for all the records. This can
be done with the help of OCBB_Recovery_PrmFileUpdate.sh script. It can be ignored if this
process is done for other Source Systems as well.
6) Run the CDC jobs to load the day 1 data to the Staging tables
7) Repeat the same process for more days of hold.
4.2 CARM
Refer to the following steps pertaining to the recovery process for CARM:
1) Restore the Day 1 file from the archive directory using the restore process
2) Retrieve parmDate from the backup filename
3) Update parm_dt to the parameter file to get correct LOAD_DT for all the records
4) Run the CARM jobs to process the Day1 data
5) Repeat the same process for more days of hold.
4.3 OHC
Refer to the following steps pertaining to the recovery process for OHC:
1) Restore the Day 1 file from the archive directory using the restore process
2) Retrieve parm_date from the backup filename
3) Update the parm_dt to the parameter file to get correct LOAD_DT for all the records
4) Run the OHC combine jobs to process the Day1 data
5) Repeat the same process for more days of hold.
4.4 BASEL
Refer to the following steps pertaining to the recovery process for BASEL:
1) Restore the Day 1 file from the archive directory using the restore process
2) Retrieve parm_date from the backup filename
3) Update the parm_dt to the parameter file to get correct LOAD_DT for all the records
4) Run the OHC combine jobs to process the Day1 data
5) Repeat the same process for more days of hold.
4.5 External
Refer to the following steps pertaining to the recovery process for External files data:
1) Restore the Day 1 file from the archive directory using the restore process
2) Retrieve parm_date from the backup filename
3) Update the parm_dt to the parameter file to get correct LOAD_DT for all the records
4) Run the External jobs to process the Day1 data
5) Repeat the same process for more days of hold.
4.6 XXXXXnet
Refer to the following steps pertaining to the recovery process for XXXXXnet:
1) Restore the Day 1 file from the archive directory using the restore process
2) Retrieve parm_date from the backup filename
3) Update the parm_dt to the parameter file to get correct LOAD_DT for all the records.
4) Run the OHC combine jobs to process the Day1 data.
5) Repeat the same process for more days of hold.