12/15/13

Managing ETL dependencies with SAP BusinessObjects Data Services (Part 3) | Clariba Blog
Support Portal C ontact Us C areers Login
Search Clariba...

About Us

Solutions

SAP Technology

BI Services

Customer Results

News

News Room
Latest News Clariba Blog Twitter SAP News Feed Clariba Connection

Clariba Blog
Welcome to the Clariba business intelligence blog. We invite you to read our posts related to SAP BusinessObjects, Data Management and other BI topics, written by our consultants. If you have questions or comments, please let us know. Clariba Job Opportunities
Clariba as a BI leader consulting company strives to constantly improve and adapt to the changing market. This is why we recruit talented individuals, with potential to help us go further. Simple Forecast Exercise with SAP Predictive Analysis

- Senior BI Consultants - Middle East

Categories
BI on Demand (2) Methodology (23) Business Intelligence

Not missing a hit in SAP BusinessObjects using HTTP traffic logs

Think big. Pump up your BI expertise and project management experience w ith big data and big projects.

- BI Consultants - Europe
Subscribe to our Blog LinkedIn Follow us on Twitter YouTube Channel

Cloud (2) Take your BI career SAP to the next level. Develop innovative solutions and gainComputing project management experience. Managing ETL dependencies with BusinessObjects Customer Success Story (1) Data Services (Part 3)

- BI Consultants - Middle East

Data Integrator (10)

Move up the BI ladder. Engage on end-to-end projects to provide ground-breaking BI(15) solutions. Data Services June 17th, 2013 / Pierre-Emmanuel Larrouturou Data Warehouse (8) Are you satisfied with the way you currently manage the dependencies in your

- BI Analysts - Europe Indexing (2) ETL? In Part 1 of this series, I talked about the features I expect from a Start flexing your BI muscles. Actively participate in the development of challenging projects(3) in Europe. Desktop Intelligence dependency management system, and what are the main possibilities offered ETL (13) (directly or indirectly) by Data Services. In Part 2, I proposed an architecture - BI Analyst - Middle East Events (13) (structure and expected behavior) for a dependency management system
Flex (1) inside Data Services. Now I will give you the implementation details, while a feedback on how it went in real life as well as possible improvements will Gartner Magic Quadrant (2) Read more ab out our career opportunities and sub mit your CV. come in part 4. IBM Websphere (1) So how do we implement this theoretical solution in Data Services? Lets start with creating the needed tables (syntax below is for Oracle, things might be slightly different for your DBMS). Create Tab le FLOW_DEPENDENCIES ( FLOW_NAME VARCHAR2(100) not Null, PREREQUISITE VARCHAR2(100) not Null, Constraint FLOW_DEP_PK Primary Key (FLOW_NAME, PREREQUISITE) Enab le); Create Tab le FLOW_STATUS ( JOB_KEY NUMBER(28, 0) not Null, FLOW_NAME VARCHAR2(100) not Null, STATUS VARCHAR2(50) not Null, Constraint FLOW_STATUS_PK Primary Key (JOB_KEY, FLOW_NAME) Enab le); Now what about the Data Services part? Assuming that we want to upgrade an existing ETL, we want the practical solution to be as quick as possible to implement, and to allow an easy maintenance. It implies that components should be as standardized as possible, with minimal to zero customization for each flow. Lets imagine the case where we have 3 flows (Work flows WF_A and WF_C, Data flow DF_B) that perform a series of operations, for example the update of the account dimension. They have the same prerequisites and can be considered as one unique flow (for the needs of our dependency management system). Industry Remarks (4) Information Design (2) Infoview (6) LCM (1) Linux (2) Microsoft SharePoint (2) Mobility (15) BI Mobile (2) Exxova MyBI (3) SAP Explorer (5) OLAP (1) Open Source / Linux (3) Oracle Database (7) Rapid Marts (4) SAP BusinessObjects (125) BI 4.0 (30) BI 4 Migration (4) BI Support (1) Crystal Reports (2) Metadata (1) SAP Predictive Analysis (1) SAP BusinessObjects Dashboards (18) SAP BW (4) SAP Dashboards (2) SAP HANA (6) Big Data (3) SAP Incident Management (1) SAP Lumira (1) SDK (2) Single Sign On (1) SMEs run SAP (6) Were going to create a workflow for managing the dependencies of these 3 flows. We will name it WFDM_Account (for Work flow Dependency Management Account dimension). Social Media (4) social media analytics (4) SQL Server (3) Strategy Management (9) StreamWork (1) Uncategorized (3) Universe Designer (7) Web Intelligence (18) Xcelsius (22) Inside, well put a conditional object which will execute the flow only if needed. XI Migration (4)
Start fueling your BI tank. Support BI projects w ith our most prominent customers in the Middle East.

Popular posts

www.clariba.com/blog/managing-etl-dependencies-with-sap-businessobjects-data-services-part-3/

1/4

12/15/13

Managing ETL dependencies with SAP BusinessObjects Data Services (Part 3) | Clariba Blog
Migration from BusinessObjects XI R2 to XI R3.1 Both Versions on the Same Machine: Disaster Recovery Installing Business Objects on Linux (Part 2): Managing Your

Before we get into the details of the script, its time we think about the variables well need:
$G_Job_Key is the unique identifier for this specific job. We assume it has been generated earlier in the job. $G_Previous_Job_Key is the unique identifier of a previous job. Its set manually, and only in case we want to re-start the ETL and execute just the flows which failed or did not run. In the example from the part 2 of this article, we would have run the ETL for the second time by setting $G_Previous_Job_Key = 1 (the Job key of the first run).

System Tips for Installing Xcelsius 2008 with Office 2010 SAP Business Objects XI 4.0 Release: Connecting to OLAP data sources Embedding Xcelsius dashboards in Dashboard Builder

In order to make the script easier, Ill use four custom functions. The names and purpose are below, the code will come later. Check_Flow($P_Flow_Name) is the main function. It returns Yes if the flow is allowed to run. Otherwise it returns No and inserts a row with the corresponding reason in the table FLOW_STATUS. Get_Flow_Status($P_Job_Key, $P_Flow_Name) returns the status for a given flow and a given job key. Insert_Flow_Status($P_Job_Key, $P_Flow_Name, $P_Status) inserts a new status row for a given flow and a given job key. Is_Prerequisite_OK($P_Job_Key, $P_Flow_Name) returns No if any prerequisite of a given flow for a given job run is missing, Yes otherwise. November 2013 October 2013 September 2013 August 2013

Archives
December 2013

Twitter Updates
RT @SAPMENA: Visit #SAPMENA stand @Gitex to learn about latest innovation and network with our partners - Stand 57, Hall 6 #sapgitex htt ... Final arrangements for #GITEX 2012 #Dubai! we are excited to be there next week with @EXXOVAworldwide, showcasing @MyBIMobile

Below is the code of the function Check_Flow($P_Flow_Name) If (($G_Previous_Job _Key is not Null) and (Get_Flow_Status($G_Previous_Job _Key, $P_Flow_Name) in (Success, Already run))) Begin Insert_Flow_Status($G_Job _Key, $P_Flow_Name, Already run); Return No; End Else b egin If (Is_Prerequisite_OK($G_Job _Key, $P_Flow_Name) = No) b egin Insert_Flow_Status($G_Job _Key, $P_Flow_Name, Missing Prerequisite); Return No; end Else Return Yes; End

Our lovely finance and admin manager Marije welcomed her firstborn into the world today! Welcome to the Clariba family baby Daniel! =)

The two functions Get_Flow_Status and Insert_Flow_Status are easy ones: Get_Flow_Status($P_Job_Key, $P_Flow_Name): If ($P_Job _Key is not null) Return sql(Datawarehouse, select STATUS from FLOW_STATUS where JOB_KEY = [$P_Job _Key] and FLOW_NAME = {$P_Flow_Name} ); Else Return Null; Insert_Flow_Status($P_Job_Key, $P_Flow_Name, $P_Status): Sql(Datawarehouse, insert into FLOW_STATUS values ([$P_Job _Key], {$P_Flow_Name}, {$P_Status}); Return 0; The function Is_Prerequisite_OK($P_Job_Key, $P_Flow_Name) is a bit more complicated because of the join. It compares the number of rows in the table FLOW_STATUS which have a Success/Already run status with the number of rows from the table FLOW_DEPENDENCIES. If (sql(Datawarehouse, select count(*) from FLOW_DEPENDENCIES D join FLOW_STATUS S on D.PREREQUISITE = S.FLOW_NAME

www.clariba.com/blog/managing-etl-dependencies-with-sap-businessobjects-data-services-part-3/

2/4

12/15/13

Managing ETL dependencies with SAP BusinessObjects Data Services (Part 3) | Clariba Blog
where S.JOB_KEY = [$P_Job _Key] and S.STATUS in (\Success\, \Already run\) and D.FLOW_NAME = {$P_Flow_Name}) = (sql(Datawarehouse, select count(*) from FLOW_DEPENDENCIES where FLOW_NAME = {$P_Flow_Name}))) Return Yes; Else Return No; We can now have a look at the content of the conditional object Run_flow. The if condition checks if the current workflow is allowed to run. If thats not the case, we dont need to do anything, so the Else part of the object is empty. But if the workflow is allowed to run, then we run it (here WF_A, DF_B and WF_C). In order to update the FLOW_STATUS table in case of success, we use a script Success which simply inserts a status line: Insert_Flow_Status($G_Job _Key, workflow_name(), Success); And in order to update the FLOW_STATUS table in case of failure, we use a trycatch structure with a simple script in the Catch object: Insert_Flow_Status($G_Job _Key, workflow_name(), Failure);

Thats it! Once youve tested that, I would recommend you to replicate the parent flow, rename it to WFDM_Template. You can now replicate the template as many times as needed and simply insert the needed work flows and data flows. The last step to really make the dependencies work is of course to insert all the flow/prerequisite couples in the table FLOW_DEPENDENCIES. They will be based on the names of the work flows replicated from WFDM_Template. In the last part of this article, Ill give you a feedback on how it was implemented in a customer project, as well as possible improvements. Until then, Im looking forward to your opinion on this solution. Does it look good? Do you have another solution? Leave a comment below. This entry was posted on Monday, June 17th, 2013 at 1:26 pm and is filed under Data Services, ETL, SAP BusinessObjects. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Share:

Leave a Reply
Name (required) Mail (will not be published) (required) Website

Submit Comment

CLARIBA BLOG

SAP BusinessObjects Web Intelligence report showing prompts with scientific notation

www.clariba.com/blog/managing-etl-dependencies-with-sap-businessobjects-data-services-part-3/

3/4

12/15/13

Managing ETL dependencies with SAP BusinessObjects Data Services (Part 3) | Clariba Blog
INTERNATIONAL OFFICES: Spain: Balmes 109, 4-1, Barcelona UAE: Unit 905, Sidra Tower, Dubai Qatar: Al Hitmi Village, Bldg#8, Doha

About Us
Our Team Our Customers Our Partner Strategy Careers Contact Us Resource Center

Solutions
By Industry By Function By Application Custom Developments

SAP Technology
Why SAP Business Intelligence SAP Data Services SAP Business Planning and Consolidation SAP Strategy Management SAP Information Steward SAP HANA BI Portal & SDK Clariba Support Portal

BI Services
360 BI Assessment BI Implementation BI Migration Data Management Education Support

Customer Results
Banking Education High Tech & Software Pharma & Healthcare Sports & Entertainment Telecommunications Distribution / Wholesale Oil & Gas

C opyright 2013 C lariba. All rights reserved.

www.clariba.com/blog/managing-etl-dependencies-with-sap-businessobjects-data-services-part-3/

4/4