Informatica Tips and Tricks

Informatica Tips and Tricks Version 0.1

By Prajuna Padmanabhan

Wipro Technologies 10/03/2010

Page 1 of 8

Informatica Tips and Tricks

Table of contents

1. Introduction........................................................................................................................... 2 2. Points to Remember ............................................................................................................ 3 2.1. Mapping level generic information.................................................................................3 2.2. Session level generic information..................................................................................4 3. Transformation Tips.............................................................................................................. 5 3.1. oo!"up Transformation................................................................................................. 5 3.2. #oiner Transformation.................................................................................................... $ 3.3. Sorter Transformation.................................................................................................... $ 3.4. %nion Transformation.................................................................................................... $ 3.5. &ggregator Transformation............................................................................................ $ 3.$. Router Transformation................................................................................................... ' 4. S(stem level.......................................................................................................................... ' 5. &dd")n.................................................................................................................................. ' $. *onclusion ........................................................................................................................... +

1.

Introduction

Page 2 of 8

Informatica Tips and Tricks

This Document deals with the points to note to increase the performance in mapping/workflow/sessions; Transformation plays an important role in Data warehouse. Transformation is used when data is moved from source to destination. Depending upon criteria transformations are done. This would be useful for developers getting introduced to Informatica willing to know how the performance can be increased and the usage of various transformations in it. Also would have a clarification on the changes indulged in it. Performance can be improved by a variety of ways out of which some are mentioned here.

2.
2.1.

Points to Remember
Mapping level generic information 1. Relational data sources !inimi"e the connections to sources# targets and informatica server to improve session performance. !oving target database into server system may improve session performance. 2. If flat files are used as source# store the flat files on a machine that consists of informatica server. $etching the flat file data from the informatica server machine will be easier than fetching the data from the other location.

%. !inimi"e the number of flows while creating the mapping. &ource to the &tage and &tage to the Target !ultiple phases may also cause issues with session performance. If you use staging areas you force Informatica server to perform multiple data passes. '. &() *verride acts as a filter transformation in the mapping/session which enables to reduce the inclusion of another filter transformation in the mapping. It minimi"es the number of rows fetched from the source to the target. +. &() *verride enables to ,oin two or more tables acting as a ,oiner transformation using the various ,oins. -)eft outer# Right outer etc.. /. &() *verride pushes the transformation 0uery to database resources instead of performing in the Informatica &erver. 1. 2sage of *RD3R 45 or 6R*2P 45 clauses in the &ource 0ualifier 0uery will benefit from optimi"ation. 7. $ilter Transformation should be created nearer to the sources for proper cleansing of data after which it proceeds to flow to target. The ultimate goal of data cleansing is to improve the 0uality in data. 8. )ook9ups not using unconnected lookups when only returning one port and the connected lookups should only have the ports being returned or in the condition all other ports removed.

Page

of 8

Informatica Tips and Tricks

:;. !inimi"e the number of lookups used in the transformation. ::. 2sage of mapplets and worklets reduces the number of mappings and sessions getting created and would incorporate the running tasks as a single batch. In this case performance would increase. <e have to ensure that the dependency of the mappings and sessions using the mapplets and worklets are taken care while using the reusable components. :=. De9coupling the process of loading done in the mappings would increase session performance. :%. !apping Template <i"ards 3nables faster creation of !appings# creates a template for further decision on the mapping creation. Type : > Type % &lowly changing dimensions Remove Duplicates Incremental )oads ?ustom Templates :'. ?reate more re9usable transformations which will increase the productivity. :+. &et proper si"ing attributes for mapping and transformation ob,ects. :/. )imit the number of connected input/output ports used in the transformations so that the data stored in the Data ?ache is reduced. Avoid unnecessary links in the mapping. :1. 2se ltrim @ rtrim while loading the records from the flat file to the staging table itself. This will avoid adding them in the Aoins while going for subse0uent mappings. :7. Avoid using too many mappings in the single mapping.

2.2.

Session level generic information :. ?ommit interval9 Bave a proper commit interval depending upon the amount of data being loaded or Tune the commit points in the session to match the database performance setup. =. !ake use of 4ulk )oading when too much load work is in one Process $lowC else go for partitioning to reduce debugging and maintainability issues. Also make use of mapplets to segment mapping. %. Do not use constraint based load# if it is not re0uired.

Page ! of 8

Informatica Tips and Tricks

'. ?reate Reusable &chedulers for easier to 0uickly grab information from !etadata on which <orkflows run. +. Avoid transformation errors to improve the session performance. /. 5ou can improve session performance by setting the optimal location and si"e for the caches. 1. 5ou can increase the buffer memory allocation for sources and targets that re0uire additional memory blocks. If the Integration &ervice cannot allocate enough memory blocks to hold the data# it fails the session. 7. )imiting the number of connected input/output or output ports reduces the amount of data the transformations store in the data cache.

3.
3.1.

Transformation Tips
Loo !up Transformation

:. 4y default# when you create a lookup on a table# Power?enter gives you all the columns in the table# but be sure to delete the unwanted columns from the lookup as they affect the lookup cache very much. 5ou only need columns that are to be used in lookup condition and the ones that have to get returned from the lookup. =. $ind the eDecution plan of the &() override and see if you can add some indeDes or hints to the 0uery to make it fetch data faster. . Si"e of t#e source versus si"e of loo up: )et us say# you have :; rows in the source and one of the columns has to be checked against a big table -: million rows.. Then Powercenter builds the cache for the lookup table and then checks the :; source rows against the cache. It takes more time to build the cache of : million rows than going to the database :; times and lookup against the table directly. 2se uncached lookup instead of building the static cache. '. 2se a $ilter transformation before a )ookup transformation using a dynamic cache when you add a <B3R3 clause to the lookup &() override. This ensures the Power?enter &erver only inserts rows in the dynamic cache and target table that match the <B3R3 clause +. If a )ookup transformation specifies several conditions# you can improve lookup performance by placing all the conditions that use the e0uality operator first in the list of conditions that appear under the ?ondition tab. /. ?ache small lookup tables -?ache is stored in the Informatica server memory and over flowed data is stored on the disk in file format which will be

Page " of 8

Informatica Tips and Tricks

automatically deleted after the successful completion of the session run. If you want to store that data you have to use a persistent cache.. 1. The )ookup transformation will automatically order by the column that is being returned. To bypass or suppress the order by# put a E99E after the new order by clause. 3.2. $oiner Transformation :. <here ever possible perform the ,oins in a database. =. Aoin the sorted data to increase the ,oiner performance. . Designate the master source as the source with the fewer rows. '. Aoiner transformation may often decrease the session performance. 4ecause they must group data before processing it. To improve session performance in this case use sorted ports option. %.%. :. Sorter Transformation The Transformation that used the maDimum amount of memory/?ostly# if the session is not able to allocate the needed memory then the session will fail. Allocate enough memory to sort the data while using the sorter transformation. &pecify a different work directory for each partition in the sorter transformation.

=.

3.%. :.

&nion Transformation Please make a note that there is no option available in a 2nion transformation to handle duplicates. It does a F2GI*G A))F. There in returning Duplicates. If handling of duplicates is the re0uirement then &orter can be used. (ggregator Transformation :. The sorted input option has to be checked on in an aggregator transformation to enhance performance. =. In situations where sorted input cannot be supplied# we need to configure data cache and indeD cache at session/transformation level to allocate more space to support aggregation.

=. 3.'.

Page # of 8

Informatica Tips and Tricks

%. Aggregator transformation often slows performance because they must group data before processing it. 6roup by simple columns. '. If you can capture changes from the source that affect less than half the target# you can use Incremental Aggregation to optimi"e the performance of Aggregator transformations. 3.). Router Transformation 1. $outer is a group of %i&ters. Instead of using mu&tip&e fi&ters in a mapping' a router can be used. 2. Performance degrades on using a router as mu&tip&e checks in done on a&& the groups defined (ithin the router transformation to route the ro(s according&y to the respecti)e targets.

%.

S*stem level

1. ?heck on the network speed. =. RA!/!emory consumption also takes part in the performance and can cause degradation.

%. ?onfigure the system to use more ?P2Hs to improve performance.

'.

(dd!+n

?ommands for performing some transformation operations :. A)T I ? =. A)T I P %. A)T I J '. A)TI 2 +. A)TI < /. A)TI ? *. +,T- % 9 ?uts the Port 9 Pastes the Port 9 Jalidates the Port 3Dpression 9 To !ove a Port upwards 9 To !ove a Port downwards 9 Delete a Port . +dd a /e( %ie&d in a Transformation

Page * of 8

Informatica Tips and Tricks

).

,onclusion

Informatica is 3ngine driven 3T) tool mostly used in Data Integration and Data warehouse environment. A simplified tool in a 62I environment. &upported by 3T) developers# the responsibility is to ensure that the goals of the business with regards to data management are met. The developer should be able to design and implement a plan for all types of data movement or modification re0uested. This 3T) tool saves time but can be eDpensive but provides maintainability# ease of development with !erging/?leansing and &ampling done with a graphical view of the flow. They are more efficient than writing code because the code is built using drag and drop. These code files are organi"ed into mappings or packages. -?ode rarely has to be written and code errors are managed and fiDed in the debugger.. The 3T) ob,ects can be nested shared and reused. They can be manipulated and eDecuted using control structures ensuring accurate Data delivery at the *rgani"ation level with a little of performance tuning increasing the potential of the session so that the data retrieval takes place with a 0uicker eDecution time.

Page 8 of 8