Tuesday, 9 November 2021

Reformat with lookup file versus JOIN

 Should I use a Reformat component with a lookup file or a join component?

There are situations where you cannot use a Reformat with a lookup file instead of a JOIN.Forex - We can not do a full outer join using a Reformat and a Lookup. However, the two methods offer very similar performance when both of the following are true :

* Reformat or a Join can use used.

* If a join is used, its non-driving input fits into the available memory.


When considering whether to use a JOIN or a lookup file, the flexibility of the graph is also a concern. when you use a lookup file, the file consumes address space and depending on whether this file grows over time, this could be a problem.

For an In-memory join, as much of the non-driving input as can fit in the memory specified by the max-core parameter is loaded into memory, and the rest lands to disk.

* For a JOIN, the memory issue can be addressed by changing the max-core limit, which if exceeded handled by writing files to disk, Although this can be costly, it makes the graph fairly Robust. Using a JOIN also makes it easy to read and understand what the graph is doing.


*If the memory limits are exceeding with a lookup file, the graph fails due to lack of memory. Generally, if the information from the file is being used in other parts of the graph, using a lookup file reduces the amount of memory required.


No comments:

Post a Comment

how to create dml dynamically in Ab-initio

 $[ begin let int i = 0; let string(int) complete_list = "emp_nm,billable,age,designation,location"; let string(int) file_content ...