Should I use a Reformat component with a lookup file or a join component?
There are situations where you cannot use a Reformat with a lookup file instead of a JOIN.Forex - We can not do a full outer join using a Reformat and a Lookup. However, the two methods offer very similar performance when both of the following are true :
* Reformat or a Join can use used.
* If a join is used, its non-driving input fits into the available memory.
When considering whether to use a JOIN or a lookup file, the flexibility of the graph is also a concern. when you use a lookup file, the file consumes address space and depending on whether this file grows over time, this could be a problem.
For an In-memory join, as much of the non-driving input as can fit in the memory specified by the max-core parameter is loaded into memory, and the rest lands to disk.
* For a JOIN, the memory issue can be addressed by changing the max-core limit, which if exceeded handled by writing files to disk, Although this can be costly, it makes the graph fairly Robust. Using a JOIN also makes it easy to read and understand what the graph is doing.
*If the memory limits are exceeding with a lookup file, the graph fails due to lack of memory. Generally, if the information from the file is being used in other parts of the graph, using a lookup file reduces the amount of memory required.
No comments:
Post a Comment