Tuesday, 30 November 2021

Datasets and Lineage -

 Data files, Tables and queues are represented in the EME by Logical datasets (also called EME Datasets), which are placeholders that contain metadata - The location and record format of actual, physical data. They do not have contain data itself. (Physical data are not checked in into the EME). When a graph is checked in, The EME inspects all the file and table components in the graph, comparing them with the logical datasets that already exist in the EME datastore. If no corresponding dataset is found, The EME creates one, by default in of the following locations:

For a file component, in the location derived from the component URL.

For a table component, in the tables directory of the project, where the .dbc file is located.

In the EME datastore, you want to make sure that all data files are logically the same are represented by the same logical dataset, so lineage remain intact. you can design for and enable lineage in a number of ways.

Datasets Component - 

The dataset components represent records or act on records, as of follow - 

BLOCK_COMPRESSES LOOKUP TEMPLATE

INPUT FILE

INPUT TABLE

INTERMEDIATE FILE

LOOKUP FILE

LOOKUP TEMPLATE

OUTPUT FILE

OUTPUT TABLE

READ MULTIPLE FILES

WRITE MULTIPLE FILES




No comments:

Post a Comment

how to create dml dynamically in Ab-initio

 $[ begin let int i = 0; let string(int) complete_list = "emp_nm,billable,age,designation,location"; let string(int) file_content ...