Skip to content
Technical preview

Technical preview function is in a state that it can be tried. The development is complete, there is documentation and there are samples, tutorials and hands-on labs as appropriate.

The community is looking for feedback on the function before declaring it stable. This feedback may result in changes to the external interfaces.

Lineage warehouse services

The lineage warehouse services support the behavior of the Lineage Warehouse and its main purpose to preserve and report lineage metadata.

Figure 1 - Services in Lineage Warehouse

Following sequence explains how different services interact in and out the Lineage Warehouse:

  1. Metadata instance events from the cohort are distributed to Metadata Access Server running Asset Lineage OMAS. This access service is always paired with Lineage Warehouse.

  2. Asset Lineage OMAS feeds the lineage warehouse by selecting and sending the instance types suitable for lineage. It retrieves asset related metadata and completes the asset context portion of the graph. Once ready, this lineage context graph structured as vertices and edges is sent on Lineage Output Topic for further processing and preservation.

  3. Lineage event listener consumes asset context events and uses technology connector such as Lineage Warehouse Janus Connector to persist these lineage elements building up the large lineage graph.

  4. Lineage jobs run tasks that get executed in server background to:

    • Poll for changes and request updates using REST API (AssetLineageUpdateJob); and
    • Scan the lineage graph and augment lineage paths for optimized querying (LineageGraphJob).
  5. Lineage query service provides REST API used to query asset lineage store for specific business views such as horizontal and vertical.

Further information

Raise an issue or comment below