Skip to content
Stable

This component is complete and can be used. The interfaces will be supported until the function is removed from the project via the deprecation process. There will be ongoing extensions to this function, but it will be done to ensure backward compatibility as far as possible. If there is a need to break backward compatibility, this will be discussed and reviewed in the community, with a documented timeline.

Open lineage services

The open lineage services support the behavior of the open lineage server and its main purpose to preserve and report lineage metadata.

Figure 1 - Services in Open Lineage Server

Following sequence explains how different services interact in and out the open lineage server:

  1. Metadata instance events from the cohort are distributed to Metadata Access Server running Asset Lineage OMAS. This access service is always paired with open lineage server.

  2. Asset Lineage OMAS feeds the lineage warehouse by selecting and sending the instance types suitable for lineage. It retrieves asset related metadata and completes the asset context portion of the graph. Once ready, this lineage context graph structured as vertices and edges is sent on Lineage Output Topic for further processing and preservation.

  3. Lineage event listener consumes asset context events and uses technology connector such as Lineage Janus Connector to persist these lineage elements building up the large lineage graph.

  4. Lineage jobs run tasks that get executed in server background to:

    • Poll for changes and request updates using REST API (AssetLineageUpdateJob); and
    • Scan the lineage graph and augment lineage paths for optimized querying (LineageGraphJob).

  5. Lineage query service provides REST API used to query asset lineage store for specific business views such as horizontal and vertical.

Further information

Raise an issue or comment below