Stable
This component is complete and can be used. The interfaces will be supported until the function is removed from the project via the deprecation process. There will be ongoing extensions to this function, but it will be done to ensure backward compatibility as far as possible. If there is a need to break backward compatibility, this will be discussed and reviewed in the community, with a documented timeline.
Lineage Modeling¶
The Open Integration Framework (OIF) supports the following services for its hosted integration connectors:
-
Registering a listener for OpenLineage events received either:
- through the integration daemon's
publish-open-lineage-eventREST API (also available through the Runtime Manager API), or - published by other integration connectors running in the same integration daemon.
- through the integration daemon's
-
Registering a listener for events from the open metadata ecosystem. These events notify the integration connector of any changes to open metadata elements that are visible to it. They can be used to drive the publishing of lineage information to third party technologies.
-
Publishing OpenLineage run, job and dataset events to other integration connectors running in the same integration daemon that have registered a listener for these events.
-
Cataloguing lineage information from a third party technology. Specifically this includes process structure, ports, schemas and the data assets they are working with. The lineage relationships themselves are maintained through the
LineageClientreturned by the integration context'sgetLineageClient()method.
The details of these calls are described in the integration context.
The open metadata types used to represent lineage are described in 0750 Data Passing, 0760 Business Lineage and 0770 Lineage Mapping.
Further information
- Lineage Management describes how to set up Egeria to capture, steward, preserve and use lineage.
- Building integration connectors describes how to write your own integration connectors.
Raise an issue or comment below