Skip to content
Technical preview

Technical preview function is in a state that it can be tried. The development is complete, there is documentation and there are samples, tutorials and hands-on labs as appropriate.

The community is looking for feedback on the function before declaring it stable. This feedback may result in changes to the external interfaces.

Open Lineage Event Receiver Integration Connector

Connector details

Overview

The OpenLineage Event Receiver integration connector receives open lineage events from an event topic and publishes them to the lineage integration connectors with OpenLineage listeners registered in the same instance of the Lineage Integrator OMIS.

Figure 1

Figure 1: Operation of the OpenLineage event receiver integration connector

Configuration

This connector uses the Lineage Integrator OMIS running in the Integration Daemon.

Its connection definition to use on the administration commands that configure the Lineage Integrator OMIS is a VirtualConnection with an embedded OpenMetadataTopicConnection.

Connection configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
{
   "connection" : 
                { 
                    "class" : "VirtualConnection",
                    "qualifiedName" : "Egeria:IntegrationConnector:Lineage:OpenLineageEventReceiver Connection",
                    "connectorType" : 
                    {
                        "class" : "ConnectorType",
                        "connectorProviderClassName" : "org.odpi.openmetadata.adapters.connectors.integration.openlineage.OpenLineageEventReceiverIntegrationProvider"
                    },
                    "embeddedConnections" : [ {{topicConnection}} ]
                }
}
  • Add the connection for the open metadata topic connector in the embeddedConnections section replacing {{topicConnection}}. This will have the topic name in the endpoint's networkAddress. The example below shows the connection for the Kafka open metadata topic connector

Kafka topic connection configuration

```json linenums="1" hl_lines="11-40" { "connection" : { "class" : "VirtualConnection", "qualifiedName" : "Egeria:IntegrationConnector:Lineage:OpenLineageEventReceiver Connection", "connectorType" : { "class" : "ConnectorType", "connectorProviderClassName" : "org.odpi.openmetadata.adapters.connectors.integration.openlineage.OpenLineageEventReceiverIntegrationProvider" }, "embeddedConnections" : [ { "class": "EmbeddedConnection", "embeddedConnection : { "class": "Connection", "qualifiedName": "Kafka Open Metadata Topic Connector", "connectorType": { "class": "ConnectorType", "connectorProviderClassName": "org.odpi.openmetadata.adapters.eventbus.topic.kafka.KafkaOpenMetadataTopicProvider"
}, "endpoint": { "class": "Endpoint", "address": {{openLineageTopicName}} }, "configurationProperties": { "producer": { "bootstrap.servers": {{kafkaEndpoint}} }, "local.server.id": "{{localServerId}}", "consumer": { "bootstrap.servers": {{kafkaEndpoint}} } } } }] } }

  • Add the name of the topic in {{openLineageTopicName}}; the integration daemon's server id in {{localServerId}} and the enpoint for Apache Kafka (for example localhost:9092) in {{kafkaEndpoint}}.
Back to top