Technical preview
Technical preview function is in a state that it can be tried. The development is complete, there is documentation and there are samples, tutorials and hands-on labs as appropriate.
The community is looking for feedback on the function before declaring it stable. This feedback may result in changes to the external interfaces.
Open Lineage Event Receiver Integration Connector¶
Connector details
- Connector Category: Integration Connector
- Hosting Service: Lineage Integrator OMIS
- Hosting Server: Integration Daemon
- Source Module: lineage-integration-connectors
- Jar File Name:
lineage-integration-connectors.jar
Overview¶
The OpenLineage Event Receiver integration connector receives open lineage events from an event topic and publishes them to the lineage integration connectors with OpenLineage listeners registered in the same instance of the Lineage Integrator OMIS.
Figure 1: Operation of the OpenLineage event receiver integration connector
Configuration¶
This connector uses the Lineage Integrator OMIS running in the Integration Daemon.
Its connection definition to use on the administration commands that configure the Lineage Integrator OMIS is a VirtualConnection with an embedded OpenMetadataTopicConnection.
Connection configuration
1 2 3 4 5 6 7 8 9 10 11 12 13 |
|
- Add the connection for the open metadata topic connector in the
embeddedConnections
section replacing {{topicConnection}}. This will have the topic name in the endpoint'snetworkAddress
. The example below shows the connection for the Kafka open metadata topic connector
Kafka topic connection configuration
```json linenums="1" hl_lines="11-40"
{
"connection" :
{
"class" : "VirtualConnection",
"qualifiedName" : "Egeria:IntegrationConnector:Lineage:OpenLineageEventReceiver Connection",
"connectorType" :
{
"class" : "ConnectorType",
"connectorProviderClassName" : "org.odpi.openmetadata.adapters.connectors.integration.openlineage.OpenLineageEventReceiverIntegrationProvider"
},
"embeddedConnections" : [
{
"class": "EmbeddedConnection",
"embeddedConnection :
{
"class": "Connection",
"qualifiedName": "Kafka Open Metadata Topic Connector",
"connectorType":
{
"class": "ConnectorType",
"connectorProviderClassName": "org.odpi.openmetadata.adapters.eventbus.topic.kafka.KafkaOpenMetadataTopicProvider"
},
"endpoint":
{
"class": "Endpoint",
"address": {{openLineageTopicName}}
},
"configurationProperties":
{
"producer":
{
"bootstrap.servers": {{kafkaEndpoint}}
},
"local.server.id": "{{localServerId}}",
"consumer":
{
"bootstrap.servers": {{kafkaEndpoint}}
}
}
}
}]
}
}
- Add the name of the topic in {{openLineageTopicName}}; the integration daemon's server id in {{localServerId}} and the enpoint for Apache Kafka (for example localhost:9092) in {{kafkaEndpoint}}.