Integration Daemon¶

An integration daemon is an OMAG Server that provides metadata exchange services between third party technology and the open metadata ecosystem.

The integration daemon interacts with the open metadata ecosystem through Open Metadata Access Services (OMASs) running in a metadata access server.

Typically, an integration daemon is deployed close to the artifacts/resources/data it is synchronizing because it can generate a lot of network traffic when its services are running. Multiple integration daemons may connect to the same metadata access server to provide horizontal scalability.

Inside the integration daemon are the Open Metadata Integration Services (OMISs) that each focus on metadata exchange for a specific type of technology (databases, files, other catalogs etc). An OMIS is paired with a specific Open Metadata Access Service (OMAS) running in the metadata access server.

Integration connectors¶

The code that manages the specific APIs and formats of the third party technology is encapsulated in a special type of connector called an integration connector.

Integration connectors are long-running. The integration daemon periodically calls each running integration connector in turn. This is called refreshing the connector. The time period between refresh calls to an integration connector is controlled by the integration connector's configuration. During the refresh call, an integration connector checks for changes to both the open metadata and the third party metadata they are responsible for. They make updates to correct any discrepancies. Typically, metadata changes are flowing in one direction, but two-way synchronization is also possible.

An integration connector may also register to receive event notification changes from open metadata and/or the third party technology it is working with. This allows the metadata synchronization to occur immediately without waiting for the refresh. In this style, the refresh period is much longer and the refresh processing is used to check that no metadata changes have been missed.

The specific interface that the integration connector needs to implement is defined by the integration service it is using. This interface enables the integration service to pass a context object to the connector before it is started. The context enables the connector to register a listener with the associated access service's Out Topic to listen for open metadata change notifications, or to call its REST API. By default, the context uses the integration daemon's userId for requests to the access service which means that the metadata created by the integration connector will be credited to this user. If you want to use a different userId for metadata from each connector, the server's userId can be overridden in the connector's configuration.

The integration connectors that run inside the integration daemon can be configured directly into the integration daemon's configuration document, or managed dynamically as open metadata using integration groups. Integration connectors are able to modify the integration groups dynamically, and also initiate governance actions, create to dos and raise incident reports.

Each time the integration connector is called (refreshed) by the integration daemon, it produces an integration report listing the metadata elements it has created, updated or deleted.

Raise an issue or comment below