Skip to content

Metadata Exchange

People are not good at repetitive admin tasks, particularly if these tasks are not the primary focus of their work. Therefore, your metadata catalog(s) are more complete and accurate when automation supports these mundane tasks. The automation needs to:

  • monitor for triggers that indicate that metadata needs to be updated,
  • drive the necessary changes to metadata and then
  • make those changes visible to downstream processing.

The Integration Daemon is an Egeria server that sits at the edge of the open metadata ecosystem, synchronizing metadata with third party tools. It is connected to a Metadata Access Server that provides the APIs and events to interact with the open metadata ecosystem.

Integration Daemon

The integration can be:

  • Triggered by an event from a third party technology that indicates that metadata needs to be updated in the open metadata ecosystem to make it consistent with the third party technology's configuration.

  • Triggered at regular intervals so that the consistency of the open metadata ecosystem with the third party technology can be verified and, where necessary, corrected.

  • Triggered by a change in the open metadata ecosystem indicating that changes need to be replicated to the third party technology.

To understand how Egeria's metadata exchange works, it is helpful to look at the patterns used to connect technologies together to exchange information.

The table below shows the four major integration patterns supported by technologies today.

Pattern Description
Closed Technology The technology is only accessible through a user interface.
Passive Open Technology The technology offers open APIs that can be called to configure and operate the technology.
Active Open Technology The technology provides active, on-going exchange of information with another technology that covers its operation and specific situations it has detected.
Integrated Technology The technology consumes the open APIs and events supported by another technology.

Integrated technology

Egeria does not provide any particular consideration for closed technology. However, it does offer a wide range of APIs and event services called Open Metadata Access Services (OMASs) to enable a third party technology to become an integrated technology, as shown in Figure 1. The OMASs run in a metadata access server.

Figure 1

Figure 1: Integrated technology can call the OMASs to consume the open metadata directly

Where a technology does not have direct support for open metadata, Egeria offers hosting of connectors that act as intermediaries to translate API calls and events between the third party technology's APIs and events and Egeria's.

Passive open technology

For passive open technology, Egeria will continuously poll the connector to allow it to repeatedly call the technology's API to determine if anything has changed and then pass any changes to Egeria. The connector is also able to register a listener for events from Egeria and issue calls to the third party technologies to keep it in synchronization with Egeria.

Figure 2 shows Egeria's Integration Daemon hosting connectors to passive open technologies. These connectors are called integration connectors. The integration daemon manages the metadata through an Egeria Metadata Access Server.

Figure 2

Figure 2: Integration connectors running in an integration daemon polling a third party technology

Active open technology

The support for active open technology is similar to passive open technology except that rather than polling for changes in the third party technology, the connector listens for the third party technology's events and translates them into calls to Egeria.

There are two choices for active open technology. Any third party technology can be called through an integration connector as above, or, specifically for third party metadata servers, it is possible to integrate them using a repository connector running in a repository proxy.

This is shown in Figure 3.

Figure 3

Figure 3: Using either the integration daemon or repository proxy to integrate active open technology into the open metadata ecosystem

The choice of which approach to use depends on the sophistication of the third party metadata server. Using the repository proxy connects the metadata server into the Open Metadata Repository Cohort (or Cohort for short). It will have access to all metadata exchanged through the cohort and will be called as part of federated queries across the cohort members. The connector interface is low-level but it does provide the greatest control over the metadata received and shared. It makes the metadata repository an integrated member of the open metadata ecosystem.

Using the integration daemon connects the third party metadata server on the edge of the open metadata ecosystem. A two-way exchange is possible, but its metadata must be stored in an Egeria metadata access store that is connected to the appropriate cohorts before its metadata is visible to other members of the cohort.

Next steps?

Ready to run Egeria?

There are a number of options available to you:

Running Egeria natively on your local machine

The Getting Started with Egeria blog provides a step-by-step guide to building, installing, configuring and running Egeria on your machine.

Running Egeria in IntelliJ

See Setting up IntelliJ to develop components for Egeria.

Running Egeria in Kubernetes

  • There is a base helm chart that includes a single platform with a metadata server. This provides a simple way to experiment with Egeria's APIs and User Interfaces. Link to instructions for starting the base helm chart.

  • There is a lab helm chart that provides a more complex multi-platform environment that is integrated with a variety of third party technologies. This chart is designed to support the Open Metadata Labs which provide a rich set of demonstrations of many features of Egeria. The Open Metadata Labs are a good option if you would like a guided tour of Egeria. Link to instructions for starting the lab helm chart.

Review the planning guide?

The Planning Guide describes how to plan the work to introduce Egeria into your organization.

Review the integration guide?

The Integration Guide describes how to integrate third party technology into the open metadata ecosystem.

Review the administration guide?

The Administration Guide describes how to configure OMAG Server Platforms and their servers.

Learn more about metadata governance?

The Metadata Governance Dojo is a self-guided course building on the concepts introduced here.

Raise an issue or comment below