Skip to content

Connector catalog

Egeria has a growing collection of connectors to third party technologies. These connectors help to accelerate the rollout of your open metadata ecosystem since they can be used to automate the extraction and distribution of metadata to the third party technologies.

A connector is a client to a third party technology. It supports a standard API that Egeria calls, and it then translates these calls into requests to the third party technology. Some connectors are also able to listen for notifications from the third party technology. When a notification is received, the connector converts its content into a call to Egeria to distribute the information to the open metadata ecosystem.

Connectors enable Egeria to operate in many environments and with many types of third party technologies, just by managing the configuration of the OMAG servers. The Connector Catalog list the connector implementations supplied by the Egeria community. There are three broad categories of connectors and the connector catalog is organized accordingly:

  • Connectors that support the exchange and maintenance of metadata with third party technology. This includes the resource connectors, survey action connectors, integration connectors and adapter repository connectors. These connectors are organized by the type of third part technology type work with.

  • Connectors that support the governance of open metadata. This includes the context event services and governance action services. These connectors are organized by function.

  • Connectors that support the integration of Egeria’s runtimes into the IT infrastructure where it is running. This includes the native repository connectors, event bus connectors, cohort registry stores, configuration stores, audit log destination connectors, open metadata archive stores, REST client connectors and the cohort member remote repository connectors. These connectors are organized by connector type.

Metadata exchange and maintenance connectors

The connectors that support the exchange and maintenance of metadata help to accelerate the rollout of your open metadata ecosystem, since they can be used to automate the extraction and distribution of metadata to the third party technologies.

Secrets Stores

Files

Files provide storage for many types of data. They are organizes into folders (also known as directories on some operating systems). Some connectors work with any type of file. Other connectors are able to understand the content of specific types of file formats and so these connectors are organized by file type.

Any type of File

File Folders (Directories)

  • The Basic Folder Resource Connector is for accessing the files within a folder (directory).
  • The Data Files Monitor Integration Connector maintains a DataFile asset for each file in the directory (or any subdirectory). When a new file is created, a new DataFile asset is created. If a file is modified, the lastModified property of the corresponding DataFile asset is updated. When a file is deleted, its corresponding DataFile asset is also deleted (or archived if it is still needed for lineage).
  • The Generic Folder Watchdog Governance Action Service listens for changing DataFile assets linked to a specified FileFolder element and initiates governance actions when specific events occur. This may be for files directly linked to the folder or located in sub-folders.

Data Folders

  • The Data Folder Resource Connector is for accessing data that is stored as a number of files within a folder (directory).
  • The Data folder Monitor Integration Connector maintains a DataFolder asset for the directory. The files and directories underneath it are assumed to be elements/records in the DataFolder asset and so each time there is a change to the files and directories under the monitored directory, it results in an update to the lastModified property of the corresponding DataFolder asset.

CSV Files

  • The CSV File Resource Connector is able to retrieve data from a Comma Separated Values (CSV) file where the contents are stored in logical columns with a special character delimiter between the columns.

Open Metadata Archive Files

Relational Databases

  • The JDBC Resource Connector is for accessing a database via the JDBC DataSource interface.
  • The JDBC Integration Connector automatically maintains the open metadata instances on a database server via JDBC. This includes the database schemas, tables, columns, primary keys and foreign keys.

Apache Kafka

Apache Atlas

Apache Atlas is a metadata catalog originally designed for the Hadoop ecosystem. It offers integration services called Hooks and Bridges to capture the schemas and data sets of data platforms such as Apache Hive, Apache HBase and Apache Hadoop Distributed File System (HDFS) along with the processes for creating and maintaining data sets on these platforms. The metadata descriptions of these data sets and processes are linked together using lineage relationships, allowing an understanding of how data is flowing through a Hadoop deployment. Apache Atlas also supports glossaries and a tagging system that can be used both in searches and to control access to data through Apache Ranger (using the TagSync integration).

In recent years, Apache Atlas has been embedded in popular data catalogs such as Microsoft Purview and Atlan increasing the interest in being able to integrate with this metadata catalog.

The Apache Atlas connectors provide a suite of function that integrates an Apache Atlas server into the open metadata ecosystem.

  • Apache Atlas REST Connector is a digital resource connector that acts as a Java client to the Apache Atlas Server REST API. It is used by the other Apache Atlas connectors.
  • Apache Atlas Survey Action Service reviews the types and instances stored in an Apache Atlas Server and creates a survey report. This connector helps to provide insight into the content of the Apache Atlas server to determine if it contains valuable metadata that should be integrated into the open metadata ecosystem. This connector can also be configured to create a graph schema for the server that describes is supported types and how they link together.
  • Apache Atlas Integration Connector automatically catalogues the content of an Apache Atlas Server into the open metadata ecosystem. It may also be configured to push selected open metadata into the Apache Atlas Server, such as glossary terms, tags and classifications.

Strimzi

Open API Specification

The Open API Monitor integration connector automatically maintains the open metadata instances for the APIs extracted from the Open API Specification extracted from an application.

Open Lineage Events

The open lineage connectors work with the Open Lineage standard

  • Open Lineage Event Receiver integration connector receives open lineage events from an event topic and publish them to lineage integration connectors with listeners registered in the same instance of the Lineage Integrator OMIS.
  • Governance Action to Open Lineage integration connector listens for governance actions executing in the open metadata ecosystem, generate open lineage events for them and publish them to the integration connectors running in the same instance of Lineage Integrator OMIS that are listening for OpenLineage events.
  • API-based Open Lineage Log Store integration connector calls an OpenLineage compliant API to store the open lineage events that are passed to it through the OpenLineage listener that is registered with the Lineage Integrator OMIS.
  • File-based Open Lineage Log Store integration connector stores the open lineage events that are passed to it through the OpenLineage listener that is registered with the Lineage Integrator OMIS. Each OpenLineage event is stored in its own file in JSON format. These files are organized according to the namespace and job name in the event.
  • Open Lineage Cataloguer integration connector registers an Open Lineage listener with the Lineage Integrator OMIS and to catalog any processes that are not already known to the open metadata ecosystem.

Open Metadata Governance Connectors

Runtime connectors

Runtime connectors enable Egeria's OMAG Server Platform and its hosted OMAG Servers to operate in many environments by providing plug-in points for the runtime services it needs to operate. Most of the runtime connectors relate to persistent storage, or connections to distributed services. The connectors are organized by type to allow you to choose the options available from the Egeria community.

Type Description
Repository and Event Mapper connectors Integrate metadata repositories into the open metadata ecosystem so that they can interact with one or more open metadata repository cohorts.
Platform Metadata Security Connectors manage authorization requests for the OMAG Server Platform's services.
Server Metadata Security Connectors manage authorization requests for the OMAG Server's services.
Configuration Document Store Connectors manage the persistence and retrieval of configuration documents.
Cohort Registry Store Connectors store the open metadata repository cohort membership details in the cohort registry store.
Open Metadata Archive Store Connectors read and write open metadata archives.
Audit Log Destination Connectors support different destinations for audit log records.
REST Client Connectors issue REST API calls to Egeria's deployed platforms and third party technologies.
Cohort Member Client Connector supports repository service called to remote cohort members.
Open Metadata Topic Connectors send and receive events.

Repository and Event Mapper Connectors

The repository connector, and its optional event mapper connector provide the ability to integrate a metadata repository into the open metadata ecosystem. These connector have direct access to the connected open metadata repository cohorts. There are two patterns of use for these connectors.

In the first pattern, called the native repository connector, the repository connector delegates all of its methods to a particular type of persistence store. Metadata is only accessible through the Egeria APIs and it is stored as entities, relationships and classifications enabling it to support any valid type of open metadata. This type of repository connector runs as the local repository within an Egeria Metadata Access Store server.

Native open metadata repository

Repository connector supporting a native open metadata repository

In the second pattern, called the adapter repository connector, the repository connector, and an optional event mapper connector, provide an adapter for a third party metadata repository so it can be a part of the open metadata ecosystem. These connectors run in a Repository Proxy server.

Adapter repository connectors

Repository connector and optional event mapper connector supporting an adapter to a third party metadata repository

The table below lists the repository connectors supporting the native open metadata repositories.

Native Repository Connector Description
JanusGraph OMRS Repository Connector provides a native repository for a metadata server using JanusGraph as the backend.
XTDB OMRS Repository Connector provides a native repository for a metadata server that supports historical queries, using XTDB as the persistent store.
In-memory OMRS Repository Connector provides a simple native repository implementation that "stores" metadata in HashMaps within the JVM; it is used for testing, or for environments where metadata maintained in other repositories needs to be cached locally for performance/scalability reasons.
Read-only OMRS Repository Connector provides a native repository implementation that does not support the interfaces for create, update, delete; however, it does support the search interfaces and is able to cache metadata -- this means it can be loaded with open metadata archives to provide standard metadata definitions.

The table below lists the repository connectors that act as an adapter for third party metadata repositories.

Adapter Repository Connectors Description
IBM Information Governance Catalog (IGC) OMRS Repository Connector implements read-only connectivity to the metadata repository within the IBM InfoSphere Information Server suite
SAS Viya OMRS Repository Connector implements metadata exchange to the metadata repository within the SAS Viya Platform
Sample Repository proxy (adapter) using polling to access files implements metadata exchange to a file system using a polling pattern and an embedded OMRS repository.
HMS Repository proxy (adapter) using polling to access HMS Tables implements metadata exchange to a Hive metastore using a polling pattern and an embedded OMRS repository.
Further information relating to Repository and Event Mapper connectors

Platform Metadata Security Connectors

The platform metadata security connector provides authorization support for requests to the OMAG Server Platform. There is one platform metadata security connector defined for each OMAG Server Platform.

Platform Metadata Security Connector

There is one implementation of the platform metadata security connector provided by Egeria. It is a sample that encodes information from the Coco Pharmaceutical scenarios.

Further information relating to Platform Metadata Security Connectors

Server Metadata Security Connectors

The server metadata security connector provides authorization support for requests to an OMAG Server. There is one server metadata security connector configured for each OMAG Server.

Server Metadata Security Connector

This connector is called each time a request is made to the server. It is called to:

  • Validate the user has access to the server.
  • Validate the user has access to the called operation.
  • Select which zones to associate with the requests based on the user.
  • If the request involves an asset, then does the user have access to the asset and which properties are they allowed to see.
  • If the request is for a connection for an asset, then select which one linked to the asset should be returned to the user.
  • Determine which events to sent to the cohort.
  • Determine what actions can be made on the metadata repository.

Further information

There is one implementation of the server metadata security connector provided by Egeria. It is a sample that encodes information from the Coco Pharmaceuticals scenarios.

Further information relating to Server Metadata Security Connectors

Configuration Document Store Connectors

The configuration store connectors contain the connector implementations that manage the storage of Configuration Documents for OMAG Servers.

Configuration Document Store Connector

There is one configuration document store connector defined for each OMAG Server Platform.

There are two implementations of the configuration document store connector provided by Egeria: one for an encrypted store (default) and the other for a plain text store.

Further information relating to Configuration Document Store Connectors

Cohort Registry Store Connectors

The cohort registry store maintains information about the servers registered in an open metadata repository cohort. It resides in each cohort member and represents that member's view of the cohort membership. It contains the registration information sent by this member and the responses received from the other members.

Cohort registry store connector

Inside the cohort registry store there is one local registration record describing the information sent to the other members of the cohort and a list of remote registration records received from the other members of the cohort.

Internal structure for the information stored inside a single cohort registry store

A cohort registry store connector manages the persistence of the cohort registry store. Egeria uses a connector to allow different storage methods for different deployment environments. Each member may choose their own implementation of the cohort registry store connector.

Egeria provides a single implementation of a cohort registry store connector:

Further information relating to Cohort Registry Store Connectors

Open Metadata Archive Store Connectors

The open archive store connector manages the storage of an Open Metadata Archive. It is use in a utility or archive service that is maintaining an open metadata archive, and it is called in an Metadata Access Store when it is loading metadata from the archive.

Open Metadata Archive Store Connector

Egeria provides two implementations of the open metadata archive store connector:

Further information relating to Open Metadata Archive Store Connectors

Audit Log Destination Connectors

An audit log destination connector provides support for a specific audit log destination. At least one audit log destination connector is configured in every OMAG Server's's configuration document and used by its audit log component when the server runs.

Audit Log Destination Connector

An audit log destination's purpose may be either to store, process or distribute audit log records to diagnostic systems. Its associated configuration controls which severities of audit log record it receives. The implementation for the audit log destination connector can make further choices about how each log record is processed.

Below are the connector implementations provided by Egeria

Further information relating to Audit Log Destination Connectors

REST Client Connectors

Egeria makes extensive use of REST API calls for synchronous (request-response) communication with its own deployed platforms and third party technologies. The REST client connectors are used to issue the REST API calls.

REST Client Connector

Egeria provides a single implementation for Spring.

This is embedded in Egeria's Java clients. See

Cohort Member Client Connectors

Members of an Open Metadata Repository Cohort provide the other cohort members with a Connection to a connector that supports the OMRSRepositoryConnector interface during the cohort registration process. This connector translates calls to retrieve and maintain metadata in the member's repository into remote calls to the real repository.

Cohort Member Client Connector

Egeria's Open Metadata Repository Services (OMRS) provides a default REST API implementation and a corresponding client:

The connection for this connector is configured in the LocalRepositoryRemoteConnection property of the cohort member's Local Repository Configuration.

Governance Daemon Connectors

The governance daemon connectors contain specialist connectors for the governance servers that make active use of open metadata.

  • The Integration Connector supports the ongoing exchange of metadata between open metadata and a third party technology. It runs in an Integration Daemon.
  • The Governance Service is a connector that runs in a Governance Engine hosted on the Engine Host.
  • The Lineage Warehouse Connector provides access to a specialist repository for managing lineage information extracted from the open metadata ecosystem. It runs in a Lineage Warehouse.
  • The Data Engine Connector extracts metadata from a data engine. It has been designed around the needs of IBM's DataStage ETL Engine. It runs in the Data Engine Proxy.
Connector Description
Lineage Warehouse Janus Connector The Lineage Warehouse connectors provide plugins to the Lineage Warehouse that allow the Lineage Warehouse Services to connect with databases.

Raise an issue or comment below