Connector catalog¶

Egeria has a growing collection of connectors to third party technologies. These connectors help to accelerate the rollout of your open metadata ecosystem since they can be used to automate the extraction and distribution of metadata to the third party technologies.

A connector is a client to a third party technology. It supports a standard API that Egeria calls, and it then translates these calls into requests to the third party technology. Some connectors are also able to listen for notifications from the third party technology. When a notification is received, the connector converts its content into a call to Egeria to distribute the information to the open metadata ecosystem.

Connectors enable Egeria to operate in many environments and with many types of third party technologies, just by managing the configuration of the OMAG servers. The Connector Catalog list the connector implementations supplied by the Egeria community. There are three broad categories of connectors and the connector catalog is organized accordingly:

• Connectors that support the exchange and maintenance of metadata with third party technology. This includes the resource connectors, survey action connectors, integration connectors and adapter repository connectors. These connectors are organized by the type of third part technology type work with.

• Connectors that support the governance of open metadata. This includes the context event services and governance action services. These connectors are organized by function.

• Connectors that support the integration of Egeria’s runtimes into the IT infrastructure where it is running. This includes the native repository connectors, event bus connectors, cohort registry stores, configuration stores, audit log destination connectors, open metadata archive stores, REST client connectors and the cohort member remote repository connectors. These connectors are organized by connector type.

Metadata exchange and maintenance connectors¶

The connectors that support the exchange and maintenance of metadata help to accelerate the rollout of your open metadata ecosystem, since they can be used to automate the extraction and distribution of metadata to the third party technologies.

• Secrets Store connectors manage the retrieval of secrets (passwords, certificates, ...) from secured locations at runtime.
• File connectors work with different types of files.
• JDBC Database connectors make use of the JDBC standards to work with different types of relational databases.
• Apache Kafka connectors work with the topics and/or events passing through the Apache Kafka event broker.
• Apache Atlas connectors work with an Apache Atlas server.
• Strimzi connector works with the cloud-based Apache Kafka deployment called Strimzi.
• Open API Specification connectors extract metadata about APIs through the Open API interfaces provided through the Swagger API.
• Open Lineage Event connectors works with the open lineage event standard.

Secrets Stores¶

Secrets stores externalize secrets such as passwords, tokens and certificates so they do not need to be stored in either the configuration document or open metadata repositories.

• The Environment Variables Secret Store connector retrieves secret values from environment variables.
• The YAML File Secret Store connector retrieves secret values from environment variables.

Files¶

Files provide storage for many types of data. They are organizes into folders (also known as directories on some operating systems). Some connectors work with any type of file. Other connectors are able to understand the content of specific types of file formats and so these connectors are organized by file type.

Any type of File¶

• The Basic File Resource Connector provides support to read and write to a file using the Java File object.
• The Move/Copy File Provisioning Governance Action Service moves or copies files from one location to another and maintains the lineage of the action.

File Folders (Directories)¶

• The Basic Folder Resource Connector is for accessing the files within a folder (directory).
• The Data Files Monitor Integration Connector maintains a DataFile asset for each file in the directory (or any subdirectory). When a new file is created, a new DataFile asset is created. If a file is modified, the lastModified property of the corresponding DataFile asset is updated. When a file is deleted, its corresponding DataFile asset is also deleted (or archived if it is still needed for lineage).
• The Generic Folder Watchdog Governance Action Service listens for changing DataFile assets linked to a specified FileFolder element and initiates governance actions when specific events occur. This may be for files directly linked to the folder or located in sub-folders.

Data Folders¶

• The Data Folder Resource Connector is for accessing data that is stored as a number of files within a folder (directory).
• The Data folder Monitor Integration Connector maintains a DataFolder asset for the directory. The files and directories underneath it are assumed to be elements/records in the DataFolder asset and so each time there is a change to the files and directories under the monitored directory, it results in an update to the lastModified property of the corresponding DataFolder asset.

CSV Files¶

• The CSV File Resource Connector is able to retrieve data from a Comma Separated Values (CSV) file where the contents are stored in logical columns with a special character delimiter between the columns.

Open Metadata Archive Files¶

• The File-based Open Metadata Archive Store Connector stores an open metadata archive as a plain text JSON file.

Relational Databases¶

• The JDBC Resource Connector is for accessing a database via the JDBC DataSource interface.
• The JDBC Integration Connector automatically maintains the open metadata instances on a database server via JDBC. This includes the database schemas, tables, columns, primary keys and foreign keys.

Unity Catalog¶

Unity Catalog is a data manager catalog that governs access to data. It is typically managing data from data lakes and lakehouses where much of the data is in a parquet format, with a table abstraction over the top. Unity catalog also supports folders of files (called Volumes) and functions. Unity catalog is able to provide access to the data in its catalogs, and run the functions.

The Unity Catalog connectors provide a suite of function that integrates a Unity Catalog server into the open metadata ecosystem.

• The Unity Catalog Resource Connector is a digital resource connector that acts as a Java client to the Unity Catalog Server REST API. It is used by the other Unity Catalog connectors.
• The Unity Catalog Server Synchronizer is an integration connector that exchanges details about the catalogs defined for a Unity Catalog Server.
• The Unity Catalog Inside Catalog Synchronizer is an integration connector that exchanges details about the resources (schemas, tables, volumes and functions) defined within a Catalog found in a Unity Catalog Server.
• The Unity Catalog Server Survey Service is a Survey Action Service that surveys the resources defined in a Unity Catalog Server.
• The Unity Catalog Inside Catalog Survey Service is a Survey Action Service that surveys the resources defined in a Catalog inside a Unity Catalog Server.
• The Unity Catalog Inside Schema Survey Service is a Survey Action Service that surveys the resources defined in a Schema inside a Unity Catalog Server.

Apache Kafka¶

• The Kafka Open Metadata Topic Connector implements a resource connector for a topic that exchanges Java Objects as JSON payloads across an Apache Kafka event bus. It is configured in the Egeria OMAG Servers through the Event Bus Configuration. This the connector that is used by default in the Egeria runtimes to exchange events (notifications between the OMAG Servers).
• The Kafka Monitor Topic integration connector automatically maintains the open metadata instances for the topics hosted on an Apache Kafka server .
• The Kafka Audit Topic integration connector validates that topics that are active in an Apache Kafka server are also catalogued in open metadata. Creates an audit log record for each topic that is not catalogued.
• The Sample Lineage Integration Connector is a sample connector to listen for lineage events (not in an open lineage format) and catalog the associated assets, processes, schema and lineage. |

Apache Atlas¶

Apache Atlas is a metadata catalog originally designed for the Hadoop ecosystem. It offers integration services called Hooks and Bridges to capture the schemas and data sets of data platforms such as Apache Hive, Apache HBase and Apache Hadoop Distributed File System (HDFS) along with the processes for creating and maintaining data sets on these platforms. The metadata descriptions of these data sets and processes are linked together using lineage relationships, allowing an understanding of how data is flowing through a Hadoop deployment. Apache Atlas also supports glossaries and a tagging system that can be used both in searches and to control access to data through Apache Ranger (using the TagSync integration).

In recent years, Apache Atlas has been embedded in popular data catalogs such as Microsoft Purview and Atlan increasing the interest in being able to integrate with this metadata catalog.

The Apache Atlas connectors provide a suite of function that integrates an Apache Atlas server into the open metadata ecosystem.

• Apache Atlas REST Connector is a digital resource connector that acts as a Java client to the Apache Atlas Server REST API. It is used by the other Apache Atlas connectors.
• Apache Atlas Survey Action Service reviews the types and instances stored in an Apache Atlas Server and creates a survey report. This connector helps to provide insight into the content of the Apache Atlas server to determine if it contains valuable metadata that should be integrated into the open metadata ecosystem. This connector can also be configured to create a graph schema for the server that describes is supported types and how they link together.
• Apache Atlas Integration Connector automatically catalogues the content of an Apache Atlas Server into the open metadata ecosystem. It may also be configured to push selected open metadata into the Apache Atlas Server, such as glossary terms, tags and classifications.

Strimzi¶

• Strimzi Monitor topic integration connector automatically maintains the open metadata instances for the topics hosted in Strimzi .

Open API Specification¶

The Open API Monitor integration connector automatically maintains the open metadata instances for the APIs extracted from the Open API Specification extracted from an application.

Open Lineage Events¶

The open lineage connectors work with the Open Lineage standard

• Open Lineage Event Receiver integration connector receives open lineage events from an event topic and publish them to lineage integration connectors with listeners registered in the same instance of the Lineage Integrator OMIS.
• Governance Action to Open Lineage integration connector listens for governance actions executing in the open metadata ecosystem, generate open lineage events for them and publish them to the integration connectors running in the same instance of Lineage Integrator OMIS that are listening for OpenLineage events.
• API-based Open Lineage Log Store integration connector calls an OpenLineage compliant API to store the open lineage events that are passed to it through the OpenLineage listener that is registered with the Lineage Integrator OMIS.
• File-based Open Lineage Log Store integration connector stores the open lineage events that are passed to it through the OpenLineage listener that is registered with the Lineage Integrator OMIS. Each OpenLineage event is stored in its own file in JSON format. These files are organized according to the namespace and job name in the event.
• Open Lineage Cataloguer integration connector registers an Open Lineage listener with the Lineage Integrator OMIS and to catalog any processes that are not already known to the open metadata ecosystem.

Open Metadata Governance Connectors¶

• The Generic Element Watchdog Governance Action Service listens for changing metadata elements and initiates governance action processes when certain events occur.
• The Origin Seeker Governance Action Service walks backwards through the lineage relationships to discover the origin of the data asset.
• The Zone Publisher Governance Action Service updates the zone membership on the target action assets.

Runtime connectors¶

Runtime connectors enable Egeria's OMAG Server Platform and its hosted OMAG Servers to operate in many environments by providing plug-in points for the runtime services it needs to operate. Most of the runtime connectors relate to persistent storage, or connections to distributed services. The connectors are organized by type to allow you to choose the options available from the Egeria community.

Repository and Event Mapper Connectors¶

The repository connector, and its optional event mapper connector provide the ability to integrate a metadata repository into the open metadata ecosystem. These connector have direct access to the connected open metadata repository cohorts. There are two patterns of use for these connectors.

In the first pattern, called the native repository connector, the repository connector delegates all of its methods to a particular type of persistence store. Metadata is only accessible through the Egeria APIs and it is stored as entities, relationships and classifications enabling it to support any valid type of open metadata. This type of repository connector runs as the local repository within an Egeria Metadata Access Store server.

Repository connector supporting a native open metadata repository

In the second pattern, called the adapter repository connector, the repository connector, and an optional event mapper connector, provide an adapter for a third party metadata repository so it can be a part of the open metadata ecosystem. These connectors run in a Repository Proxy server.

Repository connector and optional event mapper connector supporting an adapter to a third party metadata repository

The table below lists the repository connectors supporting the native open metadata repositories.

The table below lists the repository connectors that act as an adapter for third party metadata repositories.

Platform Metadata Security Connectors¶

The platform metadata security connector provides authorization support for requests to the OMAG Server Platform. There is one platform metadata security connector defined for each OMAG Server Platform.

There is one implementation of the platform metadata security connector provided by Egeria. It is a sample that encodes information from the Coco Pharmaceutical scenarios.

• Coco Pharmaceuticals Platform Metadata Security Connector

Server Metadata Security Connectors¶

The server metadata security connector provides authorization support for requests to an OMAG Server. There is one server metadata security connector configured for each OMAG Server.

This connector is called each time a request is made to the server. It is called to:

• Validate the user has access to the server.
• Validate the user has access to the called operation.
• Select which zones to associate with the requests based on the user.
• If the request involves an asset, then does the user have access to the asset and which properties are they allowed to see.
• If the request is for a connection for an asset, then select which one linked to the asset should be returned to the user.
• Determine which events to sent to the cohort.
• Determine what actions can be made on the metadata repository.

There is one implementation of the server metadata security connector provided by Egeria. It is a sample that encodes information from the Coco Pharmaceuticals scenarios.

• Coco Pharmaceuticals Server Metadata Security Connector

Configuration Document Store Connectors¶

The configuration store connectors contain the connector implementations that manage the storage of Configuration Documents for OMAG Servers.

There is one configuration document store connector defined for each OMAG Server Platform.

There are two implementations of the configuration document store connector provided by Egeria: one for an encrypted store (default) and the other for a plain text store.

• Encrypted File Configuration Store Connector stores each configuration document as an encrypted JSON file.

• File Configuration Store stores each configuration document as a clear text JSON file.

Cohort Registry Store Connectors¶

The cohort registry store maintains information about the servers registered in an open metadata repository cohort. It resides in each cohort member and represents that member's view of the cohort membership. It contains the registration information sent by this member and the responses received from the other members.

Inside the cohort registry store there is one local registration record describing the information sent to the other members of the cohort and a list of remote registration records received from the other members of the cohort.

A cohort registry store connector manages the persistence of the cohort registry store. Egeria uses a connector to allow different storage methods for different deployment environments. Each member may choose their own implementation of the cohort registry store connector.

Egeria provides a single implementation of a cohort registry store connector:

• Cohort Registry File Store Connector provides the means to store the cohort registry membership details as a JSON file.

Open Metadata Archive Store Connectors¶

The open archive store connector manages the storage of an Open Metadata Archive. It is use in a utility or archive service that is maintaining an open metadata archive, and it is called in an Metadata Access Store when it is loading metadata from the archive.

Egeria provides two implementations of the open metadata archive store connector:

• File-based Open Metadata Archive Store Connector stores an open metadata archive as a plain text JSON file.

• Directory-based Open Metadata Archive File Store Connector stores an open metadata archive in a directory structure where each type definition and metadata instance is stored in JSON format in its own file.

Audit Log Destination Connectors¶

An audit log destination connector provides support for a specific audit log destination. At least one audit log destination connector is configured in every OMAG Server's's configuration document and used by its audit log component when the server runs.

An audit log destination's purpose may be either to store, process or distribute audit log records to diagnostic systems. Its associated configuration controls which severities of audit log record it receives. The implementation for the audit log destination connector can make further choices about how each log record is processed.

Below are the connector implementations provided by Egeria

• Console Audit Log Connector writes selected parts of each audit log record to stdout.

• slf4j Audit Log Connector writes full log records to the slf4j ecosystem.

• File Audit Log Connector creates log records as JSON files in a shared directory.

• Event Topic Audit Log Connector sends each log record as an event on the supplied event topic.

REST Client Connectors¶

Egeria provides a single implementation for Spring.

• Spring REST Client Connector uses the Spring RESTClient to issue REST API calls.

This is embedded in Egeria's Java clients. See

• Egeria's Platform API clients.
• Egeria's OMAS clients.

Cohort Member Client Connectors¶

Members of an Open Metadata Repository Cohort provide the other cohort members with a Connection to a connector that supports the OMRSRepositoryConnector interface during the cohort registration process. This connector translates calls to retrieve and maintain metadata in the member's repository into remote calls to the real repository.

Egeria's Open Metadata Repository Services (OMRS) provides a default REST API implementation and a corresponding client:

• REST Cohort Client Connector supports remote calls to the OMRS REST API.

The connection for this connector is configured in the LocalRepositoryRemoteConnection property of the cohort member's Local Repository Configuration.

Raise an issue or comment below