This component is complete and can be used. The interfaces will be supported until the function is removed from the project via the deprecation process. There will be ongoing extensions to this function, but it will be done to ensure backward compatibility as far as possible. If there is a need to break backward compatibility, this will be discussed and reviewed in the community, with a documented timeline.
Open Connector Framework (OCF)¶
The open connector framework (OCF), as the name suggests, is an open framework for supporting connectors. Connector provide client-side access to remote digital resources such as data sets, APIs and software components. OCF connectors also provide access to metadata about the resource (assets) and may call the Audit Log Framework to log audit messages.
Applications and tools benefit from using OCF connectors because:
- Network and security parameters for accessing the data resources are managed in the metadata repository as part of a named connection. The application need only supply the identifier of the connection and provided they have the appropriate security credentials then a connector is returned to them for use.
- There is no need to hard-code user ids and passwords in the application code - nor manage key stores for this sensitive information since the metadata repository handles this.
- If the location of the data changes, then the named connection configuration is changed in the metadata repository and the application will be connected to the new location the next time they request a connector.
- The OCF connector provides two sets of APIs. The first set provides access to the resource contents and the second set provides access to the properties of the resource stored in the open metadata repositories. This provides applications and tools with a simple mechanism to make use of metadata as they process the resource. It is particularly useful for data science tools where these properties can help guide the end user in the use of the resource.
- OCF connectors are not limited to representing resources as they are physically implemented. An OCF connector can represent a simplified logical (virtual) data resource that is designed for the needs of a specific application or tool. This type of connector delegates the requests it receives to one or more physical data resources.
Organizations benefit from advocating the use of OCF connectors for their systems because the OCF connectors provide a consistent approach to resource access and audit logging. This is particularly important in data-rich environments where individuals are able to combine data from different resources creating new, potentially sensitive insight. The common approach to auditing, and the linkage between the data accessed and the metadata that describes its characteristics help to detect and prevent such actions.
The following factors influenced the design of the OCF.
- There are many existing connectors and connector frameworks in the industry today. It is important that these existing connectors can be incorporated into the OCF. Thus, the OCF includes placeholders for adapters to external connector providers and connectors.
- Application developers will only adopt a connector framework if it is easy to use. Thus, the connector interfaces allow for the use of native data APIs to minimize the effort an application developer has to take in order to use the OCF connectors.
- Access to the all properties known about an asset should be available to the consumers of the corresponding resource. Therefore, the OCF provides a standard interface for accessing these properties. Different providers of these properties can plug into the OCF. Egeria provides an implementation of this interface to supply asset properties stored in open metadata repositories in the OCF metadata management common service.
There are a number of key components within the OCF:
A connector is a Java client object that provides applications with access to a data source or service (known as an resource) along with its related metadata.
An OCF connector provides four APIs:
|Connector lifecycle||Manages the lifecycle state of the connector and includes
|Metadata store initialization||If the connector is created by a metadata service then it adds a client to the metadata server called
|Specific initialization for the type of connector||Some types of connectors need additional initialization. These methods are called by the component creating the connector before the
|Asset content||This API is crafted to provide the most natural interface to the asset's contents. Therefore, the asset content API is typically different for each type of connector.|
OCF connectors are not limited to representing assets as they are physically implemented. An OCF connector can represent a simplified logical (virtual) asset, such as a data set, that is designed for the needs of a specific application or tool. This type of connector delegates the requests it receives to one or more physical data resources. It is called a virtual connector.
See the developer guide for information on writing connectors.
The connection provides the set of properties needed to create and initialize an instance of a connector.
A connection contains properties about the specific use of the connector, such as user Id and password, or parameters that control the scope or resources that should be made available to the connector.
It links to an optional:
- Connector type that describes the type of the connector that needs to be created in order to access the asset.
- Endpoint that describes the server endpoint where the asset is accessed from.
Connector types and endpoints can be reused in multiple connections.
Connections are typically managed in a metadata repository, but they can also be manually populated.
The OCF offers two implementations of the connection:
Connectionis a bean implementation of the connection used in REST API requests and events. It allows properties to be set up and retrieved.
ConnectionPropertiesis a read-only wrapper for the connection properties that is used in client interfaces that do not allow the properties to be updated.
The properties for a connection are defined in model 0201 and include:
||GUID for the connection.|
||URL of the connection definition in the metadata repository.|
||The official (unique) name for the connection. This is often defined by the IT systems management organization and should be used (when available) on audit logs and error messages. (Sourced from the
||A consumable name for the connection. Often a shortened form of the qualifiedName for use on user interfaces and messages. The displayName should be only be used for audit logs and error messages if the qualifiedName is not set.|
||Information about the TypeDef for the connection.|
||A full description of the connection covering details of the assets it connects to along with usage and version information.|
||Any additional properties associated with the connection.|
||Properties for configuring the connector.|
||Protected properties for secure log on by connector to back end server. These are protected properties that can only be retrieved by privileged connector code.|
||Name or URI of connecting user.|
||Password for the userId - needs decrypting by connector before use.|
||Password for userId - ready to use.|
||Properties that describe the connector type for the connector.|
||Properties that describe the server endpoint where the connector will retrieve the assets.|
Using Connections from open metadata repositories¶
Each connection stored in a metadata repository has a unique identifier. An application can request a connector instance through selected Egeria OMAS interfaces, such as the Asset Consumer OMAS, with just the unique identifier or name of a connection.
The OMAS retrieves the connection object from the open metadata repositories and passes it to the connector broker factory object. The connector broker (and underlying connector provider) uses the information from the connection object to create an instance of the connector.
The advantage of retrieving the connection information from a metadata repository is that the connection properties do not need to be hard-coded in the consuming applications and the metadata associated with the linked asset can be retrieved via the connector's connected asset properties interface.
Connections can be created in the open metadata repositories through the following interfaces:
- Asset Owner OMAS
- Asset Manager OMAS
- Data Manager OMAS
- Database Integrator OMIS
- Files Integrator OMIS
- Governance Action OMES
The administration guide describes how to configure Egeria's OMAG Server Platforms and servers. Both the platform and the servers use connectors for access to the external resources to support their basic operation and to coordinate metadata and governance with third party technologies. This means that the configuration includes connection definitions for these connectors.
All of these interfaces have Java clients that enable you to set up the connection using the OCF connection bean. However, if you want to use the REST API directly, then you need to specify the connection in JSON.
Example connection definition in JSON
Egeria's JSON structures map one-to-ene with the properties in the equivalent Java beans and also include
class property that gives the name of the class that it maps to. So a simple connection object
would look something like this in JSON:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
The connector broker is a generic factory class for all open connectors. Given a valid connection object, the connector broker is able to create a new instance of a connector. This means the caller does not need to know the implementation details of the connector - just its interface.
It is implemented in the
ConnectorBroker class, and is used as follows:
Example usage of a connector broker
1 2 3 4 5 6 7
When the connector instance is requested, the connector broker uses the connector type properties from the supplied connection to identify the appropriate connector provider. The connector broker delegates the connector instance request to the connector provider and returns the result to its caller.
The connector broker is used in the client code of the Open Metadata Access Services (OMAS) that provide connector instances to their consumers, for example:
||GUID for the connector type.|
||External link address for the connector type properties in the metadata repository.|
||The official (unique) name for the connector type. This is often defined by the IT systems management organization and should be used (when available) on audit logs and error messages.|
||A consumable name for the connector type. Often a shortened form of the qualifiedName for use on user interfaces and messages. The displayName should be only be used for audit logs and error messages if the qualifiedName is not set.|
||A full description of the connector type covering details of the assets it connects to along with usage and versioning information.|
||The connector provider is the factory for a particular type of connector. This property defines the class name for the connector provider that the connector broker should use to request new connector instances.|
||These are the connection's additional properties recognized by the connector implementation.|
||These are the connection's configuration properties recognized by the connector implementation.|
||These are the connection's secured properties recognized by the connector implementation.|
||Any additional properties that the connector provider needs to know in order to create connector instances.|
The connector type is linked to the connection objects that request this type of connector.
Each connector provider implements the
It has two types of methods:
- Return the connector type object that is added to a connection object used to hold the properties needed to create an instance of the connector.
- Return a new instance of the connector based on the properties in a connection object. The connection object that has all the properties needed to create and configure the instance of the connector.
ConnectorProviderBase class provides much of the implementation for a connector provider.
Example implementation of the connector provider for a simple connector
If you have a simple connector implementation then your connector provider follows the following template. It assumes the connector is for the
XXXStore and is called
With this base implementation, a specific connector provider implementation need only implement a constructor to configure the base class's function with details of itself and the Java class of the connector it needs.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Actual implementation of the connector provider for the basic file connector
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38
Connected asset properties¶
Connected asset properties are the properties known about an asset accessed through a connector, hosted by a metadata access server. These properties are presented at three levels:
AssetSummary holds asset properties that are used for displaying details of an asset in summary lists or hover text:
||metadata type information for the asset|
||GUID for the asset|
||external link for the asset|
||The official (unique) name for the asset. This is often defined by the IT systems management organization and should be used (when available) on audit logs and error messages. (Sourced from the
||A consumable name for the asset. Often a shortened form of the asset's qualifiedName for use on user interfaces and messages. The asset's displayName should be only be used for audit logs and error messages if the qualifiedName is not set. (Sourced from
||Short description about the asset. (Sourced from
||Full description of the asset. (Sourced from
||Name of the person or organization that owns the asset. (Sourced from the
||List of governance zones assigned to the asset. (Sourced from the
||Full list of the classifications assigned to the asset along with their properties.|
AssetSummary to provide all the properties directly related to this asset:
||List of identifiers for this asset that are used in other systems.|
||List of links to external media (images, audio, video) about this asset.|
||List of NoteLogs for this asset, often providing more detail on how to use the asset and its current status.|
||List of links to additional information about this asset.|
||List of connections defined to access this asset.|
||List of licenses associated with the asset.|
||List of certifications that have been awarded to this asset.|
AssetDetail, and adds information about the
common open metadata entities related to this asset:
||Glossary term(s) assigned to this asset.|
||Details of the schema type associated with the asset.|
||Details of the likes, reviews and comments, that are connected to the asset.|
||Details of the known locations of the asset.|
||Details of the lineage for the asset.|
||Details of the assets linked to this asset.|
The connector broker does not have access to a metadata repository because the OCF is metadata repository neutral. When it creates a connector, the connected asset properties are null.
Egeria Open Metadata Access Services (OMAS) such as Asset Consumer OMAS, Asset Owner OMAS and Discovery Engine OMAS, include the connector broker in their clients and support APIs for managing connections and creating connectors.
Connectors created by the Egeria access services will include the connected asset properties object configured to retrieve metadata from the same open metadata repository where the OMAS is running.
The connected asset properties are retrieved from the open metadata repositories by OCF Metadata Management common services. It will use the same user id that was used to create the connector.
The endpoint is a set of properties that defines the network address and how to connect to it for a resource deployed in the digital landscape. Its properties are:
||GUID for the endpoint.|
||External link address for the endpoint properties in the metadata repository. This URL can be stored as a property in another entity to create an explicit link to this endpoint.|
||The official (unique) name for the endpoint. This is often defined by the IT systems management organization and should be used (when available) on audit logs and error messages.|
||A consumable name for the endpoint. Often a shortened form of the qualifiedName for use on user interfaces and messages. The displayName should be only be used for audit logs and error messages if the qualifiedName is not set.|
||A description for the endpoint.|
||The location of the asset. For network connected resources, this is typically the URL and port number (if needed) for the server where the asset is located (or at least accessible by the connector). For file-based resources, this is typically the name of the file.|
||The communication protocol that the connection should use to connect to the server.|
||Describes the encryption method to use (if any). This is an open value allowing information needed by the connector user to retrieve all of the information they need to work with the endpoint.|
||Any additional properties that the connector need to know in order to access the resource.|
Open metadata repositories are able to store information needed to use OCF connectors. Details of the types involved are as follows:
- Model 0040 defines the structure of an
- Model 0201 defines the structures for
- Model 0205 defines the linkage between the connection and the connected asset.
- The OCF Metadata Management supports the retrieval of connection and connected asset properties from the open metadata repository/repositories.
- The Asset Consumer OMAS embeds the OCF to provide client-side support for connectors.
- The Open Metadata Repository Services (OMRS) make extensive use of OCF connectors for accessing open metadata repository servers and other resources. These connectors are collectively called the OMRS connectors.
- Many of the Open Metadata Governance Servers make use of OCF connectors to loosely-couple integration with a variety of underlying technologies.
- The developer guide provides more information on writing connectors for Egeria.
- The connector catalog lists the pre-built connectors supplied by Egeria.
Raise an issue or comment below