Open Connector Framework (OCF)¶

The open connector framework (OCF), as the name suggests, is an open framework for supporting connectors. Connector provide client-side access to remote digital resources such as data sets, APIs and software components. OCF connectors also provide access to metadata about the resource (assets) and may call the Audit Log Framework to log audit messages.

Benefits¶

Applications and tools benefit from using OCF connectors because:

• Network and security parameters for accessing the data resources are managed in the metadata repository as part of a named connection. The application need only supply the identifier of the connection and provided they have the appropriate security credentials then a connector is returned to them for use.
  • There is no need to hard-code user ids and passwords in the application code - nor manage key stores for this sensitive information since the metadata repository handles this.
  • If the location of the data changes, then the named connection configuration is changed in the metadata repository and the application will be connected to the new location the next time they request a connector.
• The OCF connector provides two sets of APIs. The first set provides access to the resource contents and the second set provides access to the properties of the resource stored in the open metadata repositories. This provides applications and tools with a simple mechanism to make use of metadata as they process the resource. It is particularly useful for data science tools where these properties can help guide the end user in the use of the resource.
• OCF connectors are not limited to representing resources as they are physically implemented. An OCF connector can represent a simplified logical (virtual) data resource that is designed for the needs of a specific application or tool. This type of connector delegates the requests it receives to one or more physical data resources.

Organizations benefit from advocating the use of OCF connectors for their systems because the OCF connectors provide a consistent approach to resource access and audit logging. This is particularly important in data-rich environments where individuals are able to combine data from different resources creating new, potentially sensitive insight. The common approach to auditing, and the linkage between the data accessed and the metadata that describes its characteristics help to detect and prevent such actions.

Design rationale¶

The following factors influenced the design of the OCF.

• There are many existing connectors and connector frameworks in the industry today. It is important that these existing connectors can be incorporated into the OCF. Thus, the OCF includes placeholders for adapters to external connector providers and connectors.
• Application developers will only adopt a connector framework if it is easy to use. Thus, the connector interfaces allow for the use of native data APIs to minimize the effort an application developer has to take in order to use the OCF connectors.
• Access to the all properties known about an asset should be available to the consumers of the corresponding resource. Therefore, the OCF provides a standard interface for accessing these properties. Different providers of these properties can plug into the OCF. Egeria provides an implementation of this interface to supply asset properties stored in open metadata repositories in the OCF metadata management common service.

Terminology¶

There are a number of key components within the OCF:

Connector¶

A connector is a Java client object that provides applications with access to a data source or service (known as an resource) along with its related metadata.

An OCF connector provides four APIs:

OCF connectors are not limited to representing assets as they are physically implemented. An OCF connector can represent a simplified logical (virtual) asset, such as a data set, that is designed for the needs of a specific application or tool. This type of connector delegates the requests it receives to one or more physical data resources. It is called a virtual connector.

Connection¶

The connection provides the set of properties needed to create and initialize an instance of a connector.

A connection contains properties about the specific use of the connector, such as user Id and password, or parameters that control the scope or resources that should be made available to the connector.

It links to an optional:

• Connector type that describes the type of the connector that needs to be created in order to access the asset.
• Endpoint that describes the server endpoint where the asset is accessed from.

Connector types and endpoints can be reused in multiple connections.

Connections are typically managed in a metadata repository, but they can also be manually populated.

Connection implementations¶

The OCF offers two implementations of the connection:

• Connection is a bean implementation of the connection used in REST API requests and events. It allows properties to be set up and retrieved.
• ConnectionProperties is a read-only wrapper for the connection properties that is used in client interfaces that do not allow the properties to be updated.

Connection properties¶

The properties for a connection are defined in model 0201 and include:

Using Connections from open metadata repositories¶

Each connection stored in a metadata repository has a unique identifier. An application can request a connector instance through selected Egeria OMAS interfaces, such as the Asset Consumer OMAS, with just the unique identifier or name of a connection.

The OMAS retrieves the connection object from the open metadata repositories and passes it to the connector broker factory object. The connector broker (and underlying connector provider) uses the information from the connection object to create an instance of the connector.

The advantage of retrieving the connection information from a metadata repository is that the connection properties do not need to be hard-coded in the consuming applications and the metadata associated with the linked asset can be retrieved via the connector's connected asset properties interface.

Connections can be created in the open metadata repositories through the following interfaces:

• Asset Owner OMAS
• Asset Manager OMAS
• Data Manager OMAS
• Database Integrator OMIS
• Files Integrator OMIS
• Governance Action OMES

Configuring connections¶

The administration guide describes how to configure Egeria's OMAG Server Platforms and servers. Both the platform and the servers use connectors for access to the external resources to support their basic operation and to coordinate metadata and governance with third party technologies. This means that the configuration includes connection definitions for these connectors.

All of these interfaces have Java clients that enable you to set up the connection using the OCF connection bean. However, if you want to use the REST API directly, then you need to specify the connection in JSON.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

{
  "connection" : 
  {
    "class" : "Connection",
    "connectorType" : 
    {
      "class" : "ConnectorType",
      "connectorProviderClassName" : "...fully qualified class name..."
    },
    "endpoint" : 
    {
      "class" : "Endpoint",
      "address" : "... network address of resource ..."
    }
  }
}

Connector broker¶

The connector broker is a generic factory class for all open connectors. Given a valid connection object, the connector broker is able to create a new instance of a connector. This means the caller does not need to know the implementation details of the connector - just its interface.

It is implemented in the ConnectorBroker class, and is used as follows:

1
2
3
4
5
6
7

import org.odpi.openmetadata.frameworks.connectors.Connector;
import org.odpi.openmetadata.frameworks.connectors.ConnectorBroker;

// ...

ConnectorBroker   connectorBroker     = new ConnectorBroker();
Connector         connector           = connectorBroker.getConnector(connection);

When the connector instance is requested, the connector broker uses the connector type properties from the supplied connection to identify the appropriate connector provider. The connector broker delegates the connector instance request to the connector provider and returns the result to its caller.

The connector broker is used in the client code of the Open Metadata Access Services (OMAS) that provide connector instances to their consumers, for example:

• Asset Consumer OMAS
• Asset Owner OMAS

Connector type¶

The connector type is a set of properties that defines the supported capabilities and the identity of the connector provider for a connector. Its properties are:

The connector type is linked to the connection objects that request this type of connector.

Connector provider¶

A connector provider is the factory for a particular type of connector. It is typically called from the connector broker, although it may be called directly.

Each connector provider implements the ConnectorProvider interface.

It has two types of methods:

• Return the connector type object that is added to a connection object used to hold the properties needed to create an instance of the connector.
• Return a new instance of the connector based on the properties in a connection object. The connection object that has all the properties needed to create and configure the instance of the connector.

The ConnectorProviderBase class provides much of the implementation for a connector provider.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

/**
 * XXXStoreProvider is the OCF connector provider for the XXX store connector.
 */
public class XXXStoreProvider extends ConnectorProviderBase
{
    static final String  connectorTypeGUID = "Add unique GUID here";
    static final String  connectorTypeName = "XXX Store Connector";
    static final String  connectorTypeDescription = "Connector supports ... add details here";

    /**
     * Constructor used to initialize the ConnectorProviderBase with the Java class name of the specific
     * store implementation.
     */
    public BasicFileStoreProvider()
    {
        String connectorClassName = "packagename.XXXStoreConnector";

        super.setConnectorClassName(connectorClassName);

        ConnectorType connectorType = new ConnectorType();
        connectorType.setType(ConnectorType.getConnectorTypeType());
        connectorType.setGUID(connectorTypeGUID);
        connectorType.setQualifiedName(connectorTypeName);
        connectorType.setDisplayName(connectorTypeName);
        connectorType.setDescription(connectorTypeDescription);
        connectorType.setConnectorProviderClassName(this.getClass().getName());

        super.connectorTypeBean = connectorType;
    }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

/* SPDX-License-Identifier: Apache-2.0 */
/* Copyright Contributors to the Egeria project. */

package org.odpi.openmetadata.adapters.connectors.datastore.basicfile;

import org.odpi.openmetadata.frameworks.connectors.ConnectorProviderBase;
import org.odpi.openmetadata.frameworks.connectors.properties.beans.ConnectorType;

/**
 * BasicFileStoreProvider is the OCF connector provider for the basic file store connector.
 */
public class BasicFileStoreProvider extends ConnectorProviderBase
{
    static final String  connectorTypeGUID = "ba213761-f5f5-4cf5-a95f-6150aef09e0b";
    static final String  connectorTypeName = "Basic File Store Connector";
    static final String  connectorTypeDescription = "Connector supports reading of Files.";

    /**
     * Constructor used to initialize the ConnectorProviderBase with the Java class name of the specific
     * store implementation.
     */
    public BasicFileStoreProvider()
    {
        String connectorClassName = "org.odpi.openmetadata.adapters.connectors.datastore.basicfile.BasicFileStoreConnector";

        super.setConnectorClassName(connectorClassName);

        ConnectorType connectorType = new ConnectorType();
        connectorType.setType(ConnectorType.getConnectorTypeType());
        connectorType.setGUID(connectorTypeGUID);
        connectorType.setQualifiedName(connectorTypeName);
        connectorType.setDisplayName(connectorTypeName);
        connectorType.setDescription(connectorTypeDescription);
        connectorType.setConnectorProviderClassName(this.getClass().getName());

        super.connectorTypeBean = connectorType;
    }
}

Connected asset properties¶

Connected asset properties are the properties known about an asset accessed through a connector, hosted by a metadata access server. These properties are presented at three levels:

Asset summary¶

AssetSummary holds asset properties that are used for displaying details of an asset in summary lists or hover text:

Asset detail¶

AssetDetail extends AssetSummary to provide all the properties directly related to this asset:

Asset universe¶

AssetUniverse extends AssetDetail, and adds information about the common open metadata entities related to this asset:

Implementation details¶

The connector broker does not have access to a metadata repository because the OCF is metadata repository neutral. When it creates a connector, the connected asset properties are null.

Egeria Open Metadata Access Services (OMAS) such as Asset Consumer OMAS and Asset Owner OMAS, include the connector broker in their clients and support APIs for managing connections and creating connectors.

Connectors created by the Egeria access services will include the connected asset properties object configured to retrieve metadata from the same open metadata repository where the OMAS is running.

The connected asset properties are retrieved from the open metadata repositories by OCF Metadata Management common services. It will use the same user id that was used to create the connector.

Endpoint¶

The endpoint is a set of properties that defines the network address and how to connect to it for a resource deployed in the digital landscape. Its properties are:

Types¶

Open metadata repositories are able to store information needed to use OCF connectors. Details of the types involved are as follows:

• Model 0040 defines the structure of an Endpoint.
• Model 0201 defines the structures for Connections and ConnectorTypes.
• Model 0205 defines the linkage between the connection and the connected asset.

Raise an issue or comment below