Creating digital resource connectors¶
To write an open connector you need to complete four steps:
- Identify the properties for the connection.
- Write the connector provider.
- Understand the interface that the connector needs to implement and the support provided by its base class.
- Write the connector itself.
All the code you write to implement these should exist in its own module, and as illustrated by the examples could even be in its own independent code repository. Their implementation will have dependencies on Egeria's:
- Open Connector Framework (OCF)
- Audit Log Framework (ALF)
- Specific interfaces used by the type of connector
No dependency on Egeria's OMAG Server Platform
Note that there is no dependency on Egeria's OMAG Server Platform for these specific connector implementations: they could run in another runtime that supported the connector APIs. In fact, even the Egeria interface modules should not be embedded in your jar file to allow your connector to run on any version of the OMAG Server Platform that supports your connector.
Identify connection properties¶
Begin by identifying and designing the properties needed to connect to your tool or system. These will commonly include a network address, protocol, and user credentials, but could also include other information that can be stored in the configuration properties of the connection.
Code the connector provider¶
Example: connector provider for IBM DataStage
For example, the
DataStageConnectorProvider is used to instantiate connectors to IBM DataStage data processing engines. Therefore, its name and description refer to DataStage, and the connectors it instantiates are
Example: connector provider for IBM Information Governance Catalog
IGCOMRSRepositoryConnectorProvider is used to instantiate connectors to IBM Information Governance Catalog (IGC) metadata repositories. In contrast to the
IGCOMRSRepositoryConnectorProvider's name and description refer to IGC, and the connectors it instantiates are
Connectors implement Egeria interfaces, not vice versa
Note that the code of all of these connector implementations exists outside Egeria itself (in separate code repositories), and there are no direct dependencies within Egeria on these external repositories or connectors.
All connectors can be configured with the network address and credential information needed to access the underlying tool or system. Therefore, you do not need to explicitly list properties for such basic details. However, the names of any additional configuration properties that may be useful to a specific type of connector can be described through the
recognizedConfigurationProperties of the connector type.
From the two examples (
IGCOMRSRepositoryConnectorProvider ), you will see that writing a connector provider follows a simple pattern:
- Extend a connector provider base class specific to your connector's interface.
static finalclass members for the GUID, name, description and the names of any additional configuration properties.
- Write a single public constructor, with no parameters, that:
super.setConnectorClassName()with the name of your connector class.
- Creates a new
ConnectorTypeobject, sets it characteristics to the
static finalclass members, and uses
.setConnectorProviderClassName()to set the name of the connector provider class itself.
- (Optional) Creates a list of additional configuration properties from the
static finalclass members, and uses
.setRecognizedConfigurationProperties()to add these to the connector type.
super.connectorTypeBean = connectorType.
Understand the connector interface¶
Now that you have the connector provider to instantiate your connector, you need to understand what your connector actually needs to do. For a service to use your connector, the connector must provide a set of methods that are relevant to that service.
Example: data engine proxy connector interface
For example, the data engine proxy services integrate metadata from data engines with Egeria. To integrate DataStage with Egeria, we want our
DataStageConnector to be used by the data engine proxy services. Therefore, the connector needs to extend
DataEngineConnectorBase , because this defines the methods needed by the data engine proxy services.
Example: OMRS repository connector interface
Likewise, we want our
IGCOMRSRepositoryConnector to integrate IGC with Egeria as a metadata repository. Therefore, the connector needs to extend
OMRSRepositoryConnector , because this defines the methods needed to integrate with Open Metadata Repository Services (OMRS).
How would you know to extend these base classes? The connector provider implementations in the previous step each extended a base class specific to the type of connector they provide (
OMRSRepositoryConnectorProviderBase ). These connector base classes (
OMRSRepositoryConnector ) are in the same package structure as those connector provider base classes.
In both cases, by extending the abstract classes (
OMRSRepositoryConnector ) your connector must implement the methods these abstract classes define. These general methods implement your services (data engine proxy services and OMRS), without needing to know anything about the underlying technology. Therefore, you can simply "plug-in" the underlying technology: any technology with a connector that implements these methods can run your service. Furthermore, each technology-specific connector can decide how best to implement those methods for itself.
Code the connector itself¶
Which brings you to writing the connector itself. Now that you understand the interface your connector must provide, you need to implement the methods defined by that interface.
Implement the connector by:
- Retrieving connection information provided by the configuration. The default method for
initialize()saves the connection object used to create the connector. If your connector needs to override the
initialize()method, it should call
super.initialize()to capture the connection properties for the base classes.
- Implementing the
start()method, where the main logic for your connector runs. Use the configuration details from the connection object to connect to your underlying technology. If the connector is long-running, this may be the time to start up a separate thread. However, this has to conform the rules laid down for the category of connector you are implementing.
- Using pre-existing, technology-specific clients and APIs to talk to your underlying technology.
- Translating the underlying technology's representation of information into the open metadata representation used by the connector interface itself.
For the first point, you can retrieve general connection information like:
- the server address and protocol, by first retrieving the embedded
- retrieving the protocol by calling
- retrieving the address by calling
- retrieving the protocol by calling
- the user Id, by calling
- the password, by calling either
ConnectionProperties, depending on what your underlying technology can handle
Use these details to connect to and authenticate against your underlying technology, even when it is running on a different system from the connector itself. Of course, check for
null objects (like the
EndpointProperties) as well before blindly operating on them.
Retrieve additional properties by:
ConnectionProperties, which returns a
Map<>with the name of each additional property of interest
Implementation of the remaining points (2-3) will vary widely depending on the specific technology being used. See the examples previously linked to delve deeper.
Raise an issue or comment below