An Integration Daemon runs integration connectors that are exchanging metadata with third party technologies. For example, an integration connector may catalog files from a file system, or the tables and columns in a database. Alternatively, it may be gathering lineage information or distributing security information to LDAP.
Integration connectors are typically long-running. The integration daemon needs information about which integration connectors to run and the location of the Metadata Access Server that provides access to the open metadata repositories via the Open Metadata Access Services (OMASs).
The configuration document is built up using a series of administration calls:
The basic server properties are used in logging and events originating from the server. They help to document the purpose of the server (which helps with problem determination) and enable performance improvements by allowing the server to ignore activity or metadata that is not relevant to its operation.
Property
Description
localServerDescription
Description for the server. This is useful information for the administrator to understand the role of the server. The default value is null.
organizationName
Descriptive name for the organization that owns the local server/repository. This is useful when the open metadata repository cluster consists of metadata servers from different organizations, or different departments of an enterprise. The default value is null.
localServerUserId
UserId to use for server-initiated REST calls. The default is OMAGServer.
localServerPassword
Password to use for server-initiated REST calls. The default is null. This means that only the userId is sent in the HTTP header.
localServerURL
The URL of the platform where the server is to be deployed. It should be the value used by external services to call the server since its broadcast across an open metadata repository cohort and used when deploying the server's configuration document to the correct platform.
maxPageSize
The maximum page size that can be set on requests to the server. The default value is 1000. A value of zero means unlimited page size. Although supported, the zero value is not recommended because it provides no protection from a large request denial of service attack.
Typically, these values are set up in a single command.
setBasicServerProperties
Set up the basic server properties in a single request. If any values are left blank, they are cleared in the server configuration document.
StringadminUserId="garygeeke";StringserverName="active-metadata-server"StringadminPlatformURLRoot="https://127.0.0.1:9443";OMAGServerConfigurationClientconfigurationClient=newOMAGServerConfigurationClient(adminUserId,serverName,adminPlatformURLRoot);StringorganizationName="Coco Pharmaceuticals";StringserverDescription="This server supports the governance teams";StringserverUserId="cocomds2npa";StringserverPassword="secret";StringserverURLRoot="https://localhost:9443"intmaxPageSize=1000configurationClient.setBasicServerProperties(organizationName,serverDescription,serverUserId,serverPassword,serverURLRoot,maxPageSize);
admin_user_id="garygeeke"server_name="active-metadata-store"admin_platform_url_root="https://127.0.0.1:9443"config_client=CoreServerConfig(server_name,admin_platform_url_root,admin_user_id)local_server_description="This server supports the governance teams"organization_name="Coco Pharmaceuticals"local_server_url="https://127.0.0.1:9443"local_server_user_id="cocomds2npa"local_server_password="secret"max_page_size=1000config_client.set_basic_server_properties(local_server_description,organization_name,local_server_url,local_server_user_id,local_server_password,max_page_size)
POST {{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/server-properties
with a request body of:
{"localServerDescription":"This server supports the governance teams","organizationName":"Coco Pharmaceuticals","localServerURL":"https://localhost:9443","localServerUserId":"cocomds2npa","localServerPassword":"secret","maxPageSize":600}
Alternatively, you can set these properties one at a time.
setServerDescription
The server description should be set to something that describes the OMAG Server's role. It may be the name of a specific product that it is enabling, or a role in the metadata and governance landscape. Its purpose is to help administrators identify which server configuration they need to work with.
POST {{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/server-description
The description is passed in the request body as a text string.
setOrganizationName
The organization name may be the owning organization or you may use it to identify the department or team that is supported by this server.
POST {{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/organization-name?name="{{organizationName}}"
setServerUserId
The server's userId is used when processing requests that do not have an end user, such as receiving an event from a topic. The default value is OMAGServer. Ideally each server should have its own userId, so it is possible to restrict the resources that each server has access to and identify the origin of updates to the metadata elements.
POST {{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/server-user-id?id="{{serverUserId}}"
setServerPassword
If the password is specified, the userId and password combination are used to provide authentication information on each REST call made by the server.
POST {{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/server-user-password?password="{{serverUserPassword}}"
setServerURLRoot
Configure the targetPlatformURLRoot with the platform URL Root value of where the OMAG Server Platform will run.
This may not be the same as platformURLRoot if the configuration document will be deployed to a different OMAG Server Platform from the one used to maintain it.
POST {{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/server-url-root?url={{targetPlatformURLRoot}}
What is the difference between {{platformURLRoot}} and {{targetPlatformURLRoot}}?
The {{targetPlatformURLRoot}} gives the location of the OMAG Server Platform on which this configured service is intended to run, while the {{platformURLRoot}} gives the location of the OMAG Server Platform in which this configuration document is maintained.
They could be, but do not need to be, the same location.
setMaxPageSize
The maximum page size value sets an upper limit on the number of results that a caller can request on any paging REST API to this server. Setting maximum page size helps to prevent a denial of service attack that uses very large requests to overwhelm the server. A value of 0 means no limit, and leaves the server open to such attacks.
POST {{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/max-page-size?limit={{maxPageSize}}
An OMAG Server uses an event bus such as Apache Kafka to exchange events with other servers and tools.
Egeria manages the specific topic names and the event payloads; however, it needs to know where the event bus is deployed and any properties needed to configure it.
Since the event bus is used in multiple places, the configuration document allows you to set up the details of the event bus which are then incorporated into all the places where the event bus is needed.
Important sequencing information
You need to set up this information before configuring any of the following:
Using an event topic as the destination for the audit log.
The following command creates information about the event bus. This information is used on the subsequent configuration of the OMAG Server subsystems. It does not affect any subsystems that have already been configured in the configuration document and if the event bus is not needed, its values are ignored.
It is possible to add arbitrary name/value pairs as JSON in the request body. The correct properties to use are defined in the connector type for the event bus.
For example, when using Apache Kafka as your event bus you may want to configure properties that control the behavior of the consumer that receives events and the producer that sends events. This is a typical set of producer and consumer properties:
Egeria's audit log provides a configurable set of destinations for audit records and other diagnostic logging for an OMAG Server. Some destinations also support a query interface to allow an administrator to understand how the server is running.
Each audit log record has a severity that can be used to route it to one or more specific destinations. Therefore, when an audit log destination is configured, it is optionally supplied with a list of severities to filter the types of audit log records it should receive.
The audit log severities are as follows:
Severity
Description
Information
The server is providing information about its normal operation.
Event
An event was received from another member of the open metadata repository cohort.
Decision
A decision has been made related to the interaction of the local metadata repository and the rest of the cohort.
Action
An Action is required by the administrator. At a minimum, the situation needs to be investigated and if necessary, corrective action taken.
Error
An error occurred, possibly caused by an incompatibility between the local metadata repository and one of the remote repositories. The local repository may restrict some of the metadata interchange functions as a result.
Exception
An unexpected exception occurred. This means that the server needs some administration attention to correct configuration or fix a logic error because it is not operating as a proper peer in the open metadata repository cohort.
Security
Unauthorized access to a service or metadata instance has been attempted.
Startup
A new component is starting up.
Shutdown
An existing component is shutting down.
Asset
An auditable action relating to an asset has been taken.
Types
Activity is occurring that relates to the open metadata types in use by this server.
Cohort
The server is exchanging registration information about an open metadata repository cohort that it is connecting to.
Trace
This is additional information on the operation of the server that may be of assistance in debugging a problem. It is not normally logged to any destination, but can be added when needed.
PerfMon
This log record contains performance monitoring timing information for specific types of processing. It is not normally logged to any destination, but can be added when needed.
<Unknown>
Uninitialized Severity
The default audit log destination is the console audit log destination. This writes selected parts of each audit log record to "standard out" (stdout).
It is configured to receive log records of all severities except Activity, Event, Trace and PerfMon. It is added automatically to a server's configuration document when other sections are configured.
If the server is a development or test server, then the default audit log configuration is probably sufficient, and you should use the following command:
Note: Using this command overrides all previous audit log destinations configured for the server.
If this server is a production server then you will probably want to set up the audit log destinations explicitly. You can add multiple destinations and each one can be set up to receive different severities of audit log records.
There are various destinations that can be configured for the audit log:
Since the default audit log destination is also a console audit log destination, only use this option to add the Trace and PerfMon severities.
The body of the request should be a list of severities
If an empty list is passed as the request body then all severities are supported by the destination.
This writes full log records to the slf4j ecosystem. When configuring slf4j as destination you also need to specify audit log logger category via the application properties of the OMAG Server Platform. This is described in Connecting the OMAG Audit Log Framework section of the developer logging guide.
The configuration of the slf4j ecosystem determines it ultimate destination(s).
The body of the request should be a list of severities
If an empty list is passed as the request body then all severities are supported by the destination.
This sets up an audit log destination that is described though a connection. In this case, the connection is passed in the request body and the supported severities are supplied in the connection's configuration properties.
It is also possible to set up all the audit log destinations in one command as a list of connections. Using this option overrides all previous audit log destinations and so can be used as the update command. The list of connections is passed in the request body and the supported severities are supplied in each connection's configuration properties.
POST - add a list of connection-based audit log destinations
Security is configured for a specific OMAG Server by adding a connection for this connector to the server's configuration documentusing the following command.
This removes all authorization checking from the server.
Configuring the Integration Daemon Services
There are two approaches to configuring the Integration Daemon Services.
The Dynamic Integration Groups section points to lists of integration connectors stored in the open metadata repositories. The integration daemon monitors these lists to determine which integration connectors to run. It dynamically updates the connectors it is running based on the values in the open metadata repositories.
The Integration Services section provides details of the integration connectors that are statically configured into the integration daemon. They are organized by the integration services that each integration connector is using. These integration connectors will be automatically started with the integration daemon. Any changes to these integration connectors need to be made in the configuration document and then the integration daemon needs to be restarted for the changes to take effect.
Both approaches can be used in the same integration daemon instance.
Each integration service hosts one or more integration connectors. An integration connector is responsible for the exchange of metadata with a specific deployment of a third party technology. For example, the database integrator integration service supports integration connectors that work with relational databases (RDBMS). A deployment of this integration service in an integration daemon may host, say, two integration connectors each loading metadata from their own relational database server.
The descriptive information and operational status are filled out automatically by the administration services based on the integrationServiceURLMarker value that you supply. The other values are supplied on the configuration call.
Each integration service is configured with the network location of the metadata access point / metadata access store running the appropriate OMAS. There are a set of options that the integration service supports along with the list of configuration properties for the integration connectors that will be run in the integration service. The integration connector's configuration properties defines which connector implementation to use and how it should be operated.
connectorName sets up the name of the connector. This name is used for routing refresh calls to the connector as well as being used for diagnostics. Ideally it should be unique amongst the connectors for the integration service.
connectorUserId sets up the user id for the connector - if this is null, the integration daemon's userId is used on requests to the Open Metadata Access Service (OMAS).
connection sets up the connection for the integration connector.
metadataSourceQualifiedName sets up the qualified name of the metadata source for this integration connector. This is the qualified name of an appropriate software capability element stored in open metadata. This software capability is accessed via the partner OMAS and is used to control the origin information - known as metadata provenance in any metadata elements created by the integration connector. The default value is null, which means all metadata elements created by the integration connector will have local cohort provenance values and can be updated by other process in the open metadata ecosystem. If it is set up with a valid qualified name, the metadata elements will have external source provenance values and only the integration connector is able to update the values.
refreshTimeInterval sets up the number of minutes between each call to the connector to refresh the metadata. Zero means that refresh is only called at server start up and whenever the refresh REST API request is made to the integration daemon. If the refresh time interval is greater than 0 then additional calls to refresh are added spaced out by the refresh time interval.
usesBlockingCalls sets up whether the connector should be called using the engage() method to allow it to block on a listening call rather than the refresh() method.
permittedSynchronization is an optional property that defines the permitted directions of metadata flow between the third party technology and open metadata. If the integration connector attempts to flow metadata in a direction that is not permitted, it receives the UserNotAuthorizedException. The default for this value is set up automatically in the integration service's descriptive information so this value only needs to be set if it is necessary to restrict the behavior of the connector. These are the different values for this property and their effect:
TO_THIRD_PARTY - The third party technology is logically downstream of open metadata. This means the open metadata ecosystem is the originator and owner of the metadata being synchronized. Any updates detected in the third technology are overridden by the latest open metadata values.
FROM_THIRD_PARTY - The third party technology is logically upstream (the originator and owner of the metadata). Any updates made in open metadata are not passed to the third party technology and the third party technology is requested to refresh the open metadata version.
BOTH_DIRECTIONS - Metadata exchange is permitted in both directions. Synchronization is halted on a specific element if potentially clashing updates have occurred both in the third party technology and open metadata. Such conflicts are logged on the audit log and resolved through manual stewardship.
An integration group identifies a list of integration connectors that an integration daemon should run. These integration connectors are defined and maintained as open metadata elements stored in the open metadata repositories. Changes can be made to these integration connectors at any time using the Governance Server OMAS. The integration daemon is monitoring for changes and updates the integration connectors it is running in line with the changes it discovers. This approach allows changes to be made to the integration daemon's configuration without needing access to the administration commands, nor a restart of the integration daemon.
The OMASs needed in the metadata access server depend on the Open Metadata Integration Services (OMISs) in use by the integration connectors. If you are unsure of which integration services will be used by your integration connectors, configure all of the listed OMASs in the metadata access store.
Open Metadata Integration Service (OMIS)
OMIS Description
Open Metadata Access Service (OMAS) running in the metadata access server
Link to the Egeria solutions to see the integration daemon in action.
Link to the integration daemon services to understand how the integration daemon is implemented. It also describes the integration daemon's REST API calls to restart and refresh integration connectors; and to make temporary changes to an integration connector's configuration. These changes are only made in-memory and they are lost when the integration connector restarts.
