Configuring a conformance test server¶
A Conformance Test Server is configured by creating a configuration document. Below is the outline structure of the server's configuration document.
Configuring the default values used in subsequent configuration commands
Set up the default event bus¶
An OMAG Server uses an event bus such as Apache Kafka to exchange events with other servers and tools.
Egeria manages the specific topic names and the event payloads; however, it needs to know where the event bus is deployed and any properties needed to configure it.
Since the event bus is used in multiple places, the configuration document allows you to set up the details of the event bus which are then incorporated into all the places where the event bus is needed.
Important sequencing information
You need to set up this information before configuring any of the following:
- Using an event topic as the destination for the audit log.
- Configuring the access services in a metadata access store or a metadata access point.
- Configuring registration to a cohort in a metadata access store, a metadata access point, a repository proxy or a conformance test server.
The following command creates information about the event bus. This information is used on the subsequent configuration of the OMAG Server subsystems. It does not affect any subsystems that have already been configured in the configuration document and if the event bus is not needed, its values are ignored.
It is possible to add arbitrary name/value pairs as JSON in the request body. The correct properties to use are defined in the connector type for the event bus.
Fine-grained helper command
POST - configure event bus
{{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/event-bus
Example: Apache Kafka
For example, when using Apache Kafka as your event bus you may want to configure properties that control the behavior of the consumer that receives events and the producer that sends events. This is a typical set of producer and consumer properties:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | |
A different type of event bus would use different properties.
Set the server URL root¶
Configure the targetPlatformURLRoot with the platform URL Root value of where the OMAG Server Platform will run.
This may not be the same as platformURLRoot if the configuration document will be deployed to a different OMAG Server Platform from the one used to maintain it.
POST - set server URL root
{{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/server-url-root?url={{targetPlatformURLRoot}}
What is the difference between {{platformURLRoot}} and {{targetPlatformURLRoot}}?
The {{targetPlatformURLRoot}} gives the location of the OMAG Server Platform on which this configured service is intended to run, while the {{platformURLRoot}} gives the location of the OMAG Server Platform in which this configuration document is maintained.
They could be, but do not need to be, the same location.
Configuring the basic properties
Configure the basic properties¶
The basic properties are used in logging and events originating from the server. They help to document the purpose of the server (which helps with problem determination) and enable performance improvements by allowing the server to ignore activity or metadata that is not relevant to its operation.
The basic properties include two unique identifiers that are set up when you first create the configuration document:
| Property | Description |
|---|---|
localServerId |
Unique identifier for this server. By default, this is initialized to a randomly generated Universal Unique identifier (UUID). |
localServerName |
Meaningful name for the server for use in messages and UIs. Ideally this value is unique to aid administrators in understanding the source of messages and events from the server. This value is set to the server name assigned when the configuration is created. |
The other basic properties have values that can be changed through the admin services API:
| Property | Description |
|---|---|
localServerType |
Descriptive type name for the server. Again this is useful information for the administrator to understand the role of the server. The default value is Open Metadata and Governance Server. |
organizationName |
Descriptive name for the organization that owns the local server/repository. This is useful when the open metadata repository cluster consists of metadata servers from different organizations, or different departments of an enterprise. The default value is null. |
localServerUserId |
UserId to use for server-initiated REST calls. The default is OMAGServer. |
localServerPassword |
Password to use for server-initiated REST calls. The default is null. This means that only the userId is sent in the HTTP header. |
maxPageSize |
The maximum page size that can be set on requests to the server. The default value is 1000. A value of zero means unlimited page size. Although supported, the zero value is not recommended because it provides no protection from a large request denial of service attack. |
The sections that follow cover how to set up these values.
Set server type name¶
The server type name should be set to something that describes the OMAG Server's role. It may be the name of a specific product that it is enabling, or a role in the metadata and governance landscape. The default value is Open Metadata and Governance Server.
If you have no specific value to set the server type name to, we recommend that you set the server type name to null. This will cause the server start up process will derive a standard server type name based on the rest of the configuration for the server.
POST - set server type
{{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/server-type?typeName="{{serverTypeName}}"
Set organization name¶
The organization name may be the owning organization or department or team supported by the server.
POST - set organization name
{{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/organization-name?name="{{organizationName}}"
Set the server's userId and optional password¶
The server's userId is used when processing requests that do not have an end user, such as receiving an event from a topic. The default value is OMAGServer. Ideally each server should have its own userId, so it is possible to restrict the resources that each server has access to and identify the origin of updates to the metadata elements.
If the password is specified as well, the userId and password combination are used to provide authentication information on each REST call made by the server.
POST - set server's userId
{{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/server-user-id?id="{{serverUserId}}"
POST - set server's password
{{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/server-user-password?password="{{serverUserPassword}}"
Set the maximum page size for REST API requests¶
The maximum page size value sets an upper limit on the number of results that a caller can request on any paging REST API to this server. Setting maximum page size helps to prevent a denial of service attack that uses very large requests to overwhelm the server. A value of 0 means no limit, and leaves the server open to such attacks.
POST - set maximum page size
{{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/max-page-size?limit={{maxPageSize}}
Configuring the audit log
Configure the audit log¶
Egeria's audit log provides a configurable set of destinations for audit records and other diagnostic logging for an OMAG Server. Some destinations also support a query interface to allow an administrator to understand how the server is running.
Each audit log record has a severity that can be used to route it to one or more specific destinations. Therefore, when an audit log destination is configured, it is optionally supplied with a list of severities to filter audit log records it should receive.
The audit log severities are as follows:
| Severity | Description |
|---|---|
Information |
The server is providing information about its normal operation. |
Event |
An event was received from another member of the open metadata repository cohort. |
Decision |
A decision has been made related to the interaction of the local metadata repository and the rest of the cohort. |
Action |
An Action is required by the administrator. At a minimum, the situation needs to be investigated and if necessary, corrective action taken. |
Error |
An error occurred, possibly caused by an incompatibility between the local metadata repository and one of the remote repositories. The local repository may restrict some of the metadata interchange functions as a result. |
Exception |
An unexpected exception occurred. This means that the server needs some administration attention to correct configuration or fix a logic error because it is not operating as a proper peer in the open metadata repository cohort. |
Security |
Unauthorized access to a service or metadata instance has been attempted. |
Startup |
A new component is starting up. |
Shutdown |
An existing component is shutting down. |
Asset |
An auditable action relating to an asset has been taken. |
Types |
Activity is occurring that relates to the open metadata types in use by this server. |
Cohort |
The server is exchanging registration information about an open metadata repository cohort that it is connecting to. |
Trace |
This is additional information on the operation of the server that may be of assistance in debugging a problem. It is not normally logged to any destination, but can be added when needed. |
PerfMon |
This log record contains performance monitoring timing information for specific types of processing. It is not normally logged to any destination, but can be added when needed. |
<Unknown> |
Uninitialized Severity |
The default audit log destination is the console audit log destination. This writes selected parts of each audit log record to "standard out" (stdout). It is configured to receive log records of all severities except Trace and PerfMon.
The default audit log destination is added automatically to Cohort Member servers when a repository, or cohort connection is configured.
For View Servers and Governance Servers, it is only added when other audit log destinations are configured. Therefore, is it necessary to issue at least one of the following configuration commands for the audit log for these types of servers.
Add audit log destinations¶
If the server is a development or test server, then the default audit log configuration is probably sufficient, and you should use the following command:
POST - set default audit log destination
{{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/audit-log-destinations/default
Note: Using this command overrides all previous audit log destinations configured for the server.
If this server is a production server then you will probably want to set up the audit log destinations explicitly. You can add multiple destinations and each one can be set up to receive different severities of audit log records.
There are various destinations that can be configured for the audit log:
Since the default audit log destination is also a console audit log destination, only use this option to add the Trace and PerfMon severities.
POST - add console audit log destination
{{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/audit-log-destinations/console
The body of the request should be a list of severities
If an empty list is passed as the request body then all severities are supported by the destination.
This destination writes JSON files in a shared directory. One file for each audit log record.
POST - add JSON file-based audit log destination
{{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/audit-log-destinations/files
The body of the request should be a list of severities
If an empty list is passed as the request body then all severities are supported by the destination.
This destination writes each log record as an event on the supplied event topic. It assumes that the event bus is set up first.
POST - add event-based audit log destination
{{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/audit-log-destinations/event-topic
The body of the request should be a list of severities
If an empty list is passed as the request body then all severities are supported by the destination.
This writes full log records to the slf4j ecosystem. When configuring slf4j as destination you also need to specify audit log logger category via the application properties of the OMAG Server Platform. This is described in Connecting the OMAG Audit Log Framework section of the developer logging guide.
The configuration of the slf4j ecosystem determines it ultimate destination(s).
POST - add slf4j audit log destination
{{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/audit-log-destinations/slf4j
The body of the request should be a list of severities
If an empty list is passed as the request body then all severities are supported by the destination.
This sets up an audit log destination that is described though a connection. In this case, the connection is passed in the request body and the supported severities are supplied in the connection's configuration properties.
POST - add connection-based audit log destination
{{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/audit-log-destinations/connection
It is also possible to set up all the audit log destinations in one command as a list of connections. Using this option overrides all previous audit log destinations and so can be used as the update command. The list of connections is passed in the request body and the supported severities are supplied in each connection's configuration properties.
POST - add a list of connection-based audit log destinations
{{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/audit-log-destinations
Retrieving audit log destinations¶
The configured list of audit log destinations can be retrieved using this command:
GET - the list of configured audit log destinations
{{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/audit-log-destinations
Updating audit log destinations¶
Audit log destinations can be updated individually, by qualified name using the following command:
POST - update connection-based audit log destination
{{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/audit-log-destinations/connection/{{qualifiedName}}
If you are not sure what the audit log connection is called, retrieve the list of configured audit log connections and the resulting list of audit log connections will include the qualified names.
Remove audit log destinations¶
The following will remove all audit log destinations, enabling you to add a new set of audit log destinations.
DELETE - clear all audit log destinations
{{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/audit-log-destinations
It is also possible to remove a single audit log destination using its connection's qualified name.
DELETE - clear then named audit log destination
{{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/audit-log-destinations/{{qualifiedName}}
Configuring the server security connector
Configure the server security connector¶
Metadata that is being aggregated from different sources is likely to need comprehensive access controls.
Egeria provides fine-grained security control for metadata access. It is implemented in a server metadata security connector that is called whenever requests are made for to the server.
Security is configured for a specific OMAG Server by adding a connection for this connector to the server's configuration documentusing the following command.
POST - configure security connector
{{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/security/connection
This passes in a connection used to create the server security connector in the request body.
{
"class": "Connection",
"connectorType": {
"class": "ConnectorType",
"connectorProviderClassName": "{fullyQualifiedJavaClassName}"
}
}
Example: set up the sample server security connector
For example, this is the connection that would set up the sample server security connector provided for the Coco Pharmaceuticals case study:
{
"class": "Connection",
"connectorType": {
"class": "ConnectorType",
"connectorProviderClassName": "org.odpi.openmetadata.metadatasecurity.samples.OpenMetadataServerSecurityProvider"
}
}
Determine configured security¶
GET - query the server security connector setting
{{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/security/connection
Response indicating no security
{
"class": "ConnectionResponse",
"relatedHTTPCode": 200
}
Response indicating a specific security connector
If the response looks more like the JSON below, a connector is configured. The connectorProviderClassName tells you which connector is being used.
{
"class": "ConnectionResponse",
"relatedHTTPCode": 200,
"connection": {
"class": "Connection",
"connectorType": {
"class": "ConnectorType",
"connectorProviderClassName": "{fullyQualifiedJavaClassName}"
}
}
}
Remove configured security¶
DELETE - remove configured security connector
{{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/security/connection
This removes all authorization checking from the server.
Registering the server with a cohort
Configuring registration to an Open Metadata Repository Cohort¶
An OMAG Server that is capable of being a Cohort Member can register with one or more open metadata repository cohorts.
Each cohort has a memorable name - eg cocoCohort. This name needs to be used in the configuration of each member. At the heart of a cohort are 1-4 cohort topics. These are topics on an event bus that the members use to exchange information.
There is a choice of topic structure for the cohort.
- A single topic is used for all types of events
- Three topics are used, each dedicated to a specific type of cohort event:
- Registration events that exchange information about the members of the cohort.
- Type verification events that ensure consistency of the open metadata types used by the members of the cohort.
- Instance events that enable members of the cohort to share metadata elements.
The use of a single topic comes from the original implementation of Egeria. The use of the three dedicated topics was added later in version 2.11 to reduce the latency of cohort registration and to allow tuning of each topic's configuration. This is essential when multiple instances of an OMAG server are running in a cluster because the registration and type verification events need to be received by all server instances and the instance events need only to be received by one of the server instances.
Typically, all members of the cohort should be configured to use the same topic structure. However, if one of the members is back level and can only support the single topic then the other members can be set up to operate both topic structures. This is less efficient as these servers will process most instance events twice. However, it does provide a workaround until the back-level member can be upgraded.
The choices of topic structure are summarized in Figure 1.
Figure 1: Choices of cohort topic structures referred to as SINGLE_TOPIC, DEDICATED_TOPICS and BOTH_SINGLE_AND_DEDICATED_TOPICS reading left to right
Configuration commands¶
The commands for configuring a server as a member of a cohort are shown below. Before calling these commands, make sure that the default settings for the event bus are configured, and you know the name of the cohort and the topic structure it is using.
Add access to a cohort
The following command registers the server with a cohort using the default settings. This includes the default cohort topic structure, which is SINGLE_TOPIC before version 3.0 and DEDICATED_TOPICS for version 3.0 and above.
POST {platformURLRoot}/open-metadata/admin-services/users/{adminUserId}/servers/{serverName}/cohorts/{cohortName}
Alternatively it is possible to explicitly specify the cohort topic structure. The example below sets it to DEDICATED_TOPICS. The other options are SINGLE_TOPIC and BOTH_SINGLE_AND_DEDICATED_TOPICS.
POST {platformURLRoot}/open-metadata/admin-services/users/{adminUserId}/servers/{serverName}/cohorts/{cohortName}/topic-structure/DEDICATED_TOPICS
Both of these commands optionally support passing a map of name-value pairs in the request body. These properties are added to the additionalProperties attribute of the Connection objects for each of the cohort topics. The additional properties supported are specific to the topic connector implementation. For example, see the Apache Kafka Topic Connector Documentation.
The result of the cohort configuration call fills out an entry in the cohort list of the server's configuration document. The fields in a cohort list entry are show in Figure 2.
Figure 2: Fields in an entry in a server's cohort list
It is possible to update any of these fields directly using the following command:
POST {platformURLRoot}/open-metadata/admin-services/users/{adminUserId}/servers/{serverName}/cohorts/{cohortName}/configuration
JSON structure for a member that is using DEDICATED_TOPICS
{
"class": "CohortConfig",
"cohortName": "cocoCohort",
"cohortRegistryConnection": {
"class": "Connection",
"headerVersion": 0,
"connectorType": {
"class": "ConnectorType",
"headerVersion": 0,
"type": {
"class": "ElementType",
"headerVersion": 0,
"elementOrigin": "LOCAL_COHORT",
"elementVersion": 0,
"elementTypeId": "954421eb-33a6-462d-a8ca-b5709a1bd0d4",
"elementTypeName": "ConnectorType",
"elementTypeVersion": 1,
"elementTypeDescription": "A set of properties describing a type of connector."
},
"guid": "108b85fe-d7a8-45c3-9f88-742ac4e4fd14",
"qualifiedName": "File Based Cohort Registry Store Connector",
"displayName": "File Based Cohort Registry Store Connector",
"description": "Connector supports storing of the open metadata cohort registry in a file.",
"connectorProviderClassName": "org.odpi.openmetadata.adapters.repositoryservices.cohortregistrystore.file.FileBasedRegistryStoreProvider"
},
"endpoint": {
"class": "Endpoint",
"headerVersion": 0,
"address": "./data/servers/cocoMDS4/cohorts/cocoCohort.registrystore"
}
},
"cohortOMRSRegistrationTopicConnection": {
"class": "VirtualConnection",
"headerVersion": 0,
"connectorType": {
"class": "ConnectorType",
"headerVersion": 0,
"connectorProviderClassName": "org.odpi.openmetadata.repositoryservices.connectors.omrstopic.OMRSTopicProvider"
},
"embeddedConnections": [
{
"class": "EmbeddedConnection",
"headerVersion": 0,
"position": 0,
"displayName": "cocoCohort OMRS Topic for registrations",
"embeddedConnection": {
"class": "Connection",
"headerVersion": 0,
"connectorType": {
"class": "ConnectorType",
"headerVersion": 0,
"type": {
"class": "ElementType",
"headerVersion": 0,
"elementOrigin": "LOCAL_COHORT",
"elementVersion": 0,
"elementTypeId": "954421eb-33a6-462d-a8ca-b5709a1bd0d4",
"elementTypeName": "ConnectorType",
"elementTypeVersion": 1,
"elementTypeDescription": "A set of properties describing a type of connector."
},
"guid": "3851e8d0-e343-400c-82cb-3918fed81da6",
"qualifiedName": "Kafka Open Metadata Topic Connector",
"displayName": "Kafka Open Metadata Topic Connector",
"description": "Kafka Open Metadata Topic Connector supports string based events over an Apache Kafka event bus.",
"connectorProviderClassName": "org.odpi.openmetadata.adapters.eventbus.topic.kafka.KafkaOpenMetadataTopicProvider",
"recognizedConfigurationProperties": [
"producer",
"consumer",
"local.server.id",
"sleepTime"
]
},
"endpoint": {
"class": "Endpoint",
"headerVersion": 0,
"address": "egeria.omag.openmetadata.repositoryservices.cohort.cocoCohort.OMRSTopic.registration"
},
"configurationProperties": {
"producer": {
"bootstrap.servers": "localhost:9092"
},
"local.server.id": "73955db6-026c-4ba5-a180-1355dbf166cf",
"consumer": {
"bootstrap.servers": "localhost:9092"
}
}
}
}
]
},
"cohortOMRSTypesTopicConnection": {
"class": "VirtualConnection",
"headerVersion": 0,
"connectorType": {
"class": "ConnectorType",
"headerVersion": 0,
"connectorProviderClassName": "org.odpi.openmetadata.repositoryservices.connectors.omrstopic.OMRSTopicProvider"
},
"embeddedConnections": [
{
"class": "EmbeddedConnection",
"headerVersion": 0,
"position": 0,
"displayName": "cocoCohort OMRS Topic for types",
"embeddedConnection": {
"class": "Connection",
"headerVersion": 0,
"connectorType": {
"class": "ConnectorType",
"headerVersion": 0,
"type": {
"class": "ElementType",
"headerVersion": 0,
"elementOrigin": "LOCAL_COHORT",
"elementVersion": 0,
"elementTypeId": "954421eb-33a6-462d-a8ca-b5709a1bd0d4",
"elementTypeName": "ConnectorType",
"elementTypeVersion": 1,
"elementTypeDescription": "A set of properties describing a type of connector."
},
"guid": "3851e8d0-e343-400c-82cb-3918fed81da6",
"qualifiedName": "Kafka Open Metadata Topic Connector",
"displayName": "Kafka Open Metadata Topic Connector",
"description": "Kafka Open Metadata Topic Connector supports string based events over an Apache Kafka event bus.",
"connectorProviderClassName": "org.odpi.openmetadata.adapters.eventbus.topic.kafka.KafkaOpenMetadataTopicProvider",
"recognizedConfigurationProperties": [
"producer",
"consumer",
"local.server.id",
"sleepTime"
]
},
"endpoint": {
"class": "Endpoint",
"headerVersion": 0,
"address": "egeria.omag.openmetadata.repositoryservices.cohort.cocoCohort.OMRSTopic.types"
},
"configurationProperties": {
"producer": {
"bootstrap.servers": "localhost:9092"
},
"local.server.id": "73955db6-026c-4ba5-a180-1355dbf166cf",
"consumer": {
"bootstrap.servers": "localhost:9092"
}
}
}
}
]
},
"cohortOMRSInstancesTopicConnection": {
"class": "VirtualConnection",
"headerVersion": 0,
"connectorType": {
"class": "ConnectorType",
"headerVersion": 0,
"connectorProviderClassName": "org.odpi.openmetadata.repositoryservices.connectors.omrstopic.OMRSTopicProvider"
},
"embeddedConnections": [
{
"class": "EmbeddedConnection",
"headerVersion": 0,
"position": 0,
"displayName": "cocoCohort OMRS Topic for instances",
"embeddedConnection": {
"class": "Connection",
"headerVersion": 0,
"connectorType": {
"class": "ConnectorType",
"headerVersion": 0,
"type": {
"class": "ElementType",
"headerVersion": 0,
"elementOrigin": "LOCAL_COHORT",
"elementVersion": 0,
"elementTypeId": "954421eb-33a6-462d-a8ca-b5709a1bd0d4",
"elementTypeName": "ConnectorType",
"elementTypeVersion": 1,
"elementTypeDescription": "A set of properties describing a type of connector."
},
"guid": "3851e8d0-e343-400c-82cb-3918fed81da6",
"qualifiedName": "Kafka Open Metadata Topic Connector",
"displayName": "Kafka Open Metadata Topic Connector",
"description": "Kafka Open Metadata Topic Connector supports string based events over an Apache Kafka event bus.",
"connectorProviderClassName": "org.odpi.openmetadata.adapters.eventbus.topic.kafka.KafkaOpenMetadataTopicProvider",
"recognizedConfigurationProperties": [
"producer",
"consumer",
"local.server.id",
"sleepTime"
]
},
"endpoint": {
"class": "Endpoint",
"headerVersion": 0,
"address": "egeria.omag.openmetadata.repositoryservices.cohort.cocoCohort.OMRSTopic.instances"
},
"configurationProperties": {
"producer": {
"bootstrap.servers": "localhost:9092"
},
"local.server.id": "73955db6-026c-4ba5-a180-1355dbf166cf",
"consumer": {
"bootstrap.servers": "localhost:9092"
}
}
}
}
]
},
"cohortOMRSTopicProtocolVersion": "V1",
"eventsToProcessRule": "ALL"
}
Controlling the name of the cohort topic(s)
Typically, a production deployment of an event bus requires the topics to be explicitly defined in its configuration. In addition, many organizations have naming standards for topics. Therefore, Egeria provides commands to query the topic names from the configuration for easy automation and the ability to override the topic names.
The default single topic name is egeria.omag.openmetadata.repositoryservices.cohort.{cohortName}.OMRSTopic and the default dedicated topic names are:
- For registration events -
egeria.omag.openmetadata.repositoryservices.cohort.{cohortName}.OMRSTopic.registration - For type verification events -
egeria.omag.openmetadata.repositoryservices.cohort.{cohortName}.OMRSTopic.types - For instance events -
egeria.omag.openmetadata.repositoryservices.cohort.{cohortName}.OMRSTopic.instances
This is the command to query the single topic name.
GET {platformURLRoot}/open-metadata/admin-services/users/{adminUserId}/servers/{serverName}/cohorts/{cohortName}/topic-name
{
"class": "StringResponse",
"relatedHTTPCode": 200,
"resultString": "egeria.openmetadata.repositoryservices.cohort.cocoCohort.OMRSTopic"
}
{
"class": "StringResponse",
"relatedHTTPCode": 200
}
{
"class": "StringResponse",
"relatedHTTPCode": 400,
"exceptionClassName": "org.odpi.openmetadata.adminservices.ffdc.exception.OMAGInvalidParameterException",
"exceptionErrorMessage": "OMAG-ADMIN-400-033 The OMAG server cocoMDS1 is unable to override the cohort topic until the cocoCohortXXX cohort is set up",
"exceptionSystemAction": "No change has occurred in this server's configuration document.",
"exceptionUserAction": "Add the cohort configuration using the administration services and retry the request."
}
This is the command to retrieve the dedicated topics:
GET {platformURLRoot}/open-metadata/admin-services/users/{adminUserId}/servers/{serverName}/cohorts/{cohortName}/dedicated-topic-names
The result looks like this with the registration topic showing first, then the type verification topic and lastly the "instances topic":
{
"class": "DedicatedTopicListResponse",
"relatedHTTPCode": 200,
"dedicatedTopicList": {
"registrationTopicName": "egeria.omag.openmetadata.repositoryservices.cohort.cocoCohort.OMRSTopic.registration",
"typesTopicName": "egeria.omag.openmetadata.repositoryservices.cohort.cocoCohort.OMRSTopic.types",
"instancesTopicName": "egeria.omag.openmetadata.repositoryservices.cohort.cocoCohort.OMRSTopic.instances"
}
}
Override the value for the cohort topic
It is also possible to change the name of the topics used by a cohort. Any changes must be issued against each member of the cohort so that they are all connecting to the same cohort topic(s). The new value takes affect the next time the server is started.
Changing the single topic name is done with the following command
POST {platformURLRoot}/open-metadata/admin-services/users/{adminUserId}/servers/{serverName}/cohorts/{cohortName}/topic-name-override
{newTopicName}
The {newTopicName} flows in the request body as raw text.
This is the command for changing the registration topic name:
POST {platformURLRoot}/open-metadata/admin-services/users/{adminUserId}/servers/{serverName}/cohorts/{cohortName}/topic-name-override/registration
{newTopicName}
This is the command for changing the type verification topic name:
POST {platformURLRoot}/open-metadata/admin-services/users/{adminUserId}/servers/{serverName}/cohorts/{cohortName}/topic-name-override/types
{newTopicName}
This is the command for changing the "instances topic" name:
POST {platformURLRoot}/open-metadata/admin-services/users/{adminUserId}/servers/{serverName}/cohorts/{cohortName}/topic-name-override/instances
{newTopicName}
Disconnect from a cohort
This command unregisters a server from a cohort.
DELETE {platformURLRoot}/open-metadata/admin-services/users/{adminUserId}/servers/{serverName}/cohorts/{cohortName}
Configuring the test workbenches
Configure the workbenches¶
The workbenches are configured using the OMAG Server Platform Administration Services. This defines which workbenches to run and how to connect to the technology to test. This configuration defines an OMAG Server that will run the requested conformance suite tests.
Configure the OMAG Server that will run the requested conformance suite tests. The requested workbenches will begin to execute their tests as soon as the OMAG Server is started.
Repository workbench¶
To run a metadata repository through the Repository Workbench, first configure a CTS server in the OMAG Server Platform by configuring its general properties like server type, event bus, cohort, etc. Before starting the CTS server instance, configure the repository workbench within it using the following command:
POST - configure repository workbench
{{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/cts/conformance-suite-workbenches/repository-workbench/repositories
Send the repository workbench configuration as the request body, similar to the following:
{
"class": "RepositoryConformanceWorkbenchConfig",
"tutRepositoryServerName": "myserver",
"testEntityTypes" : "Asset",
"maxSearchResults": 5
}
The required tutRepositoryServerName parameter defines the name of the repository server you wish to test, while the optional maxSearchResults parameter controls the sizing of the tests: both the number of instances the tests will attempt to create to carry out its tests and how extensive the search-based tests are. The testEntityTypes is also optional and is used to restrict the types of the instances tested to:
- These listed entities and their super types.
- Any relationships that have the selected entity types at both ends.
- Any classifications that can be attached to the selected entity types.
If testEntityTypes is either an empty list or not set, then all known types from the technology under test is tested.
Start the technology under test after the CTS server
This repository server to test (myserver in the example above) should be configured and started after starting the CTS repository workbench instance. Once the CTS server instance is started it will wait for the technology under test (the server named by the tutRepositoryServerName parameter) to be up and running before then starting its suite of tests.
The OMAG Server also supports a REST API for querying the results of running the conformance suite tests. These commands include:
- Retrieve the results from a single named workbench.
- Retrieve the results from all workbenches and test cases (beware that the response can be 100's of MB in size, and may overflow your JVM heap).
- Retrieve the results from all failed test cases.
- Retrieve the Ids of all test cases.
- Retrieve the results from a specific test cases (for example, iterating through the above call's response).
- Retrieve the names of all profiles.
- Retrieve the details of a single profile's results (for example, iterating through the above call's response).
The resulting reports can be large
Ensure the jvm running the CTS server has at least 1GB heap to avoid any Java heap errors.
The Open Metadata Conformance Suite also has a client called OpenMetadataConformanceTestReport that will retrieve the conformance report and all details. It will store a summarized report in openmetadata_cts_summary.json, and the full details of each profile and test case in profile-details and test-cases sub-directories, respectively. The client also outputs a summary of the test run.
Example output for a successful CTS run
This output is an example of a successful run:
$ OpenMetadataConformanceTestReport cSuiteServer https://localhost:9444
=======================================
Open Metadata Conformance Test Report
=======================================
Contacting conformance suite server: cts (https://localhost:9443)
Saving full profile details into 'profile-details' directory...
Summary of profile results:
... Metadata sharing: CONFORMANT_FULL_SUPPORT
... Reference copies: CONFORMANT_FULL_SUPPORT
... Metadata maintenance: CONFORMANT_FULL_SUPPORT
... Dynamic types: UNKNOWN_STATUS
... Graph queries: CONFORMANT_FULL_SUPPORT
... Historical search: CONFORMANT_FULL_SUPPORT
... Entity proxies: CONFORMANT_FULL_SUPPORT
... Soft-delete and restore: CONFORMANT_FULL_SUPPORT
... Undo an update: CONFORMANT_FULL_SUPPORT
... Reidentify instance: CONFORMANT_FULL_SUPPORT
... Retype instance: CONFORMANT_FULL_SUPPORT
... Rehome instance: CONFORMANT_FULL_SUPPORT
... Entity search: CONFORMANT_FULL_SUPPORT
... Relationship search: CONFORMANT_FULL_SUPPORT
... Entity advanced search: CONFORMANT_FULL_SUPPORT
... Relationship advanced search: CONFORMANT_FULL_SUPPORT
Saving full test case details into 'test-case-details' directory (can take 1-2 minutes)...
Summary:
... number of tests: 4965
... number of tests passed: 4965
... number of tests failed: 0
... number of tests skipped: 0
Congratulations, technology under test is conformant
Process finished with exit code 0
Example output for an unsuccessful CTS run
The example below is for an unsuccessful run (where one of the Entity search tests has failed):
$ OpenMetadataConformanceTestReport cSuiteServer https://localhost:9444
=======================================
Open Metadata Conformance Test Report
=======================================
Contacting conformance suite server: cts (https://localhost:9443)
Saving full profile details into 'profile-details' directory...
Summary of profile results:
... Metadata sharing: CONFORMANT_FULL_SUPPORT
... Reference copies: CONFORMANT_FULL_SUPPORT
... Metadata maintenance: CONFORMANT_FULL_SUPPORT
... Dynamic types: UNKNOWN_STATUS
... Graph queries: CONFORMANT_FULL_SUPPORT
... Historical search: CONFORMANT_FULL_SUPPORT
... Entity proxies: CONFORMANT_FULL_SUPPORT
... Soft-delete and restore: CONFORMANT_FULL_SUPPORT
... Undo an update: CONFORMANT_FULL_SUPPORT
... Reidentify instance: CONFORMANT_FULL_SUPPORT
... Retype instance: CONFORMANT_FULL_SUPPORT
... Rehome instance: CONFORMANT_FULL_SUPPORT
... Entity search: NOT_CONFORMANT
... Relationship search: CONFORMANT_FULL_SUPPORT
... Entity advanced search: CONFORMANT_FULL_SUPPORT
... Relationship advanced search: CONFORMANT_FULL_SUPPORT
Saving full test case details into 'test-case-details' directory (can take 1-2 minutes)...
Summary:
... number of tests: 4965
... number of tests passed: 4964
... number of tests failed: 1
... number of tests skipped: 0
Technology under test is not yet conformant
Process finished with exit code 1
Raise an issue or comment below