Skip to content

Writing a Governance Action Service

A governance action service is a specialized connector that performs monitoring of metadata changes, validation of metadata, triage of issues, assessment and/or remediation activities on request. Some governance action services invoke functions in external engines that are working with data and related assets.

A governance action service runs in the Governance Action Open Metadata Engine Service (OMES) hosted by the Engine Host OMAG Server.

Governance Action Service

Governance action services implement interfaces defined by the Governance Action Framework (GAF). The GAF offers embeddable functions and APIs to simplify the implementation of governance action services, and their integration into the broader digital landscape, whilst being resilient and with good performance.

It is possible to implement complex governance actions in a single governance action service. Alternatively there are five specialized types of governance action services that help you to break down your governance function into reusable components that can be choreographed by governance action processes to maximise the flexibility of your governance automation. When a governance action service completes, it produces guards that define what needs to be done next along with a list of action targets.

  • Watchdog Governance Action Service listens for changes to metadata and initiates new governance actions, governance action processes or an incident report.

  • Verification Governance Action Service validates that a rule or policy is being followed. This is often a test that the metadata elements, relationships and classification are set up as they should be. For example, it may check that a new asset has an owner, is set up with governance zones and includes a connection and a schema there possible. Verification governance action services

  • Triage Governance Action Service runs triage rules to determine how to manage a situation or request, such as a request for action from an open discovery service. Often this involves a human decision maker. It may initiate an external workflow, wait for manual decision or create a ToDo for a specific person.

  • Remediation Governance Action Service makes updates to metadata elements, relationships between them and classifications. Examples of remediation governance action services include:

    • Classification and linking of metadata elements such as adding owners, governance zones and origin classifications to assets.
    • Duplicate detection, linking and consolidating.
  • Provisioning Governance Action Service invokes a provisioning service whenever a provisioning request is made. Typically, the provisioning service is an external service. It may also create lineage metadata to describe the work of the provisioning service if the provisioning service is not able to create lineage itself.

The interfaces for governance action services is defined in the governance-action-framework module.

Governance action service example - data onboarding process

The governance action services are best understood through examples. Consider an onboarding process where new files are being copied into a landing area. They need to be catalogued in open metadata and moved into the data lake folder.

Data onboarding scenario

Operation of the data onboarding process

Initialization

At the start, there is an integration connector called Data Files Monitoring Integration Connector that will detect new files in the landing area folder and create an Asset metadata element to describe the file.

There is also governance action service called New Asset Watchdog that has registered a listener for new Asset metadata elements.

Data onboarding startup

New files arrive

When a new file arrives, Data Files Monitoring Integration Connector detects is and catalogues it as an Asset in open metadata. This triggers a call to New Asset Watchdog which then creates a governance action to initate a provisioning governance action service .

Data onboarding startup

Provisioning to the data lake

The governance action identifies the governance action service called Clinical Trial Provisioning and so it is started in the engine host. It moves the file to the data lake folder and adds lineage metadata to describe the data movement and the new Asset for the file in the data lake. The original asset is still in the metadata repository since it is needed to show the source of the data movement.

Data onboarding startup

Archive the asset for the landing area

The result of the provisioning removes the file from the landing area folder. This is detected by Data Files Monitoring Integration Connector which then archives the corresponding Asset. This addes a Memento classification to the Asset which means it is only retrievable on lineage requests.

Data onboarding startup

This is a summary of the flow:

Data onboarding overview

  1. New file detected by the Integration Connector.
  2. An Asset describing the file is created in the Metadata Access Server.
  3. New Asset event passed to Watchdog Governance Action Service.
  4. New Governance Action created that results in notification to Engine Host.
  5. Engine Host claims Governance Action and activates Provisioning Governance Action Service.
  6. Provisioning Governance Action Service moves file and writes lineage.
  7. Deleted file is detected by the Integration Connector.
  8. File's Asset is archived (adding a Memento classification to the Asset).

Since the watchdog governance action service calls the provisioning governance action process explicitly via the governance action, their implementations are somewhat tied together. The alternative is that the watchdog governance action service can invoke a governance action process that will choreograph the execution of one or more governance services based on a flow definition managed in open metadata. The governance action process separates the implementation of the watchdog governance action service from the follow-on governance actions since changes to the follow-on processing is maintained through open metadata rather than requiring code changes to the watchdog governance action service code.

Governance action context

A governance action service is passed a context as it is started. This provides access to the request type and associated request parameters (name-value pairs) used to invoke the governance action service, along with its action targets. There is also a client to access open metadata through the Governance Engine OMAS.

Structure of the governance context

The methods of the context are colour-coded. The methods shown in white are available for all types of governance action services. These are divided into three groups:

  • Understanding the request

    • getRequestType - returns the governance request type used to invoke the governance action service.
    • getRequestParameters - returns the name-value pairs of properties passed with the request type.
    • getRequestSourceElements - returns details of the process that invoked the governance action.
    • getActionTargetElements - returns the list of action target elements that this governance action service is to work on.
  • Performing the action

    • getOpenMetadataStore - returns a client to the Governance Engine OMAS that supports the retrieval of metadata elements, their classifications and relationships.
    • createIncidentReport - creates an incident report that can provide a focal point for collaboration to resolve a particular issue. Incident reports are often managed by an incident management tool.
  • Recording the outcome of the processing

    • updateActionTargetStatus - records that the governance action service has completed processing a specific action target. This is optional, but useful to show progress when a governance action process is performing action on a long list of action targets.
    • recordCompletionStatus - declares that the governance action service has completed processing. It has three parameters:

      • completionStatus - with values:

        • Actioned - the governance action service has successfully completed processing.
        • Invalid - the governance action service has not performed the requested action because it is not appropriate (for example, a false positive).
        • Failed - The governance action service failed to execute the requested action.
        • Other - Undefined or unknown completion status.
      • outputGuards - the list of guards that are used to determine which governance action to run next.

      • newActionTargets - the list of action targets that the follow-on governance services should process.
    • getCompletionStatus - returns the status passed on recordCompletionStatus or null if no status has been posted. It is used to coordinate the shutdown of multiple threads operated by the governance action service.

The methods in blue are addition services offered to governance action services implementing a watchdog capability:

  • registerListener - requests that it is called whenever certain open metadata events occur. It is able to specify a range of conditions from receiving all event, events for particular actions (such as creation of a new metadata element), events for particular metadata types or events on particular instances.
  • initiateGovernanceAction - requests that a new governance action is run.
  • initiateGovernanceActionProcess - requests that a new governance action process is run.

The methods in purple are additional services offered to governance action services implementing a provisioning capability to record lineage:

  • createAsset - create an asset - typically the destination of the provisioning.
  • createProcess - create a representation of the provisioning process.
  • createPort - create a port for the process
  • createLineageMapping - create a lineage relationship.

There are no additional services for governance action services implementing a verification capability which is why the green box is empty.

The methods in yellow are additional services offered to governance action services implementing a triage capability to initiate a stewardship action:

  • createToDo - create a ToDo that is assigned to an individual.

The methods in red are additional services offered to governance action services implementing a remediation capability to make updates to open metadata. They are divided into four groups:

  • Working with metadata elements

    • createMetadataElement
    • updateMetadataElement
    • updateMetadataElementStatus
    • deleteMetadataElement
  • Working with classification on a metadata element

    • classifyMetadataElement
    • reclassifyMetadataElement
    • updateClassificationStatus
    • unclassifyMetadataElement
  • Working with relationships (links) between metadata elements

    • createRelatedElements
    • updateRelatedElements
    • updateRelatedElementsStatus
    • deleteRelatedElements
  • Performing specialist governance actions

    • linkElementsAsPeerDuplicates
    • LinkConsolidatedDuplicate

A governance action service has a choice of base classes to extend that controls the methods provided in the governance action context:

  • GovernanceActionService - for the complete context
  • WatchdogGovernanceActionService - for the white and blue methods
  • ProvisioningGovernanceActionService - for the white and purple methods
  • VerificationGovernanceActionService - for the white and green methods
  • TriageGovernanceActionService - for the white and yellow methods
  • RemediationGovernanceActionService - for the white and red methods

The Writing the governance action service connector section provides more information on how to use the methods.

Writing the connector provider

The connector provider for your governance action service provides the factory method to create an instance of the governance action service. It also supports methods to describe the options that the governance action service provides, such as its supported governance request types, request parameters, action target types, configuration properties and output guards.

Below is the implementation of the connector provider for the Move Copy File Provisioning Governance Action Service. This is a highly configurable governance action service that can be instructed to move, copy or delete a file, and has different styles of lineage it can create. The action it performs is supplied in the governance request type. The source file and destination folder can be supplied either through the request parameters or as action targets. There are two guards: "provisioning-complete" for success and "provisioning-failed" if something went wrong.

The methods of the connector provider enables a tool that is configuring governance actions or governance action processes to query the capabilities of the corresponding governance action service.

public class MoveCopyFileGovernanceActionProvider extends GovernanceActionServiceProviderBase
{
    private static final String  connectorTypeGUID = "e2a14ca8-57b1-48d7-9cc4-d0b44983ca79";
    private static final String  connectorTypeQualifiedName = "Egeria:GovernanceActionService:Provisioning:MoveCopyFile";
    private static final String  connectorTypeDisplayName = "Move or Copy File Governance Action Service";
    private static final String  connectorTypeDescription = "Provisioning Governance Action Service that moves or copies files on request.";

    static final String TARGET_FILE_NAME_PATTERN_PROPERTY           = "targetFileNamePattern";

    static final String NO_LINEAGE_PROPERTY                         = "noLineage";
    static final String TOP_LEVEL_PROCESS_NAME_PROPERTY             = "topLevelProcessQualifiedName";
    static final String TOP_LEVEL_PROCESS_TEMPLATE_NAME_PROPERTY    = "topLevelProcessTemplateQualifiedName";
    static final String DESTINATION_TEMPLATE_NAME_PROPERTY          = "destinationFileTemplateQualifiedName";
    static final String TOP_LEVEL_PROCESS_ONLY_LINEAGE_PROPERTY     = "topLevelProcessLineageOnly";
    static final String LINEAGE_TO_DESTINATION_FOLDER_ONLY_PROPERTY = "lineageToDestinationFolderOnly";
    static final String LINEAGE_FROM_SOURCE_FOLDER_ONLY_PROPERTY    = "lineageFromSourceFolderOnly";

    static final String COPY_REQUEST_TYPE   = "copy-file";
    static final String MOVE_REQUEST_TYPE   = "move-file";
    static final String DELETE_REQUEST_TYPE = "delete-file";

    static final String SOURCE_FILE_PROPERTY        = "sourceFile";
    static final String DESTINATION_FOLDER_PROPERTY = "destinationFolder";

    static final String PROVISIONING_COMPLETE_GUARD = "provisioning-complete";
    static final String PROVISIONING_FAILED_GUARD   = "provisioning-failed";

    private static final String connectorClassName = MoveCopyFileGovernanceActionConnector.class.getName();


    /**
     * Constructor used to initialize the ConnectorProviderBase with the Java class name of the specific
     * store implementation.
     */
    public MoveCopyFileGovernanceActionProvider()
    {
        super();
        super.setConnectorClassName(connectorClassName);

        supportedRequestTypes = new ArrayList<>();
        supportedRequestTypes.add(COPY_REQUEST_TYPE);
        supportedRequestTypes.add(MOVE_REQUEST_TYPE);
        supportedRequestTypes.add(DELETE_REQUEST_TYPE);

        supportedRequestParameters = new ArrayList<>();
        supportedRequestParameters.add(SOURCE_FILE_PROPERTY);
        supportedRequestParameters.add(DESTINATION_FOLDER_PROPERTY);

        supportedTargetActionNames = new ArrayList<>();
        supportedTargetActionNames.add(SOURCE_FILE_PROPERTY);
        supportedTargetActionNames.add(DESTINATION_FOLDER_PROPERTY);

        supportedGuards = new ArrayList<>();
        supportedGuards.add(PROVISIONING_COMPLETE_GUARD);
        supportedGuards.add(PROVISIONING_FAILED_GUARD);

        super.setConnectorClassName(connectorClassName);

        ConnectorType connectorType = new ConnectorType();
        connectorType.setType(ConnectorType.getConnectorTypeType());
        connectorType.setGUID(connectorTypeGUID);
        connectorType.setQualifiedName(connectorTypeQualifiedName);
        connectorType.setDisplayName(connectorTypeDisplayName);
        connectorType.setDescription(connectorTypeDescription);
        connectorType.setConnectorProviderClassName(this.getClass().getName());

        List<String> recognizedConfigurationProperties = new ArrayList<>();
        recognizedConfigurationProperties.add(TARGET_FILE_NAME_PATTERN_PROPERTY);
        recognizedConfigurationProperties.add(NO_LINEAGE_PROPERTY);
        recognizedConfigurationProperties.add(TOP_LEVEL_PROCESS_NAME_PROPERTY);
        recognizedConfigurationProperties.add(TOP_LEVEL_PROCESS_TEMPLATE_NAME_PROPERTY);
        recognizedConfigurationProperties.add(DESTINATION_TEMPLATE_NAME_PROPERTY);
        recognizedConfigurationProperties.add(DESTINATION_FOLDER_PROPERTY);
        recognizedConfigurationProperties.add(TOP_LEVEL_PROCESS_ONLY_LINEAGE_PROPERTY);
        recognizedConfigurationProperties.add(LINEAGE_FROM_SOURCE_FOLDER_ONLY_PROPERTY);
        recognizedConfigurationProperties.add(LINEAGE_TO_DESTINATION_FOLDER_ONLY_PROPERTY);
        connectorType.setRecognizedConfigurationProperties(recognizedConfigurationProperties);

        super.connectorTypeBean = connectorType;
    }
}

Writing the governance action service connector

A governance action service must extend one of the governance action service base classes. Your choice determine the methods available to your governance action service through the governance action context.

  • GovernanceActionService - for the complete context
  • WatchdogGovernanceActionService - for the shared (white) and watchdog (blue) methods
  • ProvisioningGovernanceActionService - for the shared (white) and provisioning (purple) methods
  • VerificationGovernanceActionService - for the shared (white) and verification (green) methods
  • TriageGovernanceActionService - for the shared (white) and triage (yellow) methods
  • RemediationGovernanceActionService - for the shared (white) and remediation (red) methods

In addition, it needs to implement the standard Open Connector Framework (OCF) methods:

  • initialize - an optional method where the connection object can be processed to, say, extract the configuration properties.
  • start - this is where the processing logic of your governance action service belongs.
  • disconnect - an optional method to free up any runtime resources that the governance action service is using. It is called after the governance action process records a completion status, or if it throws an exception from the start method.

Shared context methods

Watchdog context methods

The watchdog governance action service monitors changes in the metadata and initiates one of the following as a result:

One example of a watchdog governance action service is to monitor for new assets. Another example is to monitor the addition of open discovery reports and take action on their content.

Function of the watchdog governance action service

Verification context methods

A Verification governance action service tests the properties of specific open metadata elements to ensure they are set up correctly and do not indicate a situation where governance activity is required. For example, it may check that a new asset has an owner assigned, is set up with governance zones and includes a connection and a schema.

The verification governance action services publishes guards to report on any errors it finds. It may also create incident reports to coordinate actions to correct any errors. The guards returned from the verification governance action service can be used to trigger other governance services as part of a governance action process.

Function of the verification governance action service

Triage context methods

Triage governance action services run triage rules to determine how to manage a situation that needs human intervention. This could be to initiate an external workflow, wait for manual decision or initiate a remediation request through either an external workflow or by creating a ToDo for a specific person.

Function of the triage governance action service

Remediation context methods

The remediation governance action service performs updates to metadata. Examples of the tasks performed by a remediation governance action service are assigning classifications, duplicate linking and consolidation.

Function of the remediation governance action service

Provisioning context methods

A provisioning governance action service invokes a provisioning service whenever a provisioning request is made. Typically, the provisioning service is an external service or pipeline. The provisioning governance action service may also create lineage metadata to describe the provisioning if this is not performed by another service.

Function of the provisioning governance action service

Next steps

  • Configuring your governance action service - A collection of related governance action services are grouped into governance action engines for deployment. The governance action engine maps governance action request types to the governance action service that should be invoked along with.

Structure of a governance engine definition

These definitions are created as part of a governance engine pack or through the Governance Engine OMAS and are stored in the open metadata repositories.