Skip to content
Technical preview

Technical preview function is in a state that it can be tried. The development is complete, there is documentation and there are samples, tutorials and hands-on labs as appropriate.

The community is looking for feedback on the function before declaring it stable. This feedback may result in changes to the external interfaces.

Data Files Monitor Integration Connector

Connector details

The data files monitor integration connector monitors changes in a file directory (folder) and updates the open metadata repository/repositories to reflect the changes to both the files and folders underneath it.

Specifically:

  • A DataFile asset is created and then maintained for each file in the folder (or any sub-folder).
  • When a new file is created, a new DataFile asset is created.
  • If a file is modified, the lastModified property of the corresponding DataFile asset is updated.
  • When a file is deleted, its corresponding DataFile asset is either:
    • Archived: this means the asset is no longer returned on standard asset catalog searches, but it is still visible in lineage graphs. This is the default behavior.
    • Deleted: this means that all metadata associated with the data file is removed. Only use this option if lineage is not important for these file.
  • A FileFolder metadata asset for the monitored folder is created when the first file is catalogued, if it does not already exist.

Configuration

Figure 1

Figure 1: Operation of the data files monitor integration connector

This connector uses the Files Integrator OMIS running in the integration daemon.

Following is its connection definition to use on the administration commands that configure the Files Integrator OMIS:

Connection configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
{
  "connection": 
  {
    "class": "Connection",
    "connectorType": 
    {
      "class": "ConnectorType",
      "connectorProviderClassName": "org.odpi.openmetadata.adapters.connectors.integration.basicfiles.DataFilesMonitorIntegrationProvider"
    },
    "endpoint": 
    {
      "class": "Endpoint",
      "address": "{{folderName}}"
    },
    "configurationProperties": 
    {
      "templateQualifiedName": "{{templateQualifiedName}}",
      "allowCatalogDelete": ""         
    }
  }
}
  • Replace {{folderName}} with the path name of the folder where the files will be located.
  • The configurationProperties are optional and are used to override the connector's default behavior:
    • If templateQualifiedName is present in the configuration properties then {{templateQualifiedName}} must be set to the qualified name of a DataFile metadata element that should be used as a template for the catalog entry for new files discovered by this connector. The base properties, schema, connection, classifications and any other attachments connected to the template are copied to the new metadata element for the file. (See templated cataloging for more information on the use of templates.)
    • If allowCatalogDelete is present in the configuration properties then the metadata element for a file is deleted when the file is deleted. If this property is not in the configuration properties, then the metadata element is archived (by adding the Memento classification to its entry). The archived element is no longer returned in standard catalog queries, but it is still visible in lineage. Do not set allowCatalogDelete if lineage of these files is important.
Further information

This connector is configured in the exchangeDL01 integration daemon server in the open metadata labs

Raise an issue or comment below