Skip to content
Stable

This component is complete and can be used. The interfaces will be supported until the function is removed from the project via the deprecation process. There will be ongoing extensions to this function, but it will be done to ensure backward compatibility as far as possible. If there is a need to break backward compatibility, this will be discussed and reviewed in the community, with a documented timeline.

Files Integrator Open Metadata Integration Service (OMIS)

The Files Integrator OMIS supports the collection of information about files stored in a filesystem for and then storing it in the open metadata ecosystem.

It supports a type of integration connector that are able to create DataFile and FileFolder assets of various types along with descriptions of the structures of the data that they store.

Optionally, this metadata can be tied to an FileSystem or a FileSystem.

CSV Files

Figure 1 shows the types of metadata for CSV files that integrators connectors can create with this integration service.

Figure 1

Figure 1: CSV File metadata supported by the Files Integrator OMIS

The structure of a CSV file is made up of TabularColumns.

Figure 2 shows the open metadata types used to represent the CSV File metadata created through this service.

Figure 2

Figure 2: CSV file open metadata types supported by the Files Integrator OMIS

If a FileSystem or FileManager is defined, any CSVFile asset created is automatically linked to it using the ServerAssetUse relationship. When a TabularColumn is created for the file, the service automatically inserts the AssetSchemaType relationship, TabularSchemaType entity and AttributeForSchema relationship in between the file and the TabularColumn.

The tabular columns use the TypeEmbeddedAttribute classification method for defining the schema type.

Avro Files

Figure 1 shows the types of metadata for Avro files that integrators connectors can create with this integration service.

Figure 1

Figure 1: Avro File metadata supported by the Files Integrator OMIS

The structure of an Avro file is made up of nested ObjectAttributes.

Figure 2 shows the open metadata types used to represent the Avro File metadata created through this service.

Figure 2

Figure 2: Avro file open metadata types supported by the Files Integrator OMIS

If a FileSystem or FileManager is defined, any AvroFile asset created is automatically linked to it using the ServerAssetUse relationship. When an ObjectAttribute is created for the file, the service automatically inserts the AssetSchemaType relationship, ObjectSchemaType entity and AttributeForSchema relationship in between the file and the ObjectAttribute.

Any object attributes created through this interface will use the ObjectAttribute subtype to allow for find requests that only return schema information for object type schemas. The schema attributes also use the TypeEmbeddedAttribute classification method for defining the schema type.

JSON Files

Figure 1 shows the types of metadata for JSON files that integrators connectors can create with this integration service.

Figure 1

Figure 1: JSON File metadata supported by the Files Integrator OMIS

The structure of an JSON file is made up of nested DocumentSchemaAttributes.

Figure 2 shows the open metadata types used to represent the JSON File metadata created through this service.

Figure 2

Figure 2: JSON file open metadata types supported by the Files Integrator OMIS

If a FileSystem or FileManager is defined, any JSONFile asset created is automatically linked to it using the ServerAssetUse relationship. When a DocumentSchemaAttribute is created for the file, the service automatically inserts the AssetSchemaType relationship, DocumentSchemaType entity and AttributeForSchema relationship in between the file and the ObjectAttribute.

Any object attributes created through this interface will use the DocumentSchemaAttribute subtype to allow for find requests that only return schema information for object type schemas. The schema attributes also use the TypeEmbeddedAttribute classification method for defining the schema type.

Module Implementation

The modules are as follows:

  • files-integrator-api - defines the interface for an integration connector that is supported by the Files Integrator OMIS. This includes the implementation of the context that wraps the Data Manager OMAS's clients.

  • files-integrator-server - implements the context manager for the Files Integrator OMIS.

  • files-integrator-spring - implements a rest API for validating that a specific integration connector is able to run under this service.

  • files-integrator-client - implements a Java client for the REST API.

This integration service is paired with the Data Manager Open Metadata Access Service (OMAS).

Raise an issue or comment below