Skip to content

Files Metadata

Files are modeled using subtypes of the DataFile and FileFolder assets along with descriptions of the structures of the data that they store.

Optionally, this metadata can be tied to a FileSystem or a FileManager software capability.

CSV Files

Figure 1 shows the types of metadata for CSV files that integration connectors can create.

Figure 1

Figure 1: CSV File metadata

The structure of a CSV file is made up of TabularColumns.

Figure 2 shows the open metadata types used to represent the CSV File metadata.

Figure 2

Figure 2: CSV file open metadata types

If the FileSystem or FileManager is catalogued, link each CSVFile asset to it using the CapabilityAssetUse relationship. Between the file and its TabularColumns sit the Schema relationship, TabularSchemaType entity and AttributeForSchema relationship.

The tabular columns use the TypeEmbeddedAttribute classification method for defining the schema type.

Avro Files

Figure 1 shows the types of metadata for Avro files that integration connectors can create.

Figure 1

Figure 1: Avro File metadata

The structure of an Avro file is made up of nested ObjectAttributes.

Figure 2 shows the open metadata types used to represent the Avro File metadata.

Figure 2

Figure 2: Avro file open metadata types

If the FileSystem or FileManager is catalogued, link each AvroFile asset to it using the CapabilityAssetUse relationship. Between the file and its ObjectAttributes sit the Schema relationship, ObjectSchemaType entity and AttributeForSchema relationship.

The object attributes use the ObjectAttribute subtype to allow for find requests that only return schema information for object type schemas. The schema attributes also use the TypeEmbeddedAttribute classification method for defining the schema type.

JSON Files

Figure 1 shows the types of metadata for JSON files that integration connectors can create.

Figure 1

Figure 1: JSON File metadata supported by the file classifier from the integration context

The structure of a JSON file is made up of nested DocumentSchemaAttributes.

Figure 2 shows the open metadata types used to represent the JSON File metadata.

Figure 2

Figure 2: JSON file open metadata types supported by the file classifier from the integration context

If the FileSystem or FileManager is catalogued, link each JSONFile asset to it using the CapabilityAssetUse relationship. Between the file and its DocumentSchemaAttributes sit the Schema relationship, DocumentSchemaType entity and AttributeForSchema relationship.

The schema attributes use the DocumentSchemaAttribute subtype to allow for find requests that only return schema information for document schemas. The schema attributes also use the TypeEmbeddedAttribute classification method for defining the schema type.


Raise an issue or comment below