Unity Catalog¶

Unity Catalog is a data manager catalog that governs access to data. It is typically managing data from data lakes and lakehouses where much of the data is in a parquet format, with a table abstraction over the top. Unity catalog also supports folders of files (called Volumes) and functions. Unity catalog is able to provide access to the data in its catalogs, and run the functions.

The picture below shows Unity Catalog managing access to data in Delta Lake.

Internally, unity catalog's metadata is organized into catalogs. (So one way to think of Unity Catalog is as a 'catalog of catalogs'.) Each catalog has multiple schemas and these contain the resources:

• Tables - these are virtual tables, typically backed by an Apache Parquet file.
• Functions - these are callable functions, typically implemented in SQL, but may be a callable external component.
• Volumes - these are collections of files.

As a result of this structure, the resources in Unity Catalog have a three level name: catalogName.schemaName.resourceName.

Unity Catalog Technology Type Names¶

The technology type names (aka deployed implementation types) added to Egeria's reference data for Unity Catalog are:

• Unity Catalog Server - The OSS Unity Catalog (UC) Server is an operational data platform 'catalog of catalogs' that supports controlled access to data managed through a related data platforms.
• Unity Catalog Catalog - An operational data platform catalog within the Unity Catalog (UC) 'catalog of catalogs'.
• Unity Catalog Schema - A schema that organizes data assets for an operational data platform catalog within the Unity Catalog (UC) 'catalog of catalogs'.
• Unity Catalog Table - A relational table within the Unity Catalog (UC) 'catalog of catalogs'.
• Unity Catalog Function - A function found in Unity Catalog (UC) that is working with data.
• Unity Catalog Volume - A collection of related data files within the Unity Catalog (UC) 'catalog of catalogs'.

{
  "class": "TechnologyTypeSummaryListResponse",
  "relatedHTTPCode": 200,
  "elements": [
    {
      "technologyTypeGUID": "2d89345f-2650-4c04-bd5c-8cdbab7a0b79",
      "qualifiedName": "Egeria:ValidMetadataValue:SoftwareServer:deployedImplementationType-(Unity Catalog Server)",
      "name": "Unity Catalog Server",
      "description": "The OSS Unity Catalog (UC) Server is an operational data platform 'catalog of catalogs' that supports controlled access to data managed through a related data platforms.",
      "category": "SoftwareServer:deployedImplementationType"
    },
    {
      "technologyTypeGUID": "2b28dd27-3d4e-4c75-a3e8-cbbcbe8cb62f",
      "qualifiedName": "Egeria:ValidMetadataValue:Catalog:deployedImplementationType-(Unity Catalog Catalog)",
      "name": "Unity Catalog Catalog",
      "description": "An operational data platform catalog within the Unity Catalog (UC) 'catalog of catalogs'.",
      "category": "Catalog:deployedImplementationType"
    },
    {
      "technologyTypeGUID": "c56ca4d1-ed5a-4b05-b75b-e4b6bd3500ff",
      "qualifiedName": "Egeria:ValidMetadataValue:DeployedDatabaseSchema:deployedImplementationType-(Unity Catalog Schema)",
      "name": "Unity Catalog Schema",
      "description": "A schema that organizes data assets for an operational data platform catalog within the Unity Catalog (UC) 'catalog of catalogs'.",
      "category": "DeployedDatabaseSchema:deployedImplementationType"
    },
    {
      "technologyTypeGUID": "3a1ad610-f5c5-4aba-a766-63965ac528be",
      "qualifiedName": "Egeria:ValidMetadataValue:VirtualRelationalTable:deployedImplementationType-(Unity Catalog Table)",
      "name": "Unity Catalog Table",
      "description": "A relational table within the Unity Catalog (UC) 'catalog of catalogs'.",
      "category": "VirtualRelationalTable:deployedImplementationType"
    },
    {
      "technologyTypeGUID": "7f15dd5f-7569-4697-a3f1-491e399f4351",
      "qualifiedName": "Egeria:ValidMetadataValue:DeployedAPI:deployedImplementationType-(Unity Catalog Function)",
      "name": "Unity Catalog Function",
      "description": "A function found in Unity Catalog (UC) that is working with data.",
      "category": "DeployedAPI:deployedImplementationType"
    },
    {
      "technologyTypeGUID": "dbabe8cb-345e-4665-a665-1bef56a26ecd",
      "qualifiedName": "Egeria:ValidMetadataValue:DataFolder:deployedImplementationType-(Unity Catalog Volume)",
      "name": "Unity Catalog Volume",
      "description": "A collection of related data files within the Unity Catalog (UC) 'catalog of catalogs'.",
      "category": "DataFolder:deployedImplementationType"
    }
  ]
}

Open Metadata Type Mapping for Unity Catalog¶

The mapping from Unity Catalog metadata elements to the Open Metadata Types used in the Open Metadata Ecosystem is as follows:

In addition, each of these elements have a PropertyFacet and an External Identifier attached. The property facet contains implementation specific details; the external identifier includes the guid from unity catalog plus other mapping values such as the catalog name, schema name and short name to enable the Unity Catalog connectors to ensure that the name of a element has not changed since the last time a Unity Catalog element was retrieved.

The diagram below illustrates the mapping of the Unity Catalog metadata resource to the Open Metadata Types.

Below is the field mapping from Unity Catalog to these Open Metadata Types:

The templates that implement this mapping are described in Unity Catalog Templates.

Anchor design for Unity Catalog¶

In order to have correct delete semantics, each of the unity catalog resources is its own anchored structure. In addition, each resource is anchored to its parent. So each table, function and volume is anchored to its schema and each schema is anchored to its catalog. The catalogs are anchored to their appropriate server.

The result is, if for example, a catalog is deleted, all the schemas, tables, functions and volumes nested underneath it are deleted too - ensuring there are no orphaned fragments of metadata left in the repository.

Metadata Collections¶

Each catalog in a Unity Catalog server is assigned its own metadata collection. The schemas, tables, functions and volumes within the catalog are all part of the catalog's metadata collection making it easy to identify the origin of these metadata elements.

The unity connectors also use the metadata collections to scope the metadata they are processing.

Unity Catalog Connectors¶

The connectors shipped with Egeria are as follows:

Raise an issue or comment below