Exchanging Asset Descriptions¶

An asset describes a digital resource that is of value to the organization: a data set, a file, a database, an API. Many organizations need to exchange the descriptions of their assets with others. A government agency is obliged to list the data that it makes available to the public; a research organization shares its data sets with its partners; an enterprise harvests the open data it relies on into its own catalog so that it can be found, governed and traced alongside its internal data.

Egeria exchanges these descriptions using the Data Catalog Vocabulary (DCAT). Descriptions written by other organizations are catalogued as assets in open metadata, and the assets that an organization chooses to make public are written out as DCAT descriptions, along with a DCAT catalog that lists them, ready to be harvested by a data portal.

The Data Catalog Vocabulary (DCAT)¶

The W3C Data Catalog Vocabulary (DCAT) is the standard that governments and many other organizations use to describe the data assets that they make available: what each dataset is about, who publishes it, how it is licensed and where it can be obtained. It is the basis of profiles such as DCAT-AP in Europe and DCAT-US in the United States. DCAT describes:

a dataset (dcat:Dataset) - a collection of data that is published or curated by a single organization;
a distribution (dcat:Distribution) - a specific form in which a dataset can be obtained, such as a CSV file to download or an API to call;
a dataset series (dcat:DatasetSeries) - a group of datasets that are published separately but belong together, such as the annual editions of a survey;
a data service (dcat:DataService) - an API that provides access to one or more datasets;
a catalog (dcat:Catalog) - a curated list of datasets, data services and other catalogs.

Egeria supports DCAT version 3 in its JSON-LD serialization. Documents in Turtle or RDF/XML need to be converted to JSON-LD before they are passed to Egeria. Any arrangement of JSON-LD can be read: nested, flattened, with prefixes or with full IRIs. Egeria never retrieves a JSON-LD context from the network. A document should embed its context; one that refers to a remote context is read using the standard prefixes and names of the DCAT terms. This is enough to read the data.json catalogs published by US government agencies.

Mapping to open metadata¶

The resources described in a DCAT document are catalogued as follows.

DCAT	Open metadata
`dcat:Dataset`	DataSet asset
`dcat:Distribution`	DataFile asset (if it has a `dcat:downloadURL`) or DataStore asset (if it only has a `dcat:accessURL`), anchored to the data set and linked to it by a DataSetContent relationship. A distribution that is removed from the dataset's description is deleted.
`dcat:DatasetSeries`	DataSet asset linked to its member data sets by DataSetContent relationships
`dcat:DataService`	DeployedAPI asset, with an Endpoint for each `dcat:endpointURL`, linked to the data sets it serves by DataSetContent relationships
`dcat:Catalog`	CollectionFolder whose members are the catalog's datasets, data services and nested catalogs
`dct:title`, `dct:description`, `dct:identifier`, `dcat:version`, `dcat:landingPage`	displayName, description, identifier, versionIdentifier, url
`dcat:keyword`	SearchKeyword
`dct:publisher`	Organization linked by a ScopedBy relationship
`dct:creator`	authors
`dcat:contactPoint`	ContactDetails linked by a ContactThrough relationship
`dct:license`	LicenseType linked by a License relationship
`dct:conformsTo`, `foaf:page`	ExternalReference linked by an ExternalReferenceLink relationship labelled `conformsTo` or `documentation`
Everything else	Additional properties whose names begin `dcat.`

Each element is given a qualified name built from the IRI of the resource it represents (or the resource's identifier or title if it has no IRI), so sending a description again updates the elements rather than creating new ones. Properties that are not part of DCAT, such as those added by the DCAT-AP and DCAT-US profiles, are preserved in the dcat.otherProperties additional property and written out again when the asset is published.

Receiving descriptions¶

A DCAT document reaches Egeria in one of four ways:

In a file - the DCAT Files Receiver scans directories for .jsonld and .json files that describe DCAT resources. By default, it scans the loading-bay/dcat directory under the platform's working directory. Other directories are added by attaching their file folders to the connector as directoryToMonitor catalog targets.
On an Apache Kafka topic - the DCAT Event Receiver treats each event received from the topics attached to it as topicToMonitor catalog targets as a DCAT document.
Through an integration daemon's REST API - a document is sent to the integration daemon through its publish-dcat-document request, or through the publish-document request of the Asset Catalog OMVS.
Directly through the Asset Catalog OMVS - the import-document request catalogs a document straight away, without an integration daemon, and returns the unique identifiers of the elements that describe its resources.

In the first three cases, the document is passed to the DCAT listeners registered in the integration daemon. The DCAT Cataloguer is such a listener: it catalogs what each document describes, using the mapping above.

Publishing descriptions¶

The DCAT Publisher is told which assets to publish through a collection: the publishing catalog. An asset is published if it is a member of this collection, or of a collection folder nested in it. By default, the publishing catalog is the DCAT Publishing Catalog root collection (Egeria::DCATPublishingCatalog) that is supplied by the Core Content Pack. To publish an asset, add it to this collection; to withdraw it, remove it. Collection folders nested in the publishing catalog are a way to organize the published assets.

On each refresh, the publisher writes the following files to each of its destination directories. By default, there is one destination: the distribution-hub/dcat-catalog directory under the platform's working directory. Other directories are added by attaching their file folders to the connector as publishDirectory catalog targets. A file is only written when its content has changed.

catalog.jsonld - the publishing catalog as a dcat:Catalog, along with the description of every published asset.
datasets/<name>.jsonld - one file for each asset published as a dcat:Dataset.
data-services/<name>.jsonld - one file for each asset published as a dcat:DataService.

The type of the asset determines how it is described:

A DeployedAPI asset is published as a data service.
A DataSet asset is published as a dataset whose distributions are the assets that supply its data.
Any other type of asset is published as a dataset with a single distribution.

An asset that was catalogued from a DCAT description keeps its original IRI; any other asset is identified by urn:uuid: followed by its unique identifier.

The publisher has two configuration properties:

Configuration property	Description
`publishingCatalog`	The qualified name (or display name) of the collection that lists the assets to publish. The default is `Egeria::DCATPublishingCatalog`.
`defaultPublisher`	The name of the organization to show as the publisher of an asset that is not linked to an organization.

The DCAT description of a single asset, whether or not it is in the publishing catalog, is returned by the dcat-document request of the Asset Catalog OMVS.

Egeria's DCAT support¶

The Open Integration Framework (OIF) provides Java beans for catalogs, datasets, dataset series, distributions and data services, a formatter that reads and writes their JSON-LD serialization, the mapping between the documents and open metadata, and a distribution mechanism in the integration daemon that mirrors its Open Lineage support:

An integration connector may register a DCAT listener with its integration context to receive every document that is published to the daemon.
An integration connector may publish a document for the other listeners registered in the same daemon.
The integration daemon's REST API (publish-dcat-document) and the Asset Catalog OMVS allow external callers to publish documents into the daemon.

The DCAT connectors supplied by Egeria are packaged in the DCAT Content Pack, in an integration group called Egeria:IntegrationGroup:DCAT:

The DCAT Files Receiver scans directories and publishes the new and changed documents it finds.
The DCAT Event Receiver publishes the documents it receives from Apache Kafka topics.
The DCAT Cataloguer registers a listener and catalogs each document it receives.
The DCAT Publisher writes the DCAT description of each asset in the publishing catalog, and of the catalog itself.

An organization can therefore drop the catalog published by a partner into a directory and have its datasets appear as assets in open metadata, while the assets that its own teams add to the publishing catalog are written out in the standard format for others to harvest.

Raise an issue or comment below