Crossing Data Boundaries¶

Organizations sort their data into categories, and then manage each category separately - its own tool, its own team, its own vocabulary, often its own budget. The categories are useful; the boundaries between them are where value quietly leaks away, because any question that spans two of them turns into a project.

Egeria's contribution is not to move this data into one place. It is to hold the descriptions of all of it in one knowledge graph, and to link them, so that the relationships that cross the boundaries can be followed.

What gets linked¶

Master data is described by the same open metadata types as everything else. Assets describe the systems and data stores; actor profiles describe the people, teams and organizations; digital products describe what is offered; locations describe where things are, physically and digitally. Because these are ordinary metadata elements, a person can be linked to the role that makes them responsible for an asset, and an asset can be linked to the location and the business capability it serves.

Reference data is managed as valid values and reference data sets, so the code set exists once and every use of it points at the same definition. A data class describes the kind of value a field holds, which allows values encountered in real data to be recognized against the reference set rather than eyeballed.

Metadata is the connective tissue: the schemas describing the structure of each store, the lineage describing how data moves between them, the information supply chains describing those flows in business terms, and the governance definitions describing how each must be handled.

Analytics and insight come back into the graph rather than accumulating outside it. Survey reports hold the measurements and profiles produced by analysing resources, and their annotations attach to the assets they describe, so the analysis of a data set is found next to the data set.

The joins that make it work¶

Linking categories requires something to join on. Egeria provides several, and they are worth knowing because they are the mechanisms that make cross-boundary questions answerable:

• Glossary terms and data classes are the semantic join. When the same term is attached to a field in an operational store, a field in a warehouse table and a data field in a project's data specification, those three things are known to mean the same thing - even though nothing about their names or types agrees.
• Data fields in a data dictionary are the field-level join across stores. This is what a data sharing hub is built on: one data field identifies similar data wherever it appears, so a consumer describes what they want once rather than once per store.
• External identifiers are the identity join across tools. The same resource is known by different names in different systems; recording those names against one asset means an automated process can look up whichever identifier the tool it is calling expects.
• Reference values are the coding join. Shared code sets let data from different sources be compared and combined without a translation step being invented each time.
• Lineage and information supply chains are the flow join, connecting a transactional source to the analytics eventually derived from it, through every hop in between.

What this makes possible¶

Once the links exist, questions that used to require an investigation become queries:

• Which digital products depend on data originating in a particular country - and therefore which are affected by a change in that jurisdiction's rules?
• Which analytics and reports are built on a data set that has just been reclassified as confidential?
• Which teams and roles need to be told when a system is retired, because they own something downstream of it?
• Where else does this data field appear, and do those copies agree with the authoritative source?

The same links are what allow automation to act. A governance action can navigate from a classification to the resources it applies to, and from those resources to the people responsible, without any of that navigation being hard-coded.

Sharing across the boundary in both directions¶

Linking descriptions is one half; moving data across the boundary is the other. The Harvest and Publish pattern packages open metadata - including the master data, reference data and insight described above - into digital products delivered as tabular data sets into whatever the consuming team already uses: a CSV file, a database table, a Kafka topic. Reference data curated in Egeria can therefore be distributed to the applications that need it, and insight derived from the ecosystem can be delivered to the analysts who will act on it.

The graph below is one such family from Egeria's own product catalog - the master data of the ecosystem, published as products that other tools can subscribe to:

---
title: DigitalProductFamily - Actor, Places and Product Master Data [7fc04656-52eb-433f-bf4a-94a7c6735b18]
---
flowchart TD
%%{init: {"flowchart": {"htmlLabels": false}} }%%

1@{ shape: rounded, label: "*Digital Product Family*
**Actor, Places and Product Master Data**"}
2@{ shape: rect, label: "*Digital Product Family*
**Open Metadata Digital Products**"}
2==>|"Collection Membership"|1
3@{ shape: rect, label: "*Digital Product*
**Digital Product List**"}
1==>|"includes product [Collection Membership]"|3
4@{ shape: cyl, label: "*Reference Code Table*
**Data set for Digital Product List**"}
3==>|"product data set [Collection Membership]"|4
5@{ shape: rect, label: "*Digital Product*
**List of Locations**"}
1==>|"includes product [Collection Membership]"|5
6@{ shape: cyl, label: "*Reference Code Table*
**Data set for List of Locations**"}
5==>|"product data set [Collection Membership]"|6
7@{ shape: rect, label: "*Digital Product*
**Organization List**"}
1==>|"includes product [Collection Membership]"|7
8@{ shape: cyl, label: "*Reference Code Table*
**Data set for Organization List**"}
7==>|"product data set [Collection Membership]"|8
9@{ shape: rect, label: "*Digital Product*
**People List**"}
1==>|"includes product [Collection Membership]"|9
10@{ shape: cyl, label: "*Reference Code Table*
**Data set for People List**"}
9==>|"product data set [Collection Membership]"|10
11@{ shape: rounded, label: "*Community*
**Open Metadata Master Data special interest group**"}
11==>|"Scoped By"|1
12@{ shape: trap-t, label: "*Digital Product Manager*
**Product Manager for Actor, Places and Products**"}
12==>|"Owner [Assignment Scope]"|1
style 11 color:#000000, fill:#39add1, stroke:#004563
style 1 color:#000000, fill:#e0ab18, stroke:#004563
style 12 color:#000000, fill:#FFE599, stroke:#000000
style 2 color:#000000, fill:#838cc7, stroke:#3079ab
style 3 color:#000000, fill:#838cc7, stroke:#3079ab
style 4 color:#000000, fill:#BDB76B, stroke:#004563
style 5 color:#000000, fill:#838cc7, stroke:#3079ab
style 6 color:#000000, fill:#BDB76B, stroke:#004563
style 7 color:#000000, fill:#838cc7, stroke:#3079ab
style 8 color:#000000, fill:#BDB76B, stroke:#004563
style 9 color:#000000, fill:#838cc7, stroke:#3079ab
style 10 color:#000000, fill:#BDB76B, stroke:#004563

Actor, Places and Product Master Data: Each product in this folder lists information held in the open metadata about people, organizations, users, teams, locations and digital products. This type of data is called master data because it describes the key entities that the organization operates around. As such, some form of this data appears in most data sets. Each product in this folder is organized into a tabular data set, where each row is a specific (master data) entity. These are designed to be used as standard values that can be used for validation or to ensure that data in digital products is consistent making it easier to join data from multiple products. (Extracted from 6.0-SNAPSHOT)

Raise an issue or comment below