Next Release

The Metadata Expert API can now reverse changes, using the undo and restore operations that the repository interface has always offered:

• Undo (undoMetadataElementUpdate and undoRelationshipUpdate) puts back the previous version of a metadata element or relationship. An update is made to a single element, and so is its undo: nothing anchored to the element is changed.
• Restore (restoreMetadataElementInStore) reverses the command that removed an element, whether that was a soft delete or an archive, and reverses all of it. Restoring a database schema that was deleted with cascade brings back its tables, columns and connection, and the relationships the delete removed - including those to elements outside its anchored graph, provided those elements still exist. An element that a separate, earlier delete removed stays deleted. Restore can be requested on any of the elements the command removed. Restoring after an archive removes the Memento classifications and the _archivedOn_ suffix from the qualified names. A purged element cannot be restored.
• restoreRelationshipInStore restores a deleted relationship on its own, provided both of its ends are active.

Restore knows exactly what a command changed because every instance written by a single command - a delete, an archive, a create from a template or a restore - now carries the same lastOperationId in its header. The value is set by the server for each command and cannot be supplied by a caller, unlike the request id, which a caller can share across several calls. It is returned in the element's version information and can be used in searches, so it can also be used to check the scope of any complex command.

Every relationship type now has a relationship purpose that says why it exists: STRUCTURAL, ATTACHMENT, ASSIGNMENT, MEMBERSHIP, ASSOCIATION, PEER, DEFINITION, DEPLOYMENT, DESIGN_LINEAGE, ACTION or ACTIVITY. It is stored in the new relationshipPurpose attribute of the RelationshipDef, where it is informational for the repositories. The Open Metadata Framework defines the values in the OpenMetadataRelationshipPurpose enumeration, sets them on each relationship type in OpenMetadataType, and returns them in OpenMetadataRelationshipDef, so user interfaces can use them to organize the relationships around an element. Operations that start on an anchored element use the purpose to decide which relationships to follow (see below).

Deleting or archiving an element that is anchored to another element now takes that element and what lies downstream of it - the elements at end 2 of its STRUCTURAL and ATTACHMENT relationships that share its anchor, and anything anchored to it - and nothing else. Previously an archive followed relationships in both directions, so archiving a column took every column, the schema and the asset with it. Because an ordinary delete of an element with lineage is turned into an archive, deleting a column with lineage did the same.

• Its siblings and its anchor are never taken, and nor are elements it is linked to for any other purpose, such as the target of a ForeignKey, a lineage relationship, a glossary term relationship or the role subscribed to a notification type.
• An element anchored to another member of the graph - an Endpoint anchored to its Connection, for example - goes with that member.
• Archiving a graph gives every element the same _archivedOn_ suffix.
• Deleting or archiving an anchor still takes its whole graph.

Creating an element from a template now copies the template's anchored graph in the same shape as the template.

• Elements anchored to a member of the template's graph, such as the Endpoint of a Connection, are now copied. Previously they were linked to the copy, so the copy and the template shared them, and deleting the template removed them from the copy. In the copy they are anchored to the copy of the member, as they are in the template.
• When the template is itself a member of another element's graph - a table used as the template for new tables - only the template element and what lies downstream of it are copied. Its links into the template's graph (its parent, its siblings, the elements it links to for purposes other than STRUCTURAL and ATTACHMENT) are not reproduced, and the copy is anchored and attached where the request says. Previously the template's whole graph was copied.

The Open Metadata Access Security Connector now treats every governance zone as secured. A zone that has no security access control in the user directory - or whose control has neither a list for the requested operation nor a DEFAULT list - permits no-one. Previously such a zone was ignored, so a mistyped or undeclared zone name exposed the elements in it. An element with no zones is still open to every active user, and an element in several zones is still accessible if any one of them permits the operation.

To find the zones that need attention, the OMAGServerPlatformCataloguer now checks each GovernanceZone element against the user directory of each platform it catalogues. It raises a new MISSING-ZONE-ACCESS-CONTROL exception on a zone with no security access control, and removes the exception once one has been defined.

The new Governance Engine API (URL marker governance-engine, in development) is for governance engines that run engine actions outside of Egeria. The engine retrieves the engine actions approved to run on it (getApprovedEngineActions, new), claims them, optionally reports their progress, and records their completion with the guards that select the next steps of a governance action process. These are the same calls that Egeria's engine hosts make, from the Open Governance Framework's GovernanceCompletionInterface, so Egeria coordinates these engine actions just as it does those run in an engine host. Five of its operations have moved from the Automated Curation API - see Upgrade notes below.

The Automated Curation API has a new operation, cancelGovernanceActionProcess, that cancels a running governance action process instance, identified by the GUID returned from initiateGovernanceActionProcess. Every engine action of the process instance that has not finished is cancelled, and the instance's completionTime is set if it is not already. It is also on the Open Governance Framework's ActionControlInterface, and governance services can call it through StewardshipAction in their connector context.

The egeria_informationSupplyChain run facet has a new version, 1-1-0, with an iscQualifiedNames list for a run that moves data along several information supply chains, such as a job that publishes a data set several supply chains consume. The Open Lineage Cataloguer creates each lineage relationship, and each DataSetContent link, once for each of the run's supply chains. Events using version 1-0-0, with iscQualifiedName only, are still accepted.

The Darwin Product Dependency Manager now works across all of the layers of lineage - schema detail, the data fabric, the system fabric and the data mesh - rather than rolling the lineage up through three fixed levels. At each boundary between the layers it:

• adds the missing relationships. As well as the data flows between data assets and between software servers, it now derives a data flow between a data asset and a process from the DataMapping between a column and the process; and in the system fabric it derives ProcessCall and ControlFlow relationships as well as DataFlow, between any items of IT infrastructure. A process counts for the infrastructure whose software capability uses it in any way, not only one that owns it.
• validates the lineage that people assert. An asserted DataFlow, ControlFlow or ProcessCall that the layer beneath does not prove is now recorded as an exception, as an unproven DigitalProductDependency already was, against new exception types UnprovenDataFlow, UnprovenControlFlow and UnprovenProcessCall. A relationship is only checked when the layer beneath is catalogued for both of its ends; otherwise it is counted as unchecked, because missing detail is no evidence against it.
• establishes the information supply chain of each relationship wherever the lineage around it shows what it is. A relationship that names no supply chain can now join a path of any supply chain, and is given the path's supply chain once it is known - from another relationship on the path, or from the relationship the path proves in the layer above. Where it lies on the paths of several supply chains, Darwin leaves it untagged and records a copy of it for each.

Darwin also treats a data set and its content (DataSetContent) as the same element, and follows the DataMapping relationships between a relational table and a process, so a digital product whose member is a data set over a table reaches the lineage of that table. A relationship that Darwin created and that anyone else has since updated has been adopted: Darwin no longer changes or removes it, so a person's edits are kept, and validates it like any asserted relationship. New audit log messages DARWIN-PRODUCT-DEPENDENCY-MANAGER-0016 to -0020 report the new exceptions and supply chains, and five new test classes in darwin-fvt cover them.

Every lineage relationship, and the DataSetContent relationship, has a new attribute, derivation, that records where the relationship came from - the rule or process that derived it, for a relationship that was derived rather than asserted. The label of a lineage relationship is left to summarize the data that flows.

• The Darwin Product Dependency Manager records the rule that derived each of its relationships, such as derived from data mappings, and uses it to recognize the relationships it maintains.
• The Open Lineage Cataloguer records the part of the Open Lineage event that each relationship was derived from, such as derived from OpenLineage columnLineage facet, and on each DataSetContent link how the data set was matched to the Open Lineage dataset, such as matched by endpoint and table name.

The attribute is on the LineageRelationshipProperties and DataSetContentProperties beans, and in the Lineage Linker, Asset Maker and Product Manager request samples.

The Open Lineage Cataloguer now uses the relationship type that matches the elements it links. Lineage between assets - two data sets, or a data set and a process - is a DataFlow. Lineage that has a schema attribute such as a relational table at either end - between two tables, or between a table and the process that reads or writes it - is a DataMapping, and a DataFlow is added between the table's asset (its schema or database) and the other end so that the lineage is visible at the asset level too. Previously the cataloguer created DataFlow relationships between tables and processes, and both a DataFlow and a DataMapping between two tables.

A new functional verification test suite, anchor-fvt, exercises anchor management against both the PostgreSQL and the in-memory repositories: the scope of a delete, archive and purge, both of an anchor and of an anchored member, templated creates, restore and undo. Its fixture is a catalogued database schema with its tables, columns, foreign keys, lineage, connection and a linked glossary term.

A new script, open-metadata-test/open-metadata-fvt/run-all-fvts.zsh, runs the unit tests, every functional verification test suite except cts-fvt, and the coverage report in one go. It checks that PostgreSQL is answering before it starts.

• The digital-products governance zone in the shipped user directories (egeria-user-directory and coco-user-directory) had its security lists in the wrong place, so they were ignored and the zone was public. It is now readable by all users and maintained by digital product managers and integration connectors.
• The description of the server access controls in the shipped user directories was written as controlDescription, which is not recognised, so the descriptions were lost when the directory was loaded.
• The egeria-user-directory referred to a dataManagementProcess security role that it did not define.
• A cascaded delete of IT infrastructure, such as a host, deleted the infrastructure a second time instead of the elements deployed on it, and recursed until the server ran out of stack.
• A delete without cascade of a database schema that still had tables was not refused. The anchored members were removed before the check was made, so the schema was found to be empty and deleted.
• The PostgreSQL repository failed with a NullPointerException when a search that included deleted relationships found a relationship whose end had also been deleted.
• Every purge was reported in the audit log as OMAG-REPOSITORY-HANDLER-0001, which says the repository does not support soft-delete. The message is now only issued when that is the reason for the purge.
• cancelEngineAction overwrote the status of an engine action that had already finished, so a completed or failed engine action was recorded as CANCELLED and the record of what it did was lost. An engine action whose status is COMPLETED, INVALID, IGNORED, FAILED, CANCELLED or ABANDONED is now left unchanged.
• The RetentionProperties bean was registered for JSON under the name RetentionProperties.class, so a Retention classification could not be sent in a request.