Next Release

The Metadata Expert API can now reverse changes, using the undo and restore operations that the repository interface has always offered:

• Undo (undoMetadataElementUpdate and undoRelationshipUpdate) puts back the previous version of a metadata element or relationship. An update is made to a single element, and so is its undo: nothing anchored to the element is changed.
• Restore (restoreMetadataElementInStore) reverses the command that removed an element, whether that was a soft delete or an archive, and reverses all of it. Restoring a database schema that was deleted with cascade brings back its tables, columns and connection, and the relationships the delete removed - including those to elements outside its anchored graph, provided those elements still exist. An element that a separate, earlier delete removed stays deleted. Restore can be requested on any of the elements the command removed. Restoring after an archive removes the Memento classifications and the _archivedOn_ suffix from the qualified names. A purged element cannot be restored.
• restoreRelationshipInStore restores a deleted relationship on its own, provided both of its ends are active.

Restore knows exactly what a command changed because every instance written by a single command - a delete, an archive, a create from a template or a restore - now carries the same lastOperationId in its header. The value is set by the server for each command and cannot be supplied by a caller, unlike the request id, which a caller can share across several calls. It is returned in the element's version information and can be used in searches, so it can also be used to check the scope of any complex command.

Every relationship type now has a relationship purpose that says why it exists: STRUCTURAL, ATTACHMENT, ASSIGNMENT, MEMBERSHIP, ASSOCIATION, PEER, DEFINITION, DEPLOYMENT, DESIGN_LINEAGE, ACTION or ACTIVITY. It is stored in the new relationshipPurpose attribute of the RelationshipDef, where it is informational for the repositories. The Open Metadata Framework defines the values in the OpenMetadataRelationshipPurpose enumeration, sets them on each relationship type in OpenMetadataType, and returns them in OpenMetadataRelationshipDef, so user interfaces can use them to organize the relationships around an element. Operations that start on an anchored element use the purpose to decide which relationships to follow (see below).

Every classification type now has a classification purpose that says why it exists: MANAGEMENT, ANALYTICAL, GOVERNANCE, GOVERNANCE_IMPLEMENTATION, CATEGORY or DESCRIPTIVE. It is stored in the new classificationPurpose attribute of the ClassificationDef, where it is informational for the repositories. The Open Metadata Framework defines the values in the OpenMetadataClassificationPurpose enumeration, sets them on each classification type in OpenMetadataType, and returns them in OpenMetadataClassificationDef, so user interfaces can show each audience the classifications that matter to it.

Deleting or archiving an element that is anchored to another element now takes that element and what lies downstream of it - the elements at end 2 of its STRUCTURAL and ATTACHMENT relationships that share its anchor, and anything anchored to it - and nothing else. Previously an archive followed relationships in both directions, so archiving a column took every column, the schema and the asset with it. Because an ordinary delete of an element with lineage is turned into an archive, deleting a column with lineage did the same.

• Its siblings and its anchor are never taken, and nor are elements it is linked to for any other purpose, such as the target of a ForeignKey, a lineage relationship, a glossary term relationship or the role subscribed to a notification type.
• An element anchored to another member of the graph - an Endpoint anchored to its Connection, for example - goes with that member.
• Archiving a graph gives every element the same _archivedOn_ suffix.
• Deleting or archiving an anchor still takes its whole graph.

Creating an element from a template now copies the template's anchored graph in the same shape as the template.

• Elements anchored to a member of the template's graph, such as the Endpoint of a Connection, are now copied. Previously they were linked to the copy, so the copy and the template shared them, and deleting the template removed them from the copy. In the copy they are anchored to the copy of the member, as they are in the template.
• When the template is itself a member of another element's graph - a table used as the template for new tables - only the template element and what lies downstream of it are copied. Its links into the template's graph (its parent, its siblings, the elements it links to for purposes other than STRUCTURAL and ATTACHMENT) are not reproduced, and the copy is anchored and attached where the request says. Previously the template's whole graph was copied.

The Open Metadata Access Security Connector now treats every governance zone as secured. A zone that has no security access control in the user directory - or whose control has neither a list for the requested operation nor a DEFAULT list - permits no-one. Previously such a zone was ignored, so a mistyped or undeclared zone name exposed the elements in it. An element with no zones is still open to every active user, and an element in several zones is still accessible if any one of them permits the operation.

To find the zones that need attention, the OMAGServerPlatformCataloguer now checks each GovernanceZone element against the user directory of each platform it catalogues. It raises a new MISSING-ZONE-ACCESS-CONTROL exception on a zone with no security access control, and removes the exception once one has been defined.

The Open Metadata Access Security Connector now tells the server which governance zones each user may read, and the server adds them to the repository query of every search. Previously every element that matched a search was retrieved, and those the user could not read were removed afterwards, so a search by a user who could read few of the matching elements read far more than it returned. Each element returned is still checked by the security connector, so the results are the same. An element in no zone is still returned to every user. The filter is on by default; set the security connector's new pushDownZoneFilter configuration property to false to turn it off.

The filter uses the zones held in each element's Anchors classification: the element's own zones, or its anchor's if it has none. The server now keeps these up to date whenever an element's or its anchor's zones change. To refresh the Anchors classifications that were written before, or that are loaded from an open metadata archive, the Core Content Pack has a new governance action service, the Anchors Zone Resynchronizer, run through the new resynchronize-anchors-zones request type of the Egeria Governance Engine. With no action targets, it checks every Anchors classification in the repository and corrects those that differ; with action targets, it checks each target and, for an anchor, its members. It completes with anchors-zones-corrected, anchors-zones-up-to-date or, if any element could not be corrected, anchors-zones-resync-failed. The same check is available as the new resynchronizeAnchorsZoneMembership operation of the Open Metadata Store. Corrections to reference copies are sent to the cohort member that owns them. See Upgrade notes below.

The IN comparison operator is back in the property search conditions, supported by the in-memory and PostgreSQL repositories. An array property matches if any of its elements is in the list.

The new Governance Engine API (URL marker governance-engine, in development) is for governance engines that run engine actions outside of Egeria. The engine retrieves the engine actions approved to run on it (getApprovedEngineActions, new), claims them, optionally reports their progress, and records their completion with the guards that select the next steps of a governance action process. These are the same calls that Egeria's engine hosts make, from the Open Governance Framework's GovernanceCompletionInterface, so Egeria coordinates these engine actions just as it does those run in an engine host. Five of its operations have moved from the Automated Curation API - see Upgrade notes below.

The Automated Curation API has a new operation, cancelGovernanceActionProcess, that cancels a running governance action process instance, identified by the GUID returned from initiateGovernanceActionProcess. Every engine action of the process instance that has not finished is cancelled, and the instance's completionTime is set if it is not already. It is also on the Open Governance Framework's ActionControlInterface, and governance services can call it through StewardshipAction in their connector context.

The egeria_informationSupplyChain run facet has a new version, 1-1-0, with an iscQualifiedNames list for a run that moves data along several information supply chains, such as a job that publishes a data set several supply chains consume. The Open Lineage Cataloguer creates each lineage relationship, and each DataSetContent link, once for each of the run's supply chains. Events using version 1-0-0, with iscQualifiedName only, are still accepted.

The Darwin Product Dependency Manager now works across all of the layers of lineage - schema detail, the data fabric, the system fabric and the data mesh - rather than rolling the lineage up through three fixed levels. At each boundary between the layers it:

• adds the missing relationships. As well as the data flows between data assets and between software servers, it now derives a data flow between a data asset and a process from the DataMapping between a column and the process; and in the system fabric it derives ProcessCall and ControlFlow relationships as well as DataFlow, between any items of IT infrastructure. A process counts for the infrastructure whose software capability uses it in any way, not only one that owns it.
• validates the lineage that people assert. An asserted DataFlow, ControlFlow or ProcessCall that the layer beneath does not prove is now recorded as an exception, as an unproven DigitalProductDependency already was, against new exception types UnprovenDataFlow, UnprovenControlFlow and UnprovenProcessCall. A relationship is only checked when the layer beneath is catalogued for both of its ends; otherwise it is counted as unchecked, because missing detail is no evidence against it.
• establishes the information supply chain of each relationship wherever the lineage around it shows what it is. A relationship that names no supply chain can now join a path of any supply chain, and is given the path's supply chain once it is known - from another relationship on the path, or from the relationship the path proves in the layer above. Where it lies on the paths of several supply chains, Darwin leaves it untagged and records a copy of it for each.

Darwin also treats a data set and its content (DataSetContent) as the same element, and follows the DataMapping relationships between a relational table and a process, so a digital product whose member is a data set over a table reaches the lineage of that table. A relationship that Darwin created and that anyone else has since updated has been adopted: Darwin no longer changes or removes it, so a person's edits are kept, and validates it like any asserted relationship. New audit log messages DARWIN-PRODUCT-DEPENDENCY-MANAGER-0016 to -0020 report the new exceptions and supply chains, and five new test classes in darwin-fvt cover them.

Every lineage relationship, and the DataSetContent relationship, has a new attribute, derivation, that records where the relationship came from - the rule or process that derived it, for a relationship that was derived rather than asserted. The label of a lineage relationship is left to summarize the data that flows.

• The Darwin Product Dependency Manager records the rule that derived each of its relationships, such as derived from data mappings, and uses it to recognize the relationships it maintains.
• The Open Lineage Cataloguer records the part of the Open Lineage event that each relationship was derived from, such as derived from OpenLineage columnLineage facet, and on each DataSetContent link how the data set was matched to the Open Lineage dataset, such as matched by endpoint and table name.

The attribute is on the LineageRelationshipProperties and DataSetContentProperties beans, and in the Lineage Linker, Asset Maker and Product Manager request samples.

The Open Lineage Cataloguer now uses the relationship type that matches the elements it links. Lineage between assets - two data sets, or a data set and a process - is a DataFlow. Lineage that has a schema attribute such as a relational table at either end - between two tables, or between a table and the process that reads or writes it - is a DataMapping, and a DataFlow is added between the table's asset (its schema or database) and the other end so that the lineage is visible at the asset level too. Previously the cataloguer created DataFlow relationships between tables and processes, and both a DataFlow and a DataMapping between two tables.

An information supply chain retrieved through the Solution Architect API with its implementation now comes with five mermaid graphs instead of one, because a single graph of a large supply chain was often too big for mermaid to display:

• iscImplementationMermaidGraph - the supply chain's context, the design of its solution components and the status of their implementation. The Context, Design and Status areas are now drawn left to right.
• iscDataMeshMermaidGraph - the digital products and the dependencies between them.
• iscSystemFabricMermaidGraph - the IT infrastructure and software capabilities that the supply chain runs on.
• iscDataFabricMermaidGraph - the data and processes linked by data lineage (DataLineageRelationship and its subtypes), ControlFlow and ActionRequester relationships.
• iscDataMappingMermaidGraph - the elements linked by DataMapping relationships, such as the columns of tables.

Each layer of lineage is drawn inside a subgraph. The solution components linked to its elements, through ImplementedBy for example, are drawn outside the subgraph with their links, and so are the digital products linked to the data fabric. An element's anchor, if it is part of the supply chain, is linked to it with a line labelled Anchor for: every anchor in the data mapping graph, solution components in each of the other layers, and digital products in the data mesh. The sample information supply chain in the Core Content Pack, SAMPLE Information Supply Chain - Deployment Status Demonstration, now shows every graph.

Retrieving the lineage graph of an asset through the Asset Catalog API could take minutes on a large estate, because the walk followed lineage as far as it went and read every element it reached in full. In the CocoPharma quickstart, one schema's graph reached 1,057 elements up to 42 hops away and returned 52 MB. The walk is now bounded and much lighter:

• A hop limit. The request body has a new maxHops field: the number of lineage relationships followed away from the starting asset, counted separately upstream and downstream. It defaults to 8, including when there is no request body; 0 means no limit. Each element in the graph has two new fields: hopsFromStart, its distance from the starting asset, and hopLimitReached, set when the lineage continues past it but the walk stopped there. The edge graph does not show such an element as an ultimate source or destination.
• Lighter elements. The elements linked to the starting asset are read without their own relationships. The edges of the graph are in lineageRelationships, and in each element's upstream and downstream relationships.
• No duplicates. An element that is both upstream and downstream of the starting asset appears once. lineageRelationships only links elements that are in the graph.
• Choosing the relationships to follow. includeOnlyRelationships replaces the default relationship types, and a supertype name now includes all of its subtypes, so ["LineageRelationship"] follows every lineage relationship. skipRelationships, previously ignored by the walk, removes types and their subtypes from the default or requested list, for example ["DataSetContent"].
• One query per element. The lineage of each element is retrieved with a single query for LineageRelationship when every type being followed is one of its subtypes, as the defaults now are.

The DataSetContent and DerivedSchemaTypeQueryTarget relationship types are now subtypes of LineageRelationship, so they gain its label, description, iscQualifiedName and derivation attributes. The DataSetContentProperties and DerivedSchemaTypeQueryTargetProperties beans extend LineageRelationshipProperties, and DataSetContent relationships can be created through the Lineage Linker.

The mermaid graph returned with an element as an OpenMetadataRootElement (its mermaidGraph) is now a stable picture of the element's context. It leaves out the relationships whose relationship purpose is Action or Activity, which accumulate as governance actions are run, actions are requested and their results recorded, and which made the graph of a busy element grow without end. Two new graphs show them instead:

• actionMermaidGraph - the element with the relationships attached to it whose purpose is Action, such as the targets of a governance action or the subscribers to a notification.
• activityMermaidGraph - the element with the relationships attached to it whose purpose is Activity, such as the actions that have been requested of, or taken on, the element.

Each shows the elements nested below these relationships too, and is null when the element has none. The purpose is the one Egeria's type system declares for the relationship type. A relationship type added by an extension archive takes the purpose of its nearest Egeria super type, and is drawn in the context graph if it has none. See Context, action and activity graphs.

A new functional verification test suite, anchor-fvt, exercises anchor management against both the PostgreSQL and the in-memory repositories: the scope of a delete, archive and purge, both of an anchor and of an anchored member, templated creates, restore and undo. Its fixture is a catalogued database schema with its tables, columns, foreign keys, lineage, connection and a linked glossary term.

A new script, open-metadata-test/open-metadata-fvt/run-all-fvts.zsh, runs the unit tests, every functional verification test suite except cts-fvt, and the coverage report in one go. It checks that PostgreSQL is answering before it starts.

New types record a statement of applicability - which controls from a standard, such as the Annex A controls of ISO/IEC 27001, apply to an organization, why, and how far each one has been implemented:

• StatementOfApplicability is a new entity, a subtype of AuthoredReferenceable, with scope and reviewDate attributes.
• ControlApplicability is a new relationship between a statement of applicability and the governance definition that describes a control. Its attributes are isApplicable, justification, inclusionReasons, implementationStatus, implementationDescription, reviewDate, label and description.
• ControlImplementationStatus is a new enumeration with the values NOT_STARTED, PLANNED, PARTIALLY_IMPLEMENTED, IMPLEMENTED and OTHER.

They are maintained through new operations on the Security Officer API, using the new StatementOfApplicabilityProperties and ControlApplicabilityProperties beans and the StatementOfApplicabilityHandler. The Core Content Pack now includes a CertificationType for ISO/IEC 27001:2022.

New annotation types record the results of security testing, described in the new model 0665 Security Annotations:

• SecurityFindingAnnotation - a weakness found by a security test, with its severity, CVSS score and vector, CVE and CWE identifiers, where it was found, the evidence, the recommended remediation and whether an exploit is publicly available.
• VulnerabilityAnnotation - a known vulnerability in the software used by a resource, with the affected package URL, the installed and fixed versions, and whether a fix is available.
• PenetrationTestFindingAnnotation - a weakness that a penetration test exploited, with the attack vector, the techniques used, the privileges gained, the resources compromised and the steps to reproduce it.
• SecurityPostureAnnotation - a type of ResourceMeasureAnnotation that summarizes a test run: the open findings by severity, the resources in scope and tested, and the new and resolved findings.

They are catalogued, like other annotations, through the Data Discovery API, using the new SecurityFindingAnnotationProperties, VulnerabilityAnnotationProperties, PenetrationTestFindingAnnotationProperties and SecurityPostureAnnotationProperties beans.

A new model, 0582 Report Specifications, describes how reports are designed and how the measurements in them are produced. ReportType moves to this model from 0580.

• ReportSpec is a new entity, a subtype of ReportType, for reports about open lineage and related data. Its attributes are microflow, targetTypeName, aliases, outputTypes and includePreamble.
• MeasurementSpecification is a new entity, a subtype of DataValueSpecification, that describes a measurement in a report. Its attributes, completenessDeclared, confidenceForm, uncertaintyForm and absenceSentences, let a renderer say "at least N" rather than "N", qualify how the value was produced, and show a sentence instead of a value that is missing.
• ReportTypeInstance is a new relationship between a report type and a report generated from it.
• BasisDataField and AsOfDataField are new relationships between a measurement specification and the data fields that describe its basis (such as the denominator of a percentage) and the time of the measurement.

Related changes to the existing models:

• Report has new reportParameters and outputFormat attributes, and the new Dashboard entity, a subtype of DataSet, is a collection of reports with aliases, refreshTimeInterval and layoutStyle.
• DisplayDataContainer and DisplayDataField have new placementKind, span, presentationStyle and markdownContent attributes, and DisplayDataContainer also has renderer.
• ReportRunOf is a new relationship between a results set and a report generated from it, with resultState, resultStateReason and resultLocation.
• SchemaTypeDefinition and SchemaAttributeDefinition now allow a data structure or data field to be implemented by many schema types or schema attributes - for example, the same table in several databases.

The new relationships are maintained through new operations on the Data Designer API: linkBasisDataField, linkAsOfDataField, linkReportTypeInstance and linkReportRunOf, each with a detach operation. OpenMetadataRootElement returns them in new fields - basisDataField, providesBasisForMeasurements, asOfDataField, providesAsOfTimeForMeasurements, derivedFromReportType, reportTypeInstances, usesResultsSets and usedInReports.

• The Core Content Pack now includes the six STRIDE threats as Threat definitions, such as Threat::STRIDE::Spoofing, so that every threat model links its risks to the same six threats. The STRIDE template for Dr.Egeria refers to them.
• A new Security Content Pack contains the NIST Cybersecurity Framework (CSF) 2.0 and NIST SP 800-53 Rev. 5 control catalogs: 106 CSF subcategories and 1,014 SP 800-53 controls and control enhancements, each a GovernanceControl organized in folders by function, category and control family. It is not loaded automatically. The catalogs are generated from NIST's public-domain OSCAL catalogs.

Karma points are now awarded by the Simon Contribution Recorder, a new nanny integration connector in the OpenMetadataObservability integration group of the Core Content Pack. It replaces the Award Karma Points Lovelace watchdog action service, which has been removed.

Like the Pepys Activity Observer, Simon gathers its observations in memory as the events arrive, and writes them on each refresh (every 12 minutes by default), so a busy user's contribution record is updated once per refresh rather than once for every change they make. The users it ignores are set by its ignoredUserIdPatterns configuration property, which the Core Content Pack sets to .*npa and .*engine, so only people are awarded karma points. See how points are awarded.

When a content pack is written, any element or relationship that was in its previous edition but is no longer in it is now added to the new edition in the DELETED status, with a higher version. Loading the new edition into a repository that loaded an earlier one soft-deletes the element, rather than leaving it behind. The deleted versions are carried into every later edition, so a repository that skips an edition still receives the deletion. This is how the Award Karma Points service and its definitions are removed from existing repositories.

The PostgreSQL repository adds new indexes to its tables when the server starts, so that reading the current version of an element is no longer slowed down by the element's history. On a copy of a large repository, reading an element with 238,795 versions went from 131 ms to 0.05 ms, and retrieving the current relationships of an entity with 15,812 relationship versions went from 9.9 ms to 0.25 ms.

The numeric and boolean properties of the Open Metadata Framework property beans are now Integer, Long, Float and Boolean with a default of null, rather than int, long, float and boolean with a default of 0 or false. A value that is not set is no longer stored as 0 or false, and is left out of the response. So:

• a CVSS score that nobody assessed no longer reads as 0, and an exploit flag that nobody set no longer reads as false;
• a merge update that does not mention a number or flag leaves the stored value unchanged, instead of overwriting it with 0 or false.

Each property in OpenMetadataProperty has a new defaultValue: the value to assume when the property is not set. It is also in the defaultValue additional property of the property's valid metadata value in the Core Content Pack, so callers in other languages can find it.

Every product defined for the Open Metadata Digital Product Catalog now publishes data. The new data sets are:

• Organization Structure - a new family with the Business Capability List, the Department List (each team in an organization's structure, the team it reports to and the organization at the top) and Business Capability Departments, which links each business capability to the departments that support it.
• Survey Reports - the survey report list, annotations, requests for action and their targets, the relational, file, folder, resource and profile measurements, and the missing file classifiers.
• IT Operations Observability - Software Servers, with the location of each server.
• Governance Zone Observability - Zone Membership Totals and Zone Membership Profile. These products have moved into this family from Security Observability.
• Secrets Observability - Open Metadata Secrets Stores and Open Metadata Secrets Collections, with the profile of the user accounts in each collection.
• User Observability - Open Metadata User Identities, with each identity's actor profile, security groups and roles and secrets collections, and Open Metadata User Accounts, the accounts that the OMAG Server Platform Cataloguer records for each secrets collection. No secret is ever published: none of these products reads a secrets store.

The products in the Security Observability family, and its Secrets, User, Governance Zone and Service Observability families, describe how the ecosystem is secured. They are members of the security governance zone, so only people and services with access to that zone can find them, or tell that they exist. The product's asset, subscription processes, notification types and data specification are anchored to the product, so they take its zones through their Anchors classification and are equally hidden. Subscribers are granted a new license, Highly Sensitive - Security Team Only, which forbids sharing the data, or combining it with data that others can see, outside the security team. See Products restricted to the security team.

An ongoing update subscription type - one whose notifications are triggered by changes to its monitored resources - is now registered with the subscription manager (the Baudot Subscription Manager by default) only once it has at least one monitored resource. Previously, an ongoing update subscription type with no monitored resources was treated by Baudot as a periodic one, so its subscribers were notified every time its minimumNotificationInterval passed. This happened to the product families of the Open Metadata Digital Product Catalog whose members have no product asset, such as the observability families: on a quickstart server they accounted for about 70% of the notifications Baudot sent.

• The Product Manager API creates the subscription type and its subscription process as before, so consumers can see the offering, but leaves it out of the subscription manager's catalog targets until a monitored resource is supplied. Setting up the subscription type again with no monitored resources removes an earlier registration, and logs OPEN-METADATA-0021.
• A product family's ongoing update subscription types monitor the product assets of its members. Jacquard now passes each member's asset to the family's subscription types through the Product Manager client as the member is catalogued, and the family is registered with Baudot then. A product added to a family is picked up the next time Jacquard starts.
• One-time and periodic subscription types are registered as before.

The Apache Kafka open metadata topic connector sends and receives events much faster, and can no longer bring down the platform when Kafka falls behind.

• Sending. Events were sent one at a time, waiting for Kafka to acknowledge each before sending the next, which limited a topic to a few hundred events a second. A busy platform writing its audit log to a topic produced events faster than that, and the unsent events piled up in memory until the platform ran out. Events are now sent asynchronously, in compressed batches. In a benchmark the connector sent 210,000 events a second, against 344 before; on a quickstart platform under load, its audit log topic kept up with no backlog.
• Bounded memory. The events waiting to be sent are held in a buffer of at most 100,000 events per topic, set by the new egeria.max.buffered.events producer property. If Kafka is unavailable, or cannot keep up, for long enough to fill it, further events are discarded rather than held: the platform keeps running and the loss is reported by OCF-KAFKA-TOPIC-CONNECTOR-0020. Events that Kafka itself gives up on are reported by OCF-KAFKA-TOPIC-CONNECTOR-0021. The Kafka client now keeps unsent events for up to 10 minutes rather than 2 (delivery.timeout.ms), so a short outage loses nothing: in a test where the broker hung for 150 seconds under load, the platform stayed responsive and every request's events reached the topic.
• Receiving. When more than 100 received events were waiting to be processed, the consumer paused for a full second before polling Kafka again, which limited every topic listener to about 450 events a second. It now pauses for 10 milliseconds. Handing the received events to the listener thread also took time proportional to the square of the number waiting. Under load, a listener's lag stayed below 600 events, where it had grown to 29,000.

The activity messages that the Pepys Activity Observer counts are now written once for each external request, and name the anchor element that was requested. Previously, retrieving one element could write four records:

• METADATA-OBSERVABILITY-0010 from the view server that received the request, and again from the metadata access server that the view server called on its behalf;
• METADATA-OBSERVABILITY-0002 for the element itself when its type was an observed anchor type;
• METADATA-OBSERVABILITY-0003 for the element's anchor when it was anchored to another element.

Now a call from one server to another on the same platform, made on behalf of a request the platform is already processing, writes no 0010 of its own; this is recognised from the request identifier that is passed between the servers. An element anchored to another observed element is reported only against its anchor (0003), and each anchor element is reported at most once for each request. So retrieving an element writes one 0010 and one 0002 or 0003, and the counts that Pepys keeps of how often each element is requested reflect what users actually asked for.

Calls between servers on different platforms cannot be recognised as part of the same request, so each still writes its own 0010. Searches are unchanged: each element returned writes 0008, and each anchor of a returned element writes 0009 once per search.

The Mendel Automated Duplicate Manager now also removes exact duplicates: the identical copies that a process makes when it adds the same thing more than once, typically a connector that links or attaches everything again each time it catalogues a document. Unlike peer duplicates there is nothing for a steward to decide, so on each refresh Mendel keeps the oldest copy and deletes the rest, using the server's default delete method so that they can be restored.

• Duplicate relationships are relationships of a type listed in the new duplicateRelationshipTypes configuration property that connect the same two elements and have the same properties and effectivity dates.
• Duplicate attached elements are elements attached to the same element through a relationship type listed in duplicateAttachmentTypes, that are of the same type, have the same properties and are anchored to that element. An element that is shared, rather than anchored to the element, is never removed.

Every property must match, so elements with a unique name, such as a qualifiedName, never count as duplicates. The Core Content Pack sets the relationship types to AgreementItem and ExternalReferenceLink, and the attachment types to SearchKeywordLink. Set reportDuplicatesOnly to true to have the duplicates reported (MENDEL-DUPLICATE-MANAGER-0019) rather than removed (MENDEL-DUPLICATE-MANAGER-0020).

Engine actions, governance action process instances, to dos and the other kinds of action stay in the repository after they finish, so the action history grew without bound. The new Pasteur Action History Cleanup, a nanny integration connector in the Pasteur integration group of the Core Content Pack, runs once a day and deletes the finished actions that are old enough, along with the elements anchored to them. It uses the LOOK_FOR_LINEAGE delete method: an action with lineage relationships is archived, so lineage queries can still find it, and the others are soft-deleted - so an action removed in error can be restored. To use a different delete method, such as PURGE, set Pasteur's new deleteMethod configuration property.

• daysToCleanUp maps the name of Action or one of its subtypes to the minimum number of days after completion before an action of that type is deleted. A subtype with no entry of its own uses the entry for its nearest supertype, and a type with no entry is never deleted. The Core Content Pack sets EngineAction to 7, GovernanceActionProcessInstance to 14 and ToDo (which includes DataSharingRequest) to 90 days. Meetings, notifications - and so notes, journal and blog entries - and reviews are kept.
• completionStatuses lists the activity statuses that mean an action has finished: by default COMPLETED, INVALID and FAILED.

The completion time is the action's completionTime, or the time it was last updated if that is not set. The engine actions of a governance action process stay as long as their process instance, and are deleted with it. Each refresh reports the number of actions deleted of each type (PASTEUR-ACTION-HISTORY-CLEANUP-0003).

Every request that the generic handlers make to a repository connector for a list of entities or relationships is now split into batches of at most 500, which are read one after another and put back together in order. A page of up to 500 is still a single request. The repository connectors are unchanged, and all of them benefit.

Previously a request for a large page, or for every match, went to the repository as a single query. The PostgreSQL repository builds such a query as one condition per element, so loading an element with tens of thousands of relationships produced SQL statements with over a hundred thousand terms, which took minutes to plan and could exhaust the database server's memory.

The Open Lineage Cataloguer no longer rewrites the RunMetrics classification of a process, and the DataScope classification of each dataset it reads or writes, for every run event it receives. Like the Simon Contribution Recorder, it now queues the events for each element as they arrive, and on each refresh (and when it stops) it reads each element once, applies every queued event, and writes the classification once. The counts and timings are the same as before, but a busy job no longer creates a new version of these classifications for every event. Each write is reported by OPEN-LINEAGE-INTEGRATION-CONNECTOR-0026; if a write fails, the element's events are kept for the next refresh.

• The digital-products governance zone in the shipped user directories (egeria-user-directory and coco-user-directory) had its security lists in the wrong place, so they were ignored and the zone was public. It is now readable by all users and maintained by digital product managers and integration connectors.
• The description of the server access controls in the shipped user directories was written as controlDescription, which is not recognised, so the descriptions were lost when the directory was loaded.
• The egeria-user-directory referred to a dataManagementProcess security role that it did not define.
• A cascaded delete of IT infrastructure, such as a host, deleted the infrastructure a second time instead of the elements deployed on it, and recursed until the server ran out of stack.
• A delete without cascade of a database schema that still had tables was not refused. The anchored members were removed before the check was made, so the schema was found to be empty and deleted.
• The PostgreSQL repository failed with a NullPointerException when a search that included deleted relationships found a relationship whose end had also been deleted.
• Every purge was reported in the audit log as OMAG-REPOSITORY-HANDLER-0001, which says the repository does not support soft-delete. The message is now only issued when that is the reason for the purge.
• cancelEngineAction overwrote the status of an engine action that had already finished, so a completed or failed engine action was recorded as CANCELLED and the record of what it did was lost. An engine action whose status is COMPLETED, INVALID, IGNORED, FAILED, CANCELLED or ABANDONED is now left unchanged.
• The RetentionProperties bean was registered for JSON under the name RetentionProperties.class, so a Retention classification could not be sent in a request.
• The namespacePath of a SchemaAttribute was neither stored nor returned by the Open Metadata Framework.
• The PostgreSQL repository connector failed to start with OMRS-METADATA-COLLECTION-500-006 if no metadata collection id was set in the server's configuration. It now uses the id recorded in its control table, or creates one for a new repository.
• A PostgreSQL repository search with a condition on the properties of a classification also matched the properties of the element's other classifications, so it could return elements that did not match.
• A search that found several members of the same anchor checked the caller's access to the anchor only once. If the first member was readable through zones of its own, a later member with no zones of its own, which is governed by its anchor's zones, was returned without being checked, even when those zones hid it from the caller. Each member is now checked.
• A boolean configuration property of a connector set to false as a JSON boolean, rather than as the string "false", was read as true.
• The Open Metadata Data Types List product published the list of types rather than the data types, and the Open Metadata Properties List published the attributes for each type rather than the properties.
• The columns of the Open Metadata Digital Product Catalog's data sets that name a related element - the profile of a user identity, the secrets store of a secrets collection, the survey report of an annotation and the subject of an annotation - held the unique identifier (or name) of the row's own element instead.
• The Harvest Surveys connector wrote the survey report's start date instead of its start time, swapped the action target and annotation subject of a request for action, and wrote a file's deployed implementation type and last modification time into the wrong columns.
• The secrets files monitor stored a secrets collection's refreshTimeInterval in its additional properties instead of in the property itself.
• A notification type whose plannedStartDate was still in the future notified its subscribers anyway. The audit message OPEN-METADATA-0018 said it was being skipped, but it was not. It is now skipped until its planned start date has passed.
• Bitol documents that the Bitol Document Publisher generated from open metadata were passed to the Data Product Cataloguer and Data Contract Cataloguer in the same integration group, and catalogued again. Each round trip added another SearchKeyword for every tag and linked the agreement items and external references again, so the keywords doubled on every pass: one product gathered more than 27,000 copies of the same keyword. Generated documents are no longer passed to the cataloguers, which also no longer add a keyword, agreement item or external reference that is already attached, and the generator writes each tag once. The duplicates already created are removed by Mendel - see Mendel removes exact duplicate relationships and attached elements above.
• The Open Metadata Framework passed a page size of 0 to the repository unchanged, where it means every match, rather than replacing it with the server's maximum page size. Retrieving an element with a relationshipsPageSize of 0 therefore read all of its relationships, however many there were.
• Mendel could validate a duplicate link between two elements with different qualified names, which only a steward should do. It decided on the status and elements carried in the event, but the event's elements are read in the combined view, where two known duplicates appear as the same element, and its status could be out of date. Mendel now reads the duplicate link again, with its own elements, before deciding.