January 2025¶

Welcome to the Egeria community's January 2025 newsletter. Since our last newsletter, the community has developed some exciting new capabilities as well as improving the function, reliability and performance of existing features.

New Native Open Metadata Repository built in PostgreSQL¶

We are excited to announce a new open metadata repository choice for Egeria, hosted on PostgreSQL. The contents of the repository are stored in 7 tables within its own database schema. A single PostgreSQL server/database may host multiple repositories.

The PostgresRepositoryConnector offers high-speed metadata loading, full range of queries, including bi-temporal queries and queries into metadata properties of type array and map. It is possible to turn on debug logging and see every SQL command issued by Egeria. This transparency has helped in debugging issues both in Egeria and calling code. It has also lead to some valuable performance improvements in Egeria's core runtime. These are incorporated in Egeria's 5.2 release.

Operationally, PostgreSQL has the advantage that it is available on many (all?) cloud platforms and most organizations have experience in managing PostgreSQL database (such as high-availability, backup and restore).

Visualization using Mermaid¶

Good visualizations help to improve understanding and validation. Mermaid is an amazing technology that generates graphical visualizations from and extended form of markdown. The graphs can be rendered in Jupyter notebooks, GitHub README pages and in their own live editor. Selective REST APIs from Egeria's view services are returning a mermaid visualization string with the query results. The mermaid string describes a visualization of the query results, helping the caller get a quick understanding of the results they have just received. More details of this capability can be found on the Graph Visualizations page.

Extensions to pyegeria¶

The pyegeria python libraries continues to evolve at a rapid pace offering python interfaces to Egeria's services along with scripting commands and the hey_egeria command level interface.

In addition, there are new documentation pages that walk you through the different commands and associated displays:

• Introduction to Hey_Egeria
• Visual Command Reference
• Command Line Interface

New Governance Action Processes¶

The Technology Content Packs contain the metadata necessary for activating the connectors and services that support particular types of technologies. In the 5.2 release they also include governance action processes that simplify working with the connectors.

• Create and Survey Governance Action Processes accept the placeholders for a technology template (such as a Unity Catalog Server), creates a metadata element to represent an instance of the technology, activates an appropriate survey, and then publishes the report.
• Create and Catalog Governance Action Processes accept the placeholders for a technology template, creates a metadata element to represent an instance of the technology, configures the appropriate integration connector to set up an ongoing metadata synchronization between Egeria and the named technology.
• Delete Governance Action Processes accept the placeholders for a technology template, looks up the corresponding metadata element that represents the requested instance of the technology and deletes it. This is useful if you pass incorrect values to the previous two governance action process, since it cleans up the metadata created with the wrong values allowing you to start again.
• Provision Governance Action Processes accepts the identifier of a metadata element that describes a desired digital resource and provisions the resource in the appropriate third party technology.

As an example, below are mermaid visualizations for some of the new processes for Unity Catalog:

Governance Action Process Lineage¶

In addition to publishing open lineage events when they run, governance action processes also create a GovernanceActionProcessInstance entity linked to the engine actions that were created during the execution of the governance action process. It is then possible to query both the design and the execution of the process via pyegeria and compare then using Mermaid visualizations.

Design of Governance Action Process	Single Execution of this Governance Action Process

Improvements to template support¶

Template placeholders are now resolved in classification and relationship properties - not just entities. These placeholders may also be used for all types of properties - not just strings. This increases the range and complexity of template to use when cataloguing technology in open metadata.

Extensions to the View Services¶

There is a new view service called Metadata Explorer OMVS that offers a complex query interface, enabling you to retrieve any metadata that interests you. This REST APIs return mermaid strings with the requested metadata elements.

Improvements to the Secrets Stores¶

Secret Store Connectors are now able to support the authentication of users calling the Open Metadata View Services (OMVS). This makes it easier to integrate with a third party user directory (LDAP, Keycloak, etc) via an implementation of a Secrets Store Connector.

Roles and contribution records for teams and automation¶

Roles and contribution records used to be restricted to people. However, with the expansion of automated agents in the data management and governance space, it is valuable to be able to show where they are playing in the open metadata landscape. Release 5.2 includes:

• Reorganized open metadata types
• Generalized karma point collection
• IT Profiles added to the Integration Connectors in the Technology Content Packs so that karma points are automatically collected for their activity.

Full release notes for V5.2¶

The full release notes for version 5.2 can be found on Egeria's website.

Future plans¶

Plans for release 5.3 are open, and we are looking for suggestions. This is being track using GitHub projects

Connecting with the project¶

With our new focus on Python and supporting data practitioners directly, the community is looking for new contributors to the project. If these topics interest you, then please contact us and we can help you get started.

Raise an issue or comment below