Monitoring and Tuning Guide¶

This guide is for the people who keep an Egeria deployment healthy over time. It covers what to watch, how to read Egeria's audit log, how to tune the infrastructure Egeria depends on - Apache Kafka and PostgreSQL - and the maintenance tasks that every production deployment needs, such as backups and clearing out deleted metadata.

The day-to-day controls - starting servers, refreshing connectors, adding catalog targets - are in the Operations Guide. When monitoring shows that something has gone wrong, the Diagnostic Guide helps you find out why.

What to watch¶

flowchart TB
    subgraph egeria ["Egeria"]
    P["`**OMAG Server Platform**
    JVM, threads, REST calls`"] --> S["`**OMAG Servers**
    status, audit log`"]
    S --> C["`**Connectors**
    status, refresh time,
    activity reports`"]
    S --> E["`**Engine actions**
    waiting, in progress, failed`"]
    end
    S --> K["`**Apache Kafka**
    consumer lag, retention, disk`"]
    S --> D["`**PostgreSQL**
    connections, slow queries,
    table growth, backups`"]

The hey_egeria_ops monitors redraw themselves every few seconds, so a terminal running show platforms status and show integrations status side by side makes a simple control-room display. For a deployment that is watched by a central operations team, feed the audit log and the infrastructure metrics into the tools that team already uses.

The audit log¶

Every OMAG Server writes an audit log: a structured record of what it is doing, written by each subsystem and connector. Each record carries a message identifier (such as OMRS-AUDIT-0064), a severity, a description, and - for problems - the likely cause and the action to take. Because every message is documented in the code with its system and user actions, the audit log is the first place to look when something changes.

Where audit log records go¶

Records are passed to one or more audit log destination connectors, each configured with the severities it should receive. This is set in each server's configuration document.

Choosing severities¶

Reading the audit log through the API¶

A running server can be asked for the definitions of its severities and for a report on its audit log - which components are logging and the counts of each message:

###
# @name auditLogReport
GET {{baseURL}}/servers/{{serverName}}/open-metadata/repository-services/audit-log/report
Authorization: Bearer {{token}}

###
# @name auditLogSeverities
GET {{baseURL}}/servers/{{serverName}}/open-metadata/repository-services/audit-log/severity-definitions
Authorization: Bearer {{token}}

Integration connectors and governance engines¶

Connectors and engines do Egeria's work in the background, so they need watching even when no-one is using Egeria directly.

• Status - a connector in FAILED or CONFIG_FAILED stays that way until it is restarted. Check for them daily (the portal and show integrations status both show status), or alert on the audit log messages the integration daemon writes when a connector fails.
• Last refresh time - a connector stuck in REFRESHING for much longer than usual may be waiting on an unresponsive third party technology.
• Connector activity reports - set generateConnectorActivityReports on a connector's registration and it records what each refresh did (elements created, updated and deleted) as open metadata. These make it easy to spot a connector that has silently stopped finding anything.
• Engine actions - a growing number of engine actions waiting to start usually means the engine host that runs them is not active or is overloaded.

The main tuning controls are each connector's refresh interval and its catalog targets. These are described in Running and tuning integration connectors.

Apache Kafka¶

Egeria uses Kafka as its event bus: servers in a cohort exchange metadata over the cohort topics, the metadata access stores publish changes on their out topics (which is how integration daemons and engine hosts hear about configuration changes promptly), and the audit log can be published on a topic. Kafka is optional - a single metadata access store with no cohort can run without it, and integration daemons fall back to re-reading their configuration about every half hour - but most deployments use it.

What to watch¶

• Consumer lag - how far each consumer group is behind the end of its topic. A steadily growing lag means a server is not keeping up, or has stopped reading. Check it with kafka-consumer-groups.sh --bootstrap-server <broker> --describe --all-groups, or your Kafka monitoring.
• Retention - a server that is down for longer than a topic's retention period misses events when it comes back. Set retention on the cohort and out topics comfortably longer than the longest outage you plan for.
• Disk on the brokers, especially with the audit log on a topic, which can be high volume.

Settings that matter¶

Egeria passes producer and consumer properties to Kafka through the event bus configuration of each server. The defaults suit a development system; these are the ones most often changed:

Details are in the Kafka Open Metadata Topic Connector documentation.

PostgreSQL¶

The PostgreSQL repository connector stores each metadata access store's repository in its own database schema (for example repository_qs_metadata_store), in a small set of tables: entity, relationship and classification, each with a matching table of property values. The repository is bi-temporal: every update adds a new version rather than overwriting the old one, so the history of each element can be queried with an asOfTime. This makes the database grow steadily, even when the amount of current metadata is stable.

Repository connector settings¶

These are configuration properties on the repository connector's connection in the server's configuration document:

What to watch¶

• Slow and frequent queries - enable pg_stat_statements (the egeria-workspaces PostgreSQL does this, with track_activity_query_size=16384 so Egeria's long generated SQL is shown in full):

  CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
  -- busiest statements by total time
  SELECT calls, round(total_exec_time) AS ms, rows, left(query, 120)
  FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 20;
  

• Active connections - SELECT usename, state, count(*) FROM pg_stat_activity GROUP BY 1, 2; shows how close each server's pool is to its limit.

• Table growth - track the size of each repository schema over time:

  SELECT schemaname, relname, pg_size_pretty(pg_total_relation_size(relid)) AS size, n_live_tup, n_dead_tup
  FROM pg_stat_user_tables WHERE schemaname LIKE 'repository_%'
  ORDER BY pg_total_relation_size(relid) DESC;
  

• Vacuum - autovacuum keeps the tables in shape under normal use. After a large purge (see Managing deleted metadata) run VACUUM (ANALYZE) on the repository tables so the space is reused and the query planner has current statistics.

To see the SQL the repository connector issues, set logging.level.org.odpi.openmetadata.adapters.connectors.resource.jdbc=DEBUG in the platform's application.properties. This is very verbose: use it for an investigation, not in normal running.

Maintenance tasks¶

Backups¶

What to back up:

• The PostgreSQL repositories - this is the metadata itself. pg_dump produces a consistent copy while Egeria is running, so a nightly dump of each repository schema (or the whole database) is a sound baseline:

  pg_dump --host <db-host> --port 5432 --username <admin-user> \
          --format custom --schema 'repository_*' \
          --file egeria-$(date +%Y%m%d).dump egeria
  

  For larger deployments, or a tighter recovery point, use PostgreSQL's continuous archiving (base backups plus write-ahead log archiving) instead. * The servers' configuration documents - by default in the platform's data/servers/<serverName>/config directories, or wherever the platform's configuration store keeps them. * The platform's application.properties, keystores, trust stores and secrets stores. Treat these backups with the same care as the originals. * Kafka does not normally need backing up: the topics carry events in transit, and the repositories hold the result.

Test restores regularly. A restore of a repository must be matched with the configuration documents that refer to it (the repository's metadata collection identifier is in the configuration document), and other cohort members will re-send any changes they made while the restored server was out of date.

Managing deleted metadata¶

When metadata is deleted, Egeria chooses between four outcomes, set by the deleteMethod on the request (or on a catalog target for metadata maintained by a connector):

Soft-deleted elements stay in the repository, with their history, until they are purged. On a busy system - particularly one where connectors catalog changing technology, recreating and deleting elements as it changes - they can come to outnumber the current metadata. Egeria does not purge them by itself, because how long to keep them is a policy decision for the deploying organization: deleted glossary terms might be kept for a year in case they need restoring, while the search keywords a connector regenerates on every refresh might be kept for a week.

The practical approach is a scheduled housekeeping script that applies your retention policy. Egeria provides an example to start from:

purge-deleted-elements.py

The script:

1. Asks the metadata access store's repository services for its soft-deleted entities and relationships (using limitResultsByStatus: ["DELETED"]).
2. Skips anything whose home is another cohort member - only the home repository can purge an element.
3. Compares the time each was deleted with the retention period for its type, from a policy you supply.
4. Purges those that are past it - relationships first, then entities - or, by default, just lists what it would purge.

The retention policy maps type names to the number of days to keep a deleted element. A type includes its subtypes, an element's own type wins over a supertype, and "*" covers everything else (leave it out to purge only the types you name):

{
  "SearchKeyword": 7,
  "SearchKeywordLink": 7,
  "GlossaryTerm": 365,
  "Referenceable": 90
}

Run it by hand first, as a dry run, to see what your policy would remove:

export EGERIA_USER=garygeeke EGERIA_USER_PASSWORD=secret
python3 purge-deleted-elements.py --platform-url https://localhost:9443 \
        --server qs-metadata-store --policy retention.json --insecure

Then schedule it with --purge using cron, a Kubernetes CronJob, or your organization's job scheduler:

30 2 * * 0  /usr/bin/python3 /opt/egeria/purge-deleted-elements.py --server qs-metadata-store --policy /opt/egeria/retention.json --purge >> /var/log/egeria/purge.log 2>&1

The script exits with a non-zero code if any purge fails, so the scheduler can raise an alert. --max-purges limits the work done in one run, which spreads a large first clean-up over several nights.

The script calls the repository services REST API directly, using only the Python standard library, so it runs anywhere Python 3.9 or later is installed. The two calls it relies on are:

###
# @name findDeletedEntities
POST {{baseURL}}/servers/{{metadataStore}}/open-metadata/repository-services/users/{{userId}}/instances/entities
Authorization: Bearer {{token}}
Content-Type: application/json

{
  "class" : "EntityFindRequest",
  "typeGUID" : "{{typeGUID}}",
  "limitResultsByStatus" : ["DELETED"],
  "offset" : 0,
  "pageSize" : 500
}

###
# @name purgeEntity
POST {{baseURL}}/servers/{{metadataStore}}/open-metadata/repository-services/users/{{userId}}/instances/entity/{{deletedEntityGUID}}/purge
Authorization: Bearer {{token}}
Content-Type: application/json

{
  "class" : "TypeDefValidationForRequest",
  "typeDefGUID" : "{{typeGUID}}",
  "typeDefName" : "{{typeName}}"
}

Relationships use .../instances/relationships with a RelationshipFindRequest, and .../instances/relationship/{{deletedRelationshipGUID}}/purge.

Cleaning up audit log files¶

The file audit log destination writes a separate file for every record. On a busy server that is many thousands of files a day, so delete old ones on a schedule, after copying them to longer-term storage if your policies require it:

find data/servers/*/logs -name 'log-record-*.omalrecord' -mtime +14 -delete

Adjust the path to the directory set in the destination's endpoint. If you do not use the files, remove the file destination from the server's configuration and rely on the console or event topic destinations instead.

Further information¶

• Operations Guide - controlling servers and connectors.
• Diagnostic Guide - investigating problems.
• Security Guide - certificates, credentials, secrets stores and security audit records.
• Metadata observability - turning audit log activity into usage statistics.

Raise an issue or comment below