XTDB Native OMRS Repository Connector¶

XTDB is XTDB is a bi-temporal and dynamic relational database.

The XTDB OMRS repository connector enables the use of XTDB (formerly known as "Crux") as a native Egeria repository, including its own pluggable architecture to support a variety of underlying storage back-ends such as S3, RocksDB, Apache Kafka, LMDB, JDBC and more.

XTDB supports temporal graph queries to provide native support for storing historical information and answering temporal queries. The connector is also capable of running as a highly-available service, including self-managing write consistency across a distributed (multi-node) cluster. In addition, currently this is the highest-performance open source persistent repository for Egeria across all operations: read, write, update, search and purge.

How it works¶

The XTDB OMRS Repository Connector is a native repository connector running on a metadata access store.

XTDB itself is started as an embedded process within the connector. It can be configured to use any of the various pluggable persistence layers supported by XTDB itself, and communication between the Java code of the connector and XTDB itself (which is implemented in Clojure) occurs directly via the XTDB Java API (not via REST).

The repository connector (and metadata collection) methods of the repository connector interface simply communicate with XTDB via XTDB's Java API to read and write information to the underlying XTDB node.

XTDB itself handles write transactions and persistence guarantees via its APIs, ensuring that all data is at least recorded into the transaction log and document store prior to any write method returning.

Configuration¶

The following options are used to configure this connector, as part of the configure the local repository step when configuring a metadata server.

Pluggable persistence¶

There are many options for configuring XTDB itself.

A list of overall persistence modules and deeper configuration options for each can be found through XTDB's own documentation .

To enable persistence, there are two options:

• send in the JSON document configuration outlined in XTDB's own documentation directly to the xtdbConfig key of the configurationProperties property of Egeria's connector configuration
• send in a string to the xtdbConfigEDN key of the configurationProperties of Egeria's connector configuration, which gives the EDN form of configuration outlined in XTDB's own documentation

Both approaches are valid and should be equally functional, but occasionally a bug may crop up that makes one or the other more or less feasible for a particular configuration.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

{
  "class": "Connection",
  "connectorType": {
    "class": "ConnectorType",
    "connectorProviderClassName": "org.odpi.egeria.connectors.juxt.xtdb.repositoryconnector.XtdbOMRSRepositoryConnectorProvider"
  },
  "configurationProperties": {
    "xtdbConfig": { // (1)
      "xtdb.lucene/lucene-store": { // (2)
        "db-dir": "data/servers/xtdb/lucene"
      },
      "xtdb/index-store": {
        "kv-store": { // (3)
          "xtdb/module": "xtdb.rocksdb/->kv-store",
          "db-dir": "data/servers/xtdb/rdb-index"
        }
      },
      "xtdb/document-store": {
        "kv-store": {
          "xtdb/module": "xtdb.rocksdb/->kv-store",
          "db-dir": "data/servers/xtdb/rdb-docs"
        }
      },
      "xtdb/tx-log": {
        "kv-store": {
          "xtdb/module": "xtdb.rocksdb/->kv-store",
          "db-dir": "data/servers/xtdb/rdb-tx"
        }
      }
    }
  }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

{
  "class": "Connection",
  "connectorType": {
    "class": "ConnectorType",
    "connectorProviderClassName": "org.odpi.egeria.connectors.juxt.xtdb.repositoryconnector.XtdbOMRSRepositoryConnectorProvider"
  },
  "configurationProperties": {
    "xtdbConfigEDN": "{:xtdb/index-store {:kv-store {:xtdb/module xtdb.rocksdb/->kv-store :db-dir \"data/servers/xtdb/rdb-index\"}} :xtdb/tx-log {:kv-store {:xtdb/module xtdb.rocksdb/->kv-store :db-dir \"data/servers/xtdb/rdb-tx\"}} :xtdb.lucene/lucene-store {:db-dir \"data/servers/xtdb/lucene\" :indexer {:xtdb/module xtdb.lucene.egeria/->egeria-indexer} :analyzer {:xtdb/module xtdb.lucene.egeria/->ci-analyzer}} :xtdb/document-store {:xtdb/module xtdb.jdbc/->document-store :connection-pool {:dialect {:xtdb/module xtdb.jdbc.psql/->dialect} :db-spec {:jdbcUrl \"jdbc:postgresql://myserver.com:5432/mydb?user=myuser&password=mypassword\"}}}}"
  }
}

{:xtdb.lucene/lucene-store {
    :db-dir "data/servers/xtdb/lucene"
    :indexer {:xtdb/module xtdb.lucene.egeria/->egeria-indexer} ;; (1)
    :analyzer {:xtdb/module xtdb.lucene.egeria/->ci-analyzer}} ;; (2)

14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

  :dependencies [[org.clojure/clojure "1.10.3"]
                 [org.clojure/tools.logging "1.1.0"]
                 [com.xtdb/xtdb-core]
                 [pro.juxt.clojars-mirrors.com.github.seancorfield/next.jdbc "1.2.674"]
                 [org.clojure/java.data "1.0.86"]
                 [com.zaxxer/HikariCP "3.4.5"]
                 [pro.juxt.clojars-mirrors.com.taoensso/nippy "3.1.1"]

                 ;; Sample driver dependencies
                 [org.postgresql/postgresql "42.2.18" :scope "provided"] ;; (1)
                 [com.oracle.ojdbc/ojdbc8 "19.3.0.0" :scope "provided"] ;; (2)
                 [com.h2database/h2 "1.4.200" :scope "provided"]
                 [org.xerial/sqlite-jdbc "3.28.0" :scope "provided"]
                 [mysql/mysql-connector-java "8.0.23" :scope "provided"]
                 [com.microsoft.sqlserver/mssql-jdbc "8.2.2.jre8" :scope "provided"]]

Connector options¶

There are currently two configuration options for the connector itself:

{
  "class": "Connection",
  "connectorType": {
    "class": "ConnectorType",
    "connectorProviderClassName": "org.odpi.egeria.connectors.juxt.xtdb.repositoryconnector.XtdbOMRSRepositoryConnectorProvider"
  },
  "configurationProperties": {
    "xtdbConfig": { },
    "luceneRegexes": true,
    "syncIndex": true // (1)
  }
}

High availability¶

A sample Helm chart is provided for configuring the XTDB connector for high availability . This chart starts up a number of different elements and configures them in a specific sequence.

Startup¶

When it is first deployed, the Helm chart starts a number of pods and services: for Egeria (purple), Kafka (red), execution of the Performance Test Suite and a pod used for configuration. (As mentioned above, it assumes a pre-existing JDBC database: a vanilla PostgreSQL cluster (grey) deployed and managed independently by EnterpriseDB's k8s operator.)

Each XTDB pod runs its own separate OMAG Server Platform, in its own JVM, and a script in the init-and-report pod will wait until all three pods' OMAG Server Platforms are running before proceeding to any of the following steps. (The headless service allows each pod to be directly addressed, without load-balancing, to do such a check.)

Configure¶

The next script creates a singular configuration document via the pts pod, and deploys this common configuration to each of the pods (again using the headless service to directly address each one individually): each will have a separate xtdb server configured with the same XTDB connector (same metadata collection id).

When the /instance is called against each pod to start the connector, each will create a local index and instance of the IXtdb interface: all pointing to the same golden stores (in this example, Kafka and EDB) where all persistence for XTDB is handled. All servers will refer to the singular xtdb load-balancing service as their root URL.

Run¶

Now when we start the Performance Test Suite, all traffic to the technology under test is routed via this xtdb load-balancing service: which will round-robin each request it receives to the underlying pods running the XTDB native repository connector.

Kafka has a similar service, which handles load-balancing across its own pods for all write operations.

The underlying JDBC cluster may have a similar load-balancing service again (e.g. if the data store uses sharding), but also may not. In this example the edb-rw service layer is instead an abstraction of the primary data store (edb-1): all writes will go to this primary data store, while the others act as secondary / standby servers to which EnterpriseDB is automatically handling data replication from the primary. If the primary pod fails, EnterpriseDB can re-point the edb-rw service layer to one of these existing secondary stores (which is automatically promoted to primary by EnterpriseDB).

Outages¶

Should there be any outage (in the example above, an Egeria pod, a Kafka pod, and an EnterpriseDB pod all going down) the Kubernetes services will simply stop routing traffic to those pods and the overall service will continue uninterrupted.

Depending on how the underlying services are managed, they may also be able to self-heal:

• Kafka is deployed as a StatefulSet in kubernetes, so if any pod fails kubernetes will automatically attempt to start another in its place to keep the total number of replicas defined by the StatefulSet running at all times.
• EnterpriseDB in our example was deployed through an operator: this operator self-heals any individual pod failure to e.g. start another standby server pointing at the same PersistentVolumeClaim as the failed pod (to pick up the data that was already replicated), switch the primary server to one of the standby servers if the primary server fails, and so on.

Limitations¶

There are a number of limitations to be aware of with the high availability configuration:

Raise an issue or comment below