Patient Data Sharing Hub¶

Robbie Records has decided to take action. He works for Hampton Hospital and is responsible for the centralized Patient Data Records. He receives a lot of requests for patient data, beyond the normal day-to-day uses when treating patients. These additional requests come from the medical staff working at the hospital as part of their research projects, as well as pharmaceutical companies that are working with the hospital on clinical trials.

He currently has to manage these requests by hand, which means he often takes at least a week to fulfill a request, and sometimes he has to refuse them when he is overwhelmed with other work. He wants to automate the process so that he can focus on other tasks and improve the data sharing experience.

At the same time, the hospital has strict policies on patient confidentiality, security and privacy. He is audited regularly and needs to ensure that he is both implementing these policies AND can demonstrate to an auditor that all is working as expected.

How is patient data managed?¶

The patient data that Robbie managed is largely in a number of PostgreSQL databases plus some media files for medical images. There is a portal for the staff to view and update a particular patient's data. When a patient is being treated, new data from the various medical systems and equipment is added to the patient's data. This may arrive as files or as a stream of data that is fed into PostgreSQL. The PostgreSQL server, file system and portal is managed by the hospital's IT department, so Robbie is not responsible for them. As a result, they are backed up, upgraded and secure.

When Robbie is asked to share patient data with a third party, he creates a new database for their project and copies the data from the original database. This copying is needed to be able to demonstrate exactly what data was shared with a particular requestor.

He maintains a set of documents in a directory (folder) on the filesystem - one for each project requesting data. The documents describe the purpose of the project, how the data will be used and which database has the copy of the data that was shared.

To actually share the data, Robbie typically exports the data into CSV files that are then zipped up with a password and delivered to where the team can access it. The password is generated by a secure password generator and is only shared with the team that requested the data.

When Robbie is working with an external third party, such as a pharmaceutical company performing a clinical trial, there are often additional requirements, specified by the data sharing contract between the hospital and the pharmaceutical company. Robbie can perform simple transformations using SQL commands, but needs to get help from the department that generated the data if data values are outside acceptable ranges.

Robbie's vision¶

Robbie would like to create a patient data sharing hub that allows an in-house data requester to manage (create/view/amend) their data sharing request, including specifying the data they require and the purpose they will use it for. They may also need to agree to certain conditions on how the data must be managed once they have their copy.

Robbie also wants to handle data sharing requests from external third parties. In this case, the third party will not have direct access to the patient data sharing hub, but Robbie should be able to manage their request and requirements, as well as keeping track of any data sent to them.

Using Egeria¶

Robbie saw a presentation about the latest version of Egeria at a conference and decided to try it out. Egeria provides a robust platform for managing metadata and governance, which will be crucial for tracking and managing data sharing requests and ensuring compliance with regulations and contracts.

He downloads egeria-workspaces from GitHub and installs it on his local machine. He then starts customizing it for the Patient Data Sharing Hub.

Building a Data Dictionary for the Patient Data Sharing Hub¶

The first step is to create a data hub element in Egeria. He then catalogues the database server and folders where the digital images are located. He links them to the data hub. This process is called metadata discovery.

From this definition, Egeria automatically builds an inventory of the patient data fields he has on offer in the data hub. These are organized into a data dictionary that data requesters can select from.

This skeleton data dictionary is extracted from the schema extraction and profiling performed by Egeris. It will be complete, in terms of the data fields and their types, but lacking in descriptions that explain the values, where they come from and what they mean. Robbie will need to add these descriptions to the data dictionary.

Robbie requests an update FORM from Egeria for all the fields in the data dictionary. This form is a Markdown document with a section for each field. It is a big task, but interesting. Many of the fields, Robbie is familiar with, but he discovered other data he was not aware of. He shares the Markdown document with collegues who help him to fill out the details he is not sure of.

The document itself sparks interest and discussion. A number of people request a copy when it is finished.

Robbie loads the Markdown document into Egeria so that all the descriptions are added to the data dictionary in the open metadata repository. He uses Egeria's portal to check the descriptions were added correctly. He also notices that each field in the data dictionary is linked to the corresponding schema elements representing the tables and columns in the patient database. This linkage could allow an automated pipeline to navigate from the data fields requested by a data requestor to the actual data in the database.

He requests an HTML report of the data dictionary and installs it in Egeria's webserver. It creates a nice website for the data dictionary that is easy to read and navigate. This could be the site used by future data requestors to find out what data is available. However, it is only accessible through his machine at the moment. He is not sure about the security of opening it up to others. He puts the idea to one side for now. Instead, he requests a Markdown document REPORT of the completed data dictionary.

Robbie is pleased with the result and shares it with his colleagues. They are impressed with the level of detail and the quality of the descriptions. They also appreciate the effort that went into creating the document. He also shows them the website and they agree that would be useful too.

Creating a data sharing request form¶

Now Robbie needs to design a form that data requesters fill in to describe the data they want and the purpose for it. This is a markdown document that includes the commands that defines the data sharing agreement in Egeria. It also includes a list of the data fields from the data dictionary that the data requester wants to receive. The form is a Markdown document that is easy to read, review and edit. Once everyone is in agreement, the form is submitted to Egeria for processing.

Creating the solution blueprint¶

Robbie creates a solution blueprint that describes the data sharing hub. This is also a Markdown document that includes the commands that defines the components and how they are wired together in Egeria.

This blueprint is to show stakeholders how the solution works.

Processing a data sharing request from a member of the hospital medical staff¶

Robbie is now ready to start using Egeria to manage his data sharing requests. He works with once of the consultants, Grant Able to create the first data sharing agreement. Robbie uses this to test his data pipelines that read the data sharing agreement and delivers the data.

Processing a data sharing request from a third party¶

Robbie has one more test. Coco Pharmaceuticals is running a clinical trial for Teddy Bear Drop Foot. When Robbie receives the data sharing request for the clinical trial, he creates the data sharing agreement, and with a few minor changes, was able to get the data pipelines to produce the right data for the clinical trial.

The IT Team arrive ...¶

Once the Patient Data Sharing Hub ready to manage data sharing requests from medical staff and external third parties, Robbie invites the hospital's IT team to review it. They visit Robbie to express their concerns at him running his own system. This is a critical moment for Robbie's new system. With the manual system, auditing was always a major headache since he had to reconstruct the specific data sharing process every time he needed to demonstrate compliance with regulations and contracts.

Egeria's portal includes reports that enable Robbie to demonstrate how the system works at a general level and then dive into specific cases. There are details on the user's with access and exactly what they have access to. He can show the data sharing agreements for each request and exactly what data was shared.

The system is containerized to limit the vectors for a security breach. The IT team examines the setup and agrees it looks secure, and he has covered the privacy and security requirements well. However, there are no backups, and they wondered if Robbie was willing to manage all of the patching and upgrading of the system over time. They would like to perform some security scans on the software and perform some penetration tests too.

The IT team offered to take over the maintenance and backup of the system. Robbie would retain admin access to Egeria to allow him to make improvements. The IT team would monitor the hub and ensure that it remained up-to-date and backed up. They also offered to securely host his website containing the solution descriptions and data dictionary.

This seemed a reasonable agreement to Robbie - in fact, just perfect :).

Conclusion¶

With the patient data sharing hub in place, Robbie can now manage his data sharing requests and ensure that they are compliant with regulations and contracts.

His reputation grew in the hospital as a person who understood how to manage data. He was invited to research planning meetings and asked to take ownership of more data sources. Some of the research staff petitioned for more funding for Robbie so he could expand the types of data on offer. This resulted in:

• A digital product catalog for popular data sets.
• A glossary of terms to share understanding of complex concepts and data values.
• Hosting of research project data, removing the need for many of the research teams to manage their own copies of the data.

Raise an issue or comment below