Secure Health Data Sharing Without Lock-In

Author: Reece Robinson, CTO of Orchestral

Healthcare organisations have spent years trying to make data more usable. We have invested in interoperability standards, analytics platforms, data warehouses, reporting systems, population health tools, and increasingly, data lakes. Yet many of the hardest problems remain organisational rather than purely technical: how do we share data across agencies, partners, and platforms without losing governance, privacy, accountability, or trust?

For CTOs and architects, this is becoming a central design challenge. It is no longer enough to build a capable internal data platform. Modern health data platforms need to participate in broader ecosystems. They need to exchange data with other data lakes. They need to support analytics across organisational boundaries. They need to allow approved users to discover and query curated datasets. And they need to do all of this without creating uncontrolled copies of sensitive health data.

The architectural question is shifting from “Can we store and analyse the data?” to “Can we share and govern the data safely across institutional boundaries?”

Secure health data sharing diagram Figure 1: A governed data sharing layer allows health organisations, data lakes, warehouses, and analytics platforms to interoperate without forcing everyone onto the same proprietary platform.

Many data sharing arrangements start with a pragmatic extract. A team prepares a file, places it in a secure location, and another team consumes it. This can be entirely appropriate. For well-defined reporting feeds, operational extracts, or curated datasets, a governed pipeline remains one of the simplest and most reliable models.

The problem is that point-to-point exchange does not scale well as the number of datasets, agencies, use cases, and analytics platforms grows. Every new destination can create another copy. Every copy needs lifecycle management, access control, auditability, and policy alignment. Data quality issues are harder to trace. Retention and revocation become more complicated. Over time, the architecture can drift from governed sharing into a loose network of duplicated datasets.

In healthcare, that risk matters. Data sharing must respect privacy, consent, purpose-of-use, security obligations, and public trust. The platform architecture needs to make good governance the default, not an afterthought added downstream.

A Better Pattern: Governed Data Products and Controlled Access

A more sustainable approach is to treat shared datasets as governed data products. That means data is curated, classified, documented, access-controlled, and published for specific uses. The receiving party should not need to understand every source system detail. They should receive access to data that has been prepared for a defined purpose and wrapped in the right controls.

This does not imply a single technical mechanism. In practice, mature data platforms need several sharing patterns.

Sometimes the right answer is a governed pipeline that publishes a curated dataset into another agency’s environment. Sometimes it is controlled SQL access, where approved users can query a dataset without receiving the underlying storage directly. Sometimes it is an open sharing protocol that allows external analytics tools to consume approved data without bespoke integration work. Sometimes it is federation, where query access spans multiple data sources under a common policy model.

The important architectural principle is that storage, processing, policy enforcement, and access should be separable concerns. When these concerns are too tightly coupled to one vendor or one platform, cross-agency sharing becomes harder. When they are separated cleanly, the platform has more room to support different agencies, clouds, and analytics tools.

Open Lakehouse Architectures Help, But Governance Is the Differentiator

Open lakehouse patterns are useful because they reduce dependency on a single proprietary warehouse model. They allow analytical data to be managed in formats and structures that can be processed by multiple engines and integrated with modern cloud data environments. For architects, this matters because healthcare data ecosystems are rarely homogeneous. One organisation may be using a cloud data lake. Another may be using a warehouse. Another may have a specialist analytics platform. Another may need a controlled extract for downstream operational use.

Open storage and processing patterns make this easier, but they are not sufficient by themselves. Interoperability without governance simply makes it easier to move sensitive data around. The real requirement is governed interoperability.

That means data access decisions need to be centralised, auditable, and enforceable. Users and systems should only see the datasets, rows, and fields they are permitted to use. Sensitive attributes may need to be masked. Certain records may need to be filtered. Access should be traceable. Policies should align with data sharing agreements and organisational roles. Ideally, the same governance posture should apply whether the access path is a scheduled pipeline, an interactive query, or an external sharing mechanism.

This is where the architecture must move beyond “data lake connectivity” and toward “secure data sharing services.”

The Role of a Governed Query Layer

One useful capability is a governed query layer over the data lake. A distributed SQL engine can provide a familiar access model for analysts, applications, and partner systems while keeping the underlying data lake protected behind policy enforcement.

This pattern is powerful because it avoids forcing every consumer into the same compute environment. Users can query approved datasets through a controlled interface, while the platform retains the ability to enforce access policies, apply masking or filtering, and audit activity. It also creates a useful abstraction between the consuming organisation and the internal data layout of the platform.

For cross-agency collaboration, this matters. Agencies often want to explore and analyse data without necessarily receiving bulk copies of everything. A governed SQL access model can support that need while preserving stronger control than a raw storage-level integration.

Another important design consideration is how external parties access shared datasets. Traditional approaches often assume that all participants will standardise on the same analytics stack. In reality, that is rarely practical across government, healthcare, and partner ecosystems. Different agencies have different cloud strategies, procurement histories, security standards, and analytics tooling.

Open sharing protocols offer a more flexible model. They allow a provider to share approved datasets with external consumers without requiring every participant to adopt the same compute platform. This is particularly attractive for inter-agency analytics, where the objective is to make approved data usable while minimizing unnecessary duplication and bespoke integration.

The goal is not to eliminate pipelines, warehouses, or data lakes. Those will all continue to exist. The goal is to provide a governed sharing layer that allows each pattern to be used for the right purpose.

What This Means for Health Data Platforms

For a health data platform to support modern data sharing, it needs to provide a few core capabilities.

First, it needs a robust data lake foundation for curated analytical datasets. Second, it needs workflow orchestration so that data products can be prepared, refreshed, validated, and published reliably. Third, it needs a governed query access layer so approved users can search and analyse data safely. Fourth, it needs centralised policy enforcement and auditability. Finally, it needs a path to external sharing that does not require all parties to standardise on one vendor or one compute engine.

This combination allows a platform to support multiple sharing models: controlled extracts, recurring data products, governed query access, federated analytics, and external dataset sharing. More importantly, it allows those models to operate under a consistent governance posture.

That consistency is what turns a data lake from a storage architecture into a trusted data sharing platform.

How Orchestral HIP Approaches This

Orchestral’s Health Intelligence Platform, or HIP, has been designed around this model. HIP uses an open lakehouse architecture for analytical data management, large-scale processing for data preparation, and workflow orchestration for governed pipelines and curated data products.

For governed access, HIP includes a distributed SQL query layer through Trino, with centralised policy management and enforcement through Apache Ranger. This allows HIP to provide controlled SQL access to curated datasets while supporting fine-grained access controls, masking or filtering patterns, and auditability.

HIP also supports external connectivity and data sharing services for integration with other data lake and analytics environments. Where a partner or agency needs a curated copy of data, HIP can support governed publication through managed pipelines. Where approved users need analytical access, HIP can provide controlled query access. Where cross-organization analytics access is required, HIP supports open sharing patterns, including Delta Sharing for external dataset access.

The point is not that every data sharing use case should use the same mechanism. It should not. The point is that the platform should support several well-governed mechanisms and help architects choose the right one for the use case.

The Architectural Principle

Healthcare data sharing will continue to become more distributed. Agencies, providers, payers, researchers, and public health organisations will need to collaborate across boundaries. Data will live in multiple environments. Analytics will happen in multiple platforms. Governance expectations will only increase.

The winning architecture is not a single centralised warehouse that everyone must use. Nor is it an uncontrolled mesh of duplicated extracts. The better model is an open, governed data lake platform that can publish, query, and share data safely across organisational and technical boundaries.

For CTOs and architects, that means designing for interoperability and governance together. Open formats help. Distributed query helps. Policy enforcement helps. Sharing protocols help. But the real value comes from combining these capabilities into a coherent platform posture.

Secure data lake interoperability is not just a technical feature. In healthcare, it is becoming a prerequisite for trusted collaboration.

Secure Health Data Sharing: It’s not a Proprietary Thing

Why modern health data platforms need open lakehouse architecture, governed query access, centralized policy enforcement, and flexible external sharing models.

The Limits of Point-to-Point Data Sharing