Nothing should learn about the hospital by polling its database

Two incidents, months apart, at two different hospitals, with the same root cause.

In the first, the finance team’s month-end report ran against the live billing tables on the morning of the first, held locks long enough that the OPD billing counter froze for four minutes, and a queue of patients watched a receptionist apologise. In the second, a compliance officer asked who had opened a particular patient’s HIV result in the previous quarter. The answer came from application logs, two days later, and was incomplete.

Both are symptoms of one database serving three masters: transactions, audit and reporting. A hospital platform has to answer three different questions. Did this happen. Who saw it. How are we doing. Each question has a different consumer, a different latency, and a different tolerance for being wrong. Asking one database to answer all three means it answers each of them badly.

This post is about how HealthPlex separates them.

Commit the fact and publish the fact, in one transaction

Every meaningful state change in a HealthPlex service is a fact: a patient was registered, an encounter closed, an order resulted, a note signed, a claim’s status changed, a consent granted, a record viewed. The service that owns the change commits it and, in the same database transaction, writes an event to an outbox table. A relay publishes the outbox to Apache Kafka. If the publish fails, the row stays and is retried. No fact is lost, and no fact is published that was not committed.

Topics are named for the business fact, not the table: patient.registered, encounter.closed, order.resulted, note.signed, claim.status.changed, phi.accessed, consent.granted. Events are facts about the past, never commands to another service. Their contracts live in a schema registry and are versioned; a consumer that cannot read a new version keeps reading the old one until it is updated. The tenant is part of every key, so isolation survives into the stream.

Retention on the backbone is seven to thirty days depending on the topic, and it is replayable. That last property is the one that makes the rest safe. A consumer with a bug can be fixed and rewound to the offset where the bug started.

Three consumers sit on the stream. None of them touch the transactional database.

The audit ledger answers “who saw this”

The first consumer appends every event, and every phi.accessed event in particular, to an append-only ledger. Each entry carries the hash of the entry before it. Once a day, the hash of the last entry is signed and written to an anchor outside the platform, in an account the application cannot write to.

That structure gives the ledger a property an ordinary audit table does not have: it cannot be quietly edited. Change one field in one historical entry and its hash no longer matches. Every entry after it inherits the break. Verification starts from the external anchor, not from the database being audited, so the platform cannot rewrite its own past.

We run the tamper drill in front of auditors. Edit an entry directly in the store. Run verification from the day’s anchor. Watch the entry fail and everything after it fail with it. Then restore from the anchored copy and watch the chain heal. Five minutes, and the auditor understands the difference between a log and evidence.

The compliance officer’s question about the HIV result is now answered from the ledger in under two seconds, with the actor, the facility, the purpose of use that was declared, and the time. Retention is set per jurisdiction: what HIPAA requires for documentation and what the DPDP Act requires for erasure are different, and the ledger knows which tenant it is holding.

The operational loops answer “did anything happen”

The second consumer is the operational intelligence layer described in an earlier post. An abnormal screening result was filed and nothing has been scheduled seventy-two hours later. A behavioral health client’s authorization has fewer units left than a week of visits. A referral requested a document that was never attached.

None of these are detected by polling a table on a schedule. They are detected because the loop consumed order.resulted or auth.units.drawn and started a clock. When the clock runs out without a closing event, the loop acts inside the systems the hospital already runs, writes the result back, and refuses with a logged reason when it cannot.

Two engineering rules keep this honest. Consumers are idempotent, because Kafka delivers at least once and a duplicate order.resulted must not create a second follow-up. And refusals are events too; a loop that could not act says so on the stream, where the ledger and the analytics plane can see it.

The analytics plane answers “how are we doing”

The third consumer ingests the stream into a lakehouse. Raw events land in bronze. Conformed, deduplicated, tenant-tagged records land in silver. Curated marts for finance, operations, quality and regulatory reporting land in gold. Dashboards, Power BI and the extracts a state program or a payer requires all read from gold.

The transactional database never serves a report. The month-end query that froze the billing counter now runs against a mart that is a few minutes behind the live system, which is the correct trade. A board deck does not need to know about a bill posted ninety seconds ago. A receptionist needs the billing counter to work.

Tenant isolation is preserved through every layer. The lakehouse is multi-tenant in the same way the services are, and a hospital’s marts are its own.

What we gave up

Event-driven systems are eventually consistent. A clinician who signs a note and immediately opens the analytics dashboard will not see it for a minute or two. That is acceptable for reporting and we say so plainly. It is not acceptable for the clinical record, which is why the clinical record never depends on the stream; the service that owns a fact serves it directly.

Consumers must be written to tolerate duplicates and reordering within a partition key. That is a real cost in developer discipline, and we enforce it in review. Every new consumer ships with a replay test.

And the platform’s internal service-to-service messaging is a separate concern from the backbone. Request-scoped work between services uses synchronous calls and a lightweight broker where a queue is warranted. Kafka is the public record of facts, not the plumbing for every call.

The question to ask

If you are evaluating a hospital platform, ask one question: when a nurse opens a chart, what else happens?

If the answer is “a row is written to an audit table in the same database”, ask who can edit that table and how you would know. If the answer is “the reporting job picks it up overnight”, ask what the reporting job does to the database at 6am on the first of the month. If the answer involves a stream, a hash-chained ledger and a lakehouse, ask to see the tamper drill.