Multimodal patient records: the underestimated substrate

The data problem

Foundation models are only as good as the record they are given. Most health system records are a mix of structured fields, free text, imaging, and external documents — none of which are easily unified.

The emerging substrate

A new category of infrastructure — the multimodal patient record — is emerging to unify these sources into something usable by modern AI. Whoever wins this layer becomes the platform.

Interoperability tailwinds

FHIR maturity, TEFCA rollout, and the Cures Act information-blocking rules are all pushing in this direction. The regulatory environment is unusually favorable.

What founders should build

Domain-specific record unifiers. Not another data lake — a purpose-built substrate for a specific clinical workflow.

Why text-only records understate the problem

Most clinical AI systems today are trained and evaluated primarily against structured fields and free-text notes, which represent only a fraction of the clinically relevant information generated during a typical encounter. Imaging, waveform data, scanned referral letters, and handwritten annotations are frequently excluded entirely, not because they are unimportant but because they are harder to parse reliably.

This creates systems that perform well on the narrow slice of the record they were built to read while missing information a human clinician would consider obviously relevant, such as a scanned outside report sitting unprocessed in a document repository attached to the chart.

What a true multimodal substrate looks like

The emerging substrate treats the patient record as a heterogeneous collection of documents, images, and signals linked by a shared patient timeline rather than a single structured database, with retrieval systems designed to pull from whichever modality actually contains the answer to a given clinical question. This is architecturally closer to a search system than a traditional database query.

Building this substrate well requires solving unglamorous problems first, such as reliably linking a scanned document to the correct patient and encounter, before any more sophisticated reasoning can be layered on top. Teams that skip this step tend to produce systems that reason confidently over incomplete or misattributed inputs.

Interoperability rules are quietly doing the heavy lifting

Regulatory pushes toward information blocking prevention and standardized data exchange have made it meaningfully easier to pull a fuller patient record than was possible even a few years ago, though the practical experience of doing so still varies widely by health system and vendor. Some systems expose rich multimodal data through modern APIs; others still require manual document requests.

This unevenness means that companies building on this substrate today have to design for graceful degradation, since the same product may have access to a complete multimodal record at one site and only basic structured fields at another, and needs to remain useful in both conditions.

Founder takeaway

Companies building clinical AI products should treat the quality of their record-ingestion pipeline as a primary product investment, not preprocessing plumbing to be minimized before reaching the interesting modeling work. The teams with a genuine data moat over the next several years are more likely to be defined by ingestion quality than by any particular model choice.

It is worth resisting the temptation to declare a modality out of scope simply because it is hard to parse. The categories of information that are hardest to ingest, such as scanned documents and imaging, are often exactly where competitors are least likely to have already built a solution.

The cost of getting ingestion wrong early

Companies that build their core product around a narrow slice of the record often discover the limitation only after a health system pilot surfaces cases the product cannot handle, at which point retrofitting broader ingestion into an already-shipped architecture is considerably more expensive than building for it from the outset. This is a common and avoidable cause of stalled pilots.

The teams that avoid this trap treat the ingestion layer as a first-class architectural decision made before significant product work begins, rather than a component to be upgraded later once the narrower version has already proven initial traction.