Clinical records joined across scans, notes, tests and outcomes

The record

A hospital keeps one patient's story in several separate systems. Scans sit in an imaging archive, doctors' notes in the electronic record, blood results in a laboratory system, tissue slides in a pathology store. What happened later, such as a death, is often held by a registry outside the hospital. The systems share little more than a patient number. A poster about one London hospital trust's data lists 10+ billion rows of clinical facts across isolated systems and 8+ million medical images without standardised linkage ¹.

A worked example from a published dataset. Researchers at Stanford found about a hundred and fifty-six thousand cases of a lung-clot CT scan in their hospital's research store ². They sampled about twenty-six thousand scans with their reports and kept the twenty-four thousand or so that had a report with a summary section ². After further cleaning, the set held about twenty-three thousand scans for about nineteen thousand patients ². Each scan is joined to the radiologist's summary and to the patient's coded record, which runs to more than two hundred and twenty-five million rows, of which 4,410 are death records ³. To allow release, the authors say they removed all unstructured text, and the set is offered "for non-commercial use under a data use agreement" ².

Why it might matter for training AI

A model shown a scan beside the notes and results, and told what later happened, can learn what no single system contains. Some buyers pay for this. A biotechnology company building a cancer model announced agreements with a drug company and Tempus, whose de-identified cancer data it will use, that "include $200 million in data licensing and model development fees" to Tempus ⁴. A broker, Protege, says it supplied an unnamed model builder, under the heading "Millions of Verified Imaging Studies for Pre-Training in 30 Days" ⁵. In China, a Shandong hospital signed a deal to sell a liver-disease set of 1000余例 (a little over a thousand cases) for 3万元, which is thirty thousand yuan, to a firm building a diagnostic model, according to the university the hospital is attached to ⁶.

In the sources read, no frontier AI lab was named as a buyer of hospital records. The named buyers are drug companies and medical-AI firms. One published benchmark for health models, HealthBench, draws on doctors' time: its responses are graded against "rubrics created by 262 physicians" ⁷.

Who holds it and who may share it

The hospital or clinic holds the record. What it may do with it differs by country.

United States. The health privacy rule says a hospital or its contractor "may not sell protected health information" except in compliance with the rule's section on authorisations, which this provision cites without setting out its terms ⁸. One route the rule gives for treating information as non-identifying is removing a list of identifiers ⁹. The list includes "Biometric identifiers, including finger and voice prints" ⁹.

Australia. The privacy regulator examined I-MED, a radiology chain that passed "less than 30 million patient studies" to an AI joint venture without telling patients ¹⁰. It found the data "had been de-identified sufficiently that it was no longer personal information", closed its preliminary inquiries, and added that its report "should not be taken as an endorsement" ¹⁰. Its guidance for model developers says sensitive information "generally requires consent to be collected" ¹¹. The Privacy Act itself was not opened.

Britain. NHS England paused data processing for a model trained on the de-identified data of 57 million people after two doctors' bodies raised concerns; Politico reports that the family-doctor data involved had been approved for Covid research only ¹².

China. The personal information law lists 医疗健康 (medical and health) information as sensitive, and says that processing sensitive information needs 单独同意, separate consent ¹³. A 2019 regulation says 禁止买卖人类遗传资源, that buying and selling human genetic resources is banned, though it adds that paying or receiving a reasonable cost fee for providing or using them in scientific research under the law is not treated as a sale ¹⁴. It bars foreign organisations and individuals, and bodies they set up or control, from collecting or storing them in China or providing them abroad, and says a foreign body that needs them for research must work in cooperation with a Chinese one ¹⁴. Passing genetic-resource information to foreign bodies is allowed on filing it with the science ministry, with a security review where public health, national security or the public interest may be affected ¹⁴. Whether ordinary scans and notes count as genetic resources was not established. A 2024 rule says a data handler must apply to the national internet regulator for a data-export security assessment if, in a year, it sends abroad the sensitive personal information of 1万人以上, ten thousand people or more; the rule also lists exemptions ¹⁵.

Who is already there

• Epic Cosmos, a pool of de-identified records from many health systems, has had a family of models pretrained on 118 million of its patients ¹⁶.
• Truveta, which is owned by American health systems, pools their records and says 192 million images can be linked to each patient's history ¹⁷.
• Brokers include Protege, which lists notes, consultation audio and scans ¹⁸, Gradient Health ¹⁹ and Segmed ²⁰.
• Yidu Tech, which provides data platforms to Chinese hospitals, says it has processed nearly 9 billion authorised records ²¹. A Shenzhen hospital is reported to have completed a trade of a health data product through the city's data exchange ²², and a vendor's catalogue lists hospital records it calls anonymised for training medical models ²³.
• In Australia, the I-MED scans went to Annalise.ai, then a joint venture between I-MED and Harrison.ai ¹⁰. Heidi, a consultation-recording tool, says it "does not use consultation data, session transcriptions, or generated notes to train its AI model in any way" ²⁴.
• Free research sets exist, such as a Boston hospital's more than two hundred and twenty-seven thousand chest X-ray studies with reports ²⁵.

The record is not untouched.

What would have to be stripped

Names, dates, addresses, record numbers, text burned into images, voices, and people named in notes. I-MED hashed identifiers, shifted dates and redacted text near the edges of images, and the regulator still recorded a "very small number" of instances where personal information was shared in error ¹⁰.

Removing names does not settle it:

• A network told whether two chest X-rays came from the same person "with an AUC of 0.9940 and a classification accuracy of 95.55%", where an AUC of one would be perfect ²⁶.
• With every detected identifier masked, an attacking model "still reidentified 9% of clinical notes" ²⁷.
• The better of two GPT models asked to find identifiers in notes reached a recall of 0.8318, meaning it found about five in six ²⁸.

Joining the scan to the notes and the outcome is what makes the record valuable, and it may also be what makes each patient easier to single out.

What stands in the way

• The biggest pools already have models trained on them ¹⁶. Whether their holders would sell the joined record is not shown.
• Prices do not only rise. One drug company "replaced two scheduled $42 million annual fees with three $14 million payments" for Tempus data ²⁹.
• Sharing that a regulator clears may still change what the holder does. Crikey reported that I-MED "has since quietly begun asking customers for consent" ³⁰.
• The joining is engineering done inside each hospital. The London trust lists "High initial implementation costs" among its limits ¹.