Synthetic data hospitals can actually share
A generative pipeline that produces anonymised, multimodally-conditioned medical images — and the evaluation framework that proves the images are useful without being memorised.
The problem is not the model, it's the sharing
Hospitals hold the data that would make medical machine learning work, and they cannot share it. Not because the technology is missing but because the governance is, correctly, immovable: patient records do not leave the institution. Every multi-centre study, every attempt to train on a population larger than one hospital's catchment, runs into the same wall.
Synthetic data is the obvious escape hatch, and it is usually oversold. Generate images that look like patient scans, share those instead, everyone wins. The reason it mostly does not work is that "looks like" is doing far too much load-bearing work in that sentence — in two directions at once.
Two failure modes, pulling opposite ways
A synthetic dataset can fail by being too different from the real thing: pretty images with the wrong statistics, on which models train happily and then collapse on real patients. This is the failure everyone tests for, because FID is easy to compute.
It can also fail by being too close. A generative model that has memorised its training set will reproduce real patients' anatomy in its outputs. The dataset looks excellent by every distributional metric — it scores well precisely because it is leaking — and you have not anonymised anything. You have built a lossy, deniable copy of the records you were not allowed to share, and put a certificate on it.
Fidelity and privacy are measured by the same quantity pointed in opposite directions. Any evaluation that reports only one of them can be satisfied by a model that is failing catastrophically at the other.
The pipeline
I built a secure, privacy-preserving synthetic image pipeline spanning GANs, diffusion models, and latent consistency models, with multimodal conditioning so generation is steerable rather than a lottery — you specify the clinical characteristics you need and get a cohort with those properties, instead of sampling until something useful appears.
It operates on anonymised data and targets two concrete goals: cross-hospital data sharing, and training on the result. The RSNA work showed the second goal is real — synthetic generation improving downstream performance in multi-centre federated settings, where it also serves to offset the utility that differential privacy takes away.
The evaluation framework
This is the part that makes the pipeline defensible, and it was the subject of my MSc thesis (Novartis Industry Diplompreis). Rather than a single score, it is multifaceted by construction, spanning:
- Quality — are the images clinically plausible, not merely photorealistic?
- Diversity — does the model cover the distribution, or has it quietly collapsed onto a comfortable mode?
- Memorisation — quantified directly, as its own first-class axis, rather than assumed away.
The memorisation axis became its own line of work: a memorisation detection benchmark for generative image models, presented at the NeurIPS 2024 Safe Generative AI Workshop. If you are going to claim a synthetic dataset is safe to share, the claim has to be measured, and it has to be measured against a benchmark that was not designed by the person making the claim.
Putting experts in the loop
Automated metrics agree with clinical judgement right up until they don't. I built and deployed a web platform running expert Turing tests — clinicians distinguishing real scans from synthetic ones — to align the generative models against human judgement and to keep the automated metrics honest about what they were actually measuring.
Where it connects
This thread is not separate from the differential privacy work; it is the other half of it. DP bounds what a model can leak. Memorisation auditing measures what a model did leak. One is a guarantee, the other is evidence, and a deployment that wants to be trusted with patient data needs both.