When AI forgets to forget: Patient data at risk
New research by MIT has highlighted a growing challenge for machine learning in clinical settings: some large models trained on de‑identified electronic health records (EHRs) may unintentionally memorise elements of training data in such a waythat, when prompted, could reveal sensitive patient information. Even when data is anonymised, rarity and uniqueness can increase identifiability and the risk of sensitive attributes being inferred, which could lead to confidential patient information being leaked.
The study proposes a practical testing framework for assessing memory retention in healthcare foundation models, which it then validates using a publicly available EHR model that has been trained on de‑identified data. Put simply, the study found that the more specific the details about a patient used to question the AI, the more likely it wass to reveal sensitive information, such as a real diagnosis. By contrast, simple questions based on basic demographic information did not result in the leakage of private data. This means that the more someone attempting to misuse AI already knows about a patient, and the more specific their questions are, the greater the risk of exposing sensitive information.
Despite the risk of data leaks, the study highlights the importance of context. If someone trying to misuse the AI already has detailed information and uses it to get sensitive data, the additional damage may be limited. However, the authors also pointed out specific instances that raised concerns, where questions without clear medical context still revealed private health information. The study's additional tests help distinguish between the AI truly remembering specific patient details and merely generalising from broader population data. For organisations in the UK, particularly those operating across borders, the results directly impact their compliance with GDPR rules, potential contractual risks, and how regulators might view them.
Under the UK GDPR, if individuals can be reasonably identified from an AI model or its output, then training and using these models is considered processing personal data. This places legal obligations on the organisations controlling and processing the data. In the event of a data leak, providers and users could face legal claims and regulatory scrutiny. With new legislation such as the EU AI Act and the UK's own AI regulations on the horizon, regulators will likely expect thorough assessments of data governance and privacy risks for high‑risk clinical systems. The study’s framework is a valuable tool for auditing and demonstrating that these privacy risks have been addressed before a model is released.
