Controlling AI in Healthcare

AI must be controlled. That is to say that AI accessing data and making data is a privileged activity. It is not uncommon during the early days of a new technology for that technology to be uncontrolled. It is not uncommon for Security to be seen as There are three specific moments when AI needs to be controlled. \

1. when the AI is trained on a dataset, 
2. when the AI is used to make treatment decisions (e.g. on a given Patient),
3. when the AI is used to make payment decisions (e.g., on a given Patient)

Teaching

Teaching an AI/ML/LLM with dataset needs to be controlled to prevent ingestion of data that is not authorized to be used for this purpose. With this use-case, HL7 has identified a specific PurposeOfUse that would be used to indicate this teaching/training purpose - MLTRAINING. With this code a few things can be done:

When the training is done, the authorization request is for MLTRAINING PurposeOfUse. Thus, the access control will either permit or deny such a PurposeOfUse, and the authorization would be audited as such. This PurposeOfUse would not be given to Agent that is not authorized to use this PurposeOfUse. Thus, this PurposeOfUse can't be used by other actors.

A Dataset can be marked as forbidden for MLTRAINING PurposeOfUse, which would make that Dataset unavailable for training. This, in theory, could be done down to the data artifact basis.

There is a standard in the general AI world that I helped create to tag datasets with Provenance and Authorizations including the license that would need to be followed if the data are to be ingested by an AI/ML/LLM. The Data & Trust Alliance has published this Data Provenance Standard, that is elaborated on here.

Patient based Consent on Teaching

This MLTRAINING PurposeOfUse could be leveraged in a Patient specific Consent. This would enable a Patient to indicate that they do not want THEIR data used to teach an AI. This would mean that the Access Control is more fine-grain, in that each datum pulled from the database must be checked to see if the given subject of the data (the Patient) has authorized, or did not deny authorization for AI to learn from their data.

Treatment Decisions

There are other PurposeOfUse when the AI is used during treatment (TREATDS) or payment (PMTDS) decisions. These PurposeOfUse are specific to the outcome, and are therefore distinct so that business rules or Patient Consent can allow one but not the other. They would otherwise work rather similar.

The most likely use-case is one where Patients get to indicate that they do or do-not want AI used in making Clinical Decisions (or Payment Decisions). This is diagrammed below where each Patient has a Consent with a term around PurposeOfUse of TREATDS of go or no-go; and that is used by the AI System authorization to allow the AI to make decisions, and thus look at historic patient data.

Conclusion

These PurposeOfUse already are defined for these purposes. There may be other PurposeOfUse codes that need to be defined, this is a good exercise for discussion. The above scenarios are also not the only ones, and indeed these scenarios might not be the most likely or most useful ones. My point in this article is to show that we (Security WG) have done some thinking and developed some standards codes.

Healthcare AI Transparency ballot

Healthcare use of AI needs to be Transparent, clearly labeling and attributing when patient data was created or influenced by AI. This is the goal of a new Implementation Guide going to HL7 Ballot really soon. This Implementation Guide will also be the focus of an HL7 FHIR Connectathon testing track in January.

The guide is designed for health IT developers, clinicians and institutions that use AI (including generative AI or large language models) to generate or process health data. It provides a common format so downstream systems and human users can see what data came from AI — when, how, and by which algorithm. This helps them judge whether AI-derived data are reliable, appropriate, or need further review.

Key features include:

• Tags or flags on FHIR resources (or individual data elements) to mark AI involvement.
• Metadata about the AI tool: model name and version, timestamps, confidence or uncertainty scores.
• Documentation of human oversight (for example, whether a clinician reviewed or modified AI outputs).
• Traceability: which inputs (e.g., clinical note, image, lab result) were fed to the AI, and how outputs were used to produce or update health data.
  

For stakeholders — such as patients, clinicians, and health-system administrators — the main benefit is transparency. Users can tell whether data was AI-generated or human-authored, which supports trust, safety, and informed use of AI in care.

And when the AI model or prompt is found to produce unsafe recommendations, then this transparency indications can be used to find potential problems that can then be reexamined.

AI will be used, and attribution to that use will help us deal with the data in the future.