โ† back to home

Can physicians trust health AI?

The question I'm working on: whether doctors can trust the AI tools entering the medical field, from the data they're trained on to the answers they give.

In progressHealth AITrust & securityHealth informatics
Research direction ยท health informatics & security
Why this question

Patients already trust it. Physicians have to decide whether they can.

AI is moving into medicine from two directions at once. Patients ask chatbots about their symptoms every day, and physicians are adopting AI inside their own work for documentation, triage and decision support. Every one of those tools produces an answer that sounds confident, whether or not it's right.

The record shows how often that confidence is misplaced. A widely used hospital sepsis alert performed far worse in independent testing than its developer reported. A man was hospitalised with bromide poisoning after following a chatbot's advice on replacing table salt. In a 2026 Oxford trial, people using chatbots for medical decisions did no better than people who didn't. And when an AI gives a wrong suggestion, even experienced radiologists often follow it.

Physicians are the ones who act on these outputs and carry the responsibility. Yet they rarely have a way to judge whether a specific tool, or a specific answer, deserves their trust.

How I think about it

Trust can break at four layers

Looking at documented failures, each one breaks at one or more of these points, and by the time an answer reaches a physician, nothing on its face shows where it went wrong.

1. Training data
Is the data representative of the patients the tool will see, and labelled for the right target?
2. The model
Does reported performance hold up under independent, local validation?
3. The output
Is this specific answer correct, complete and safe for this patient?
4. Use and security
Do people, workflows and safeguards keep a real check on the AI, and can the system be manipulated or leak patient data?
Why me

Medicine, technology and security in one place

Most people in health AI can judge either the medicine or the technology. My path gives me both, plus security, which is where trust is actually enforced. My medical training lets me judge whether a clinical output is right. My computer science and cybersecurity training lets me look at how the data, the model and the system around it are built and where they can fail. My current health informatics work, normalizing and adjudicating medical terminology for a healthcare data platform, keeps me close to how clinical data is actually structured, which is where trust in any AI built on it begins.

Related work

Where this connects

I'm also co-authoring research on early dementia detection from speech, currently under review, which gives me a first-hand view of how diagnostic models are built and evaluated. My research on security for self-driving cars asked a parallel question: how a system decides whether to trust information it can't fully verify.

Working on trust, safety or security in health AI? I'd love to hear how you approach it. Let's talk โ†’
See my workGet in touch