MIT Study Warns AI Explanations Can Mislead Patients and Reduce Doctors’ Accuracy

 

Artificial intelligence explanations in healthcare do not affect every user in the same way, according to a new study led by researchers from the Massachusetts Institute of Technology and collaborating institutions.

The research found that AI explainability tools can improve diagnostic performance among people without medical training. However, much of that improvement came from users simply following the AI model’s recommendation.

Primary care providers showed a very different response. Doctors performed best when they received the AI system’s prediction and confidence score without an accompanying explanation.

The study, published in Nature Medicine, examined how patients, non-experts, and healthcare professionals use AI when diagnosing skin diseases.

AI Explanations Can Change Human Decisions

Explainable AI is designed to help users understand why a system produced a particular result.

In medical imaging, an AI system may highlight parts of an image that influenced its diagnosis. Other systems may display similar medical images or generate a written explanation using a large language model.

Researchers tested several types of AI assistance using images of skin conditions.

Participants received one of the following:

• An AI diagnosis with a confidence score
• Similar medical images supporting the prediction
• Heat maps highlighting areas of interest
• Written explanations generated by a large language model

Non-experts were asked to determine whether skin moles were cancerous. Clinicians were given a more complex task and had to provide possible diagnoses for various dermatological conditions.

Non-Experts Became More Accurate but More Dependent on AI

All tested forms of explainable AI improved the diagnostic accuracy of non-experts, particularly when identifying non-cancerous moles.

Researchers also tested an AI model designed to reduce bias against people with darker skin tones. The model improved overall performance and reduced diagnostic disparities involving skin tone.

However, the improvement came with a serious risk.

Non-experts became highly dependent on the AI system’s recommendation. When the model was correct, users often benefited. When the AI produced an incorrect result, users were more likely to follow it and make the same mistake.

Marzyeh Ghassemi, an associate professor in MIT’s Department of Electrical Engineering and Computer Science, explained that the improved performance of non-experts largely resulted from their reliance on the model.

According to Ghassemi, incorrect AI recommendations harmed performance more than correct recommendations improved it.

LLM Explanations Created the Strongest Automation Bias

Written explanations generated by large language models produced the highest level of user deference.

Participants were more likely to trust an LLM-generated explanation regardless of whether the AI diagnosis was correct.

Researchers also found that vague or generic explanations could appear more convincing to users. Participants who received LLM assistance reported greater confidence even when their answers were wrong.

This finding raises concerns about consumer-facing AI health tools. A confident and well-written explanation may appear medically authoritative even when the underlying diagnosis is inaccurate.

Roxana Daneshjou, an assistant professor of biomedical data science and dermatology at Stanford University, warned that patients with limited medical knowledge may be the most vulnerable to incorrect AI-generated explanations.

As more patients turn to AI for health information, people with the least medical expertise may be the most likely to accept an incorrect result.

Doctors Responded Differently to AI Assistance

Clinicians were less likely to follow incorrect AI recommendations or explanations.

Their medical training allowed them to compare the AI output with their own professional judgment. Doctors were therefore more capable of recognizing when an explanation did not match the evidence shown in the medical image.

The strongest performance among clinicians occurred when the AI system provided only its prediction without giving a detailed explanation.

LLM-generated explanations produced the smallest improvement in clinician accuracy among the explanation formats tested.

The result does not mean explanations are useless in medicine. Instead, it suggests that different users require different AI interfaces.

An explanation designed for a patient or beginner may not be appropriate for a trained doctor conducting a differential diagnosis.

Lead author Orson Xu, an assistant professor at Columbia University’s Department of Biomedical Informatics, said the difference comes down to how each group uses the explanation.

A clinician often begins with an independent diagnosis and evaluates the AI recommendation against medical training. A non-expert may use the AI explanation to form an opinion from the beginning.

Because of this difference, the same AI explanation can help one user while misleading another.

Timing Can Increase Automation Bias

The researchers also examined when AI assistance should appear during the diagnostic process.

Users became more likely to defer to the system when the AI explanation appeared before they had formed their own opinion.

This suggests that healthcare AI systems could be designed to ask users for an initial assessment before displaying the AI recommendation.

For clinicians, this approach may help preserve independent medical judgment while allowing the AI system to present alternative conditions for consideration.

The study also found that people who relied most heavily on AI were often the weakest performers when completing the task without assistance.

These users may benefit significantly from AI support, but they also face the greatest risk when the system makes a mistake.

Humans and AI Excelled in Different Situations

The study compared human and AI performance across different presentations of skin disease.

AI systems performed better when symptoms were subtle and difficult to detect.

Humans performed better when images contained unusual symptoms, atypical features, or unrelated visual information that could confuse the model.

This reinforces the idea that AI should support human judgment rather than replace it.

One AI Interface Will Not Work for Everyone

The researchers argue that explainability should not be treated as a universal feature that benefits every user in the same way.

A clinician may use an explanation to challenge or verify an AI diagnosis. A patient may treat the same explanation as proof that the AI is correct.

For doctors, a direct prediction with a confidence score may be more useful than a long written explanation.

For patients, AI-generated health explanations require stronger safeguards, particularly when the system presents uncertain information using confident and persuasive language.

The study’s central warning is clear: making an AI system explain itself does not automatically make it safer.

In healthcare, the effectiveness of an AI explanation depends on who receives it, when it appears, and whether the user has enough expertise to question the machine.