Medical AI explanations help clinicians and non-experts in different ways
Study reveals an expertise gap in medical AI use
An MIT-led study found that artificial intelligence generally improved skin-disease diagnosis for both non-experts and primary care providers, but explanations affected the two groups differently. Non-experts tended to accept large language model reasoning even when it was wrong, whereas clinicians were more likely to catch erroneous assistance.
The work, published in Nature Medicine, tested medical images with several forms of support: a prediction and confidence level without an explanation, similar images used to reinforce a prediction, heat maps highlighting important regions, and plain-language reasoning generated by an LLM.
Non-experts decided whether an image of a skin mole was cancerous. Clinicians faced the harder task of producing a differential diagnosis for a dermatological disease. Every explainability approach improved non-expert accuracy, largely by helping participants identify non-cancerous moles.
Why explanations can increase reliance
The improvement among non-experts was closely tied to reliance on the model. When the AI was wrong, that deference damaged performance more than a correct model improved it. The effect was strongest with LLM explanations, which also made users more confident in incorrect answers. Vague or generic explanations were often judged more convincing.
Clinicians behaved differently. They could compare an AI suggestion with an existing diagnosis grounded in their training, making them more resilient to faulty explanations. Of the methods tested, LLM reasoning produced the smallest accuracy gain for clinicians; they performed best when shown the prediction without an accompanying explanation.
Timing also mattered. Participants became more deferential when an explanation appeared before they had formed their own diagnosis. The users who relied most heavily on AI were also the weakest performers when working without assistance.
Design must reflect the user and the case
The results did not show that humans or AI were consistently superior. AI systems outperformed people when disease presentations were subtle, while humans did much better when images contained atypical symptoms or unrelated features. A fairness-constrained model designed to reduce bias against darker skin tones also significantly improved accuracy and reduced diagnostic disparities associated with skin tone.
The researchers suggest asking users to record an initial diagnostic hypothesis before displaying an AI recommendation. The system could then surface other conditions for consideration rather than presenting a confident rationale that anchors the user's judgment.
For organizations deploying AI-assisted decisions, the practical implication is to evaluate interfaces separately for experts and beginners. Accuracy testing should be paired with measurements of deference, confidence after incorrect advice, explanation timing, and users' ability to recover when the model fails.