Can AI Make Physicians Better at Diagnosis?

Overview

In this talk, I will present a series of studies on how large language models affect physician diagnostic reasoning, examining both the benefits AI training confers and the behavioral risks it introduces, with implications for human-AI collaboration in high-stakes domains. As LLMs become capable partners in clinical decision-making, the central challenge is no longer model accuracy, it is what happens to physician reasoning when AI enters the loop. This talk examines that question through four empirical studies. I will first discuss a randomized controlled trial demonstrating that a structured AI literacy intervention improved diagnostic accuracy by 27.5 percentage points (Nature Health, 2024). I will then present a mixed-methods study examining how physicians actually interact with LLMs in diagnostic contexts, mapping prompting strategies and reasoning patterns that explain variation in outcomes (CHI, 2026). I will next present a second RCT showing that even AI-trained physicians are susceptible to automation bias: when an LLM provides a confident but incorrect diagnosis, accuracy drops by 14 percentage points relative to unassisted physicians (NEJM AI, 2025). Finally, I will describe an ongoing RCT testing a behavioral nudge designed to recalibrate physician trust. Taken together, these studies offer a framework for understanding how highly capable but fallible AI systems reshape human expertise, with implications for any high-stakes domain where humans must decide how much to trust AI.