New study warns of risks in AI chatbots giving medical advice
Rigorous real-world evidence that LLM benchmarks overstate safety in high-stakes settings directly challenges procurement and deployment assumptions APS agencies may hold.
Key points
- A randomised trial of 1,298 participants found LLMs performed no better than search engines for medical decision-making.
- Benchmark test performance consistently overstated real-world usefulness, with users unable to distinguish good from bad AI advice.
- Australian agencies deploying AI in health or citizen-facing advisory contexts should note the real-world testing gap this study identifies.
Implications for Australian agencies
- Consider APS agencies procuring or piloting AI for citizen-facing advisory or health-adjacent services could assess whether their evaluation frameworks include real-world user testing rather than relying solely on benchmark scores.
- Monitor Risk and assurance teams may want to monitor how Australian health regulators and the TGA respond to evidence that LLM benchmark performance does not reliably predict safe real-world deployment in healthcare contexts.
Implications are AI-generated. Starting points, not advice — see methodology for how they're framed.
View original source
Copied.
"New study warns of risks in AI chatbots giving medical advice"
Source: Oxford Internet Institute – News
Published: 9 February 2026
URL: https://www.oii.ox.ac.uk/new-study-warns-of-risks-in-ai-chatbots-giving-medical-advice/
A University of Oxford study published in Nature Medicine — the largest user study of its kind — found that LLMs provided no measurable improvement over traditional methods such as search engines when members of the public sought medical advice. The randomised controlled trial of nearly 1,300 participants revealed a persistent gap between strong benchmark performance and real-world usefulness, with LLMs frequently mixing accurate and inaccurate recommendations in ways users could not reliably distinguish. The authors call for clinical-trial-style real-world testing before deploying AI in high-stakes health settings, arguing that current evaluation frameworks are structurally inadequate for capturing user interaction complexity.
Implications for Australian agencies:
- [Consider] APS agencies procuring or piloting AI for citizen-facing advisory or health-adjacent services could assess whether their evaluation frameworks include real-world user testing rather than relying solely on benchmark scores.
- [Monitor] Risk and assurance teams may want to monitor how Australian health regulators and the TGA respond to evidence that LLM benchmark performance does not reliably predict safe real-world deployment in healthcare contexts.
Retrieved from SIMS, 18 July 2026.