New study warns of risks in AI chatbots giving medical advice
Benchmark scores do not predict real-world safety in high-stakes settings — a direct challenge to how agencies evaluate AI tools before deployment.
Key points
- A randomised trial of 1,298 participants found LLMs performed no better than search engines for medical decision-making.
- LLM benchmark scores failed to predict real-world performance, raising questions about reliance on standardised evaluation methods.
- UK-based research with no immediate Australian regulatory parallel, though findings are relevant to health AI risk assessment globally.
Implications for Australian agencies
- Consider Agencies developing or procuring AI tools for citizen-facing or high-stakes internal use could consider whether current evaluation methods adequately capture real-user interaction risks, not just benchmark performance.
- Monitor Health and human services agencies may want to monitor emerging evidence on LLM reliability in sensitive domains as AI health tools become more prevalent in Australian contexts.
Implications are AI-generated. Starting points, not advice — see methodology for how they're framed.
View original source
Copied.
Appeared in:
Weekly digest, 9 February 2026
"New study warns of risks in AI chatbots giving medical advice"
Source: Oxford Internet Institute – News
Published: 9 February 2026
URL: https://www.oii.ox.ac.uk/news-events/new-study-warns-of-risks-in-ai-chatbots-giving-medical-advice/
A Nature Medicine study from the Oxford Internet Institute and University of Oxford, involving nearly 1,300 participants, found that LLMs provided no measurable improvement over traditional search engines or personal judgment for medical decision-making. Users struggled to provide the right inputs, received inconsistent answers, and could not distinguish good advice from poor advice within mixed responses. Critically, models that performed well on standardised benchmarks failed in real-user interactions, with researchers calling for clinical-trial-style testing of AI systems before public deployment. The findings reinforce concerns about the gap between AI evaluation methods and real-world performance in high-stakes domains.
Implications for Australian agencies:
- [Consider] Agencies developing or procuring AI tools for citizen-facing or high-stakes internal use could consider whether current evaluation methods adequately capture real-user interaction risks, not just benchmark performance.
- [Monitor] Health and human services agencies may want to monitor emerging evidence on LLM reliability in sensitive domains as AI health tools become more prevalent in Australian contexts.
Retrieved from SIMS, 18 July 2026.