We still don’t know how people are really using AI
Independent evidence on how people actually use AI tools challenges the reliability of vendor-produced usage reports - relevant for agencies evaluating AI deployments.
Key points
- MIT-led AI Observatory aggregated 85,000+ conversational turns across 52 models to map real-world AI use patterns.
- Misinformation concentrated on Grok; coding on Claude; homework assistance on ChatGPT - use patterns vary significantly by platform.
- Research highlights a data gap: company self-reports don't capture nuances that independent observatories can surface.
Implications for Australian agencies
- Consider Agencies evaluating AI tool deployments could consider how vendor-produced usage reports may understate variation in how staff or clients actually interact with different models.
- Monitor Policy teams working on AI transparency or procurement may want to monitor whether independent usage observatories produce findings applicable to Australian government contexts.
Implications are AI-generated. Starting points, not advice — see methodology for how they're framed.
View original source
Copied.
Appeared in:
Weekly digest, 17 August 2026
"We still don’t know how people are really using AI"
Source: MIT Technology Review – AI
Published: 18 August 2026
URL: https://www.technologyreview.com/2026/08/18/1142226/how-people-use-ai/
Researchers from MIT, Stanford, and the Data Provenance Initiative have built an AI Observatory aggregating over 85,000 conversational turns from 24,521 real-world conversations across 52 models. The study finds that use patterns differ markedly by platform - Grok concentrates misinformation, Claude attracts coding, Gemini social and roleplay, and ChatGPT homework assistance. Sensitive interactions declined over the study period, suggesting improving safeguards. Crucially, individual company transparency reports do not capture these cross-platform or within-model nuances, pointing to the value of independent usage monitoring as a complement to vendor self-reporting.
Implications for Australian agencies:
- [Consider] Agencies evaluating AI tool deployments could consider how vendor-produced usage reports may understate variation in how staff or clients actually interact with different models.
- [Monitor] Policy teams working on AI transparency or procurement may want to monitor whether independent usage observatories produce findings applicable to Australian government contexts.
Retrieved from SIMS, 16 September 2026.