Import AI 438: Silent sirens, flashing for us all
Evidence that AI agents with scaffolding match professional cybersecurity testers challenges how agencies assess AI-related threat surfaces and their own cyber testing assumptions.
Key points
- Stanford/CMU research shows AI agents with scaffolding match professional penetration testers at $18/hour versus $60/hour for humans.
- The ARTEMIS framework demonstrates frontier AI systems are systematically under-elicited - more capable than they appear without structured scaffolding.
- Remaining items cover robotics data transfer (OSMO glove) and AI-assisted chip design - limited direct APS relevance.
Implications for Australian agencies
- Monitor Cyber and security policy teams may want to monitor ARTEMIS-style elicitation research as it informs realistic threat modelling for AI-augmented offensive cyber capabilities.
- Consider Agencies procuring penetration testing services could consider how AI-assisted testing tools affect vendor capability assessments, pricing benchmarks, and contract specifications.
Implications are AI-generated. Starting points, not advice — see methodology for how they're framed.
View original source
Copied.
Appeared in:
Weekly digest, 22 December 2025
"Import AI 438: Silent sirens, flashing for us all"
Source: Import AI – Substack (Jack Clark)
Published: 22 December 2025
URL: https://importai.substack.com/p/import-ai-438-cyber-capability-overhang
This edition of Import AI leads with an essay on AI's 'illegibility' - the gap between AI's actual capabilities and how most people experience it day-to-day - before covering three research items. The most governance-relevant is a Stanford/CMU/Gray Swan study using ARTEMIS, a multi-agent scaffold, to compare AI agents against human penetration testers on a real university network. ARTEMIS significantly outperformed existing AI scaffolds and matched human professionals at roughly 30% of the cost, reinforcing the argument that today's frontier models are more capable than baseline evaluations suggest. Secondary items cover OSMO, an open-source tactile glove for human-to-robot skill transfer, and ChipMind, a data-structuring tool for LLM-assisted semiconductor design. A short fiction piece closes the issue.
Implications for Australian agencies:
- [Monitor] Cyber and security policy teams may want to monitor ARTEMIS-style elicitation research as it informs realistic threat modelling for AI-augmented offensive cyber capabilities.
- [Consider] Agencies procuring penetration testing services could consider how AI-assisted testing tools affect vendor capability assessments, pricing benchmarks, and contract specifications.
Retrieved from SIMS, 18 July 2026.