CAISI Evaluation of DeepSeek V4 Pro
Independent government evaluation reveals a gap between vendor-reported and independently verified AI capability - directly relevant to how APS agencies assess AI procurement claims.
Key points
- CAISI's April 2026 independent evaluation found DeepSeek V4 Pro lags US frontier models by approximately 8 months.
- DeepSeek's self-reported benchmarks overstate its capability relative to CAISI's non-public, held-out evaluations.
- DeepSeek V4 is more cost-efficient than comparable US models on most benchmarks - a procurement-relevant finding.
Implications for Australian agencies
- Consider APS agencies evaluating AI model procurement or pilots could consider applying independent or held-out benchmarks rather than relying on vendor self-reported capability claims.
- Monitor Policy and security teams may want to monitor CAISI's ongoing evaluations for signal on PRC model capabilities, particularly in cyber and software engineering domains relevant to government use.
Implications are AI-generated. Starting points, not advice — see methodology for how they're framed.
View original source
Copied.
Appeared in:
Weekly digest, 27 April 2026
"CAISI Evaluation of DeepSeek V4 Pro"
Source: NIST – AI News (topic 2753736)
Published: 1 May 2026
URL: https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro
NIST's Center for AI Standards and Innovation (CAISI) published an independent evaluation of DeepSeek V4 Pro in April 2026, finding it to be the most capable PRC model evaluated to date but approximately 8 months behind the US frontier. Crucially, CAISI's non-public benchmarks - including a held-out software engineering evaluation and a cybersecurity benchmark - showed materially worse performance than DeepSeek's own self-reported results, highlighting the risk of relying solely on vendor-supplied evaluations. On cost, DeepSeek V4 Pro was more cost-efficient than the comparable US reference model (GPT-5.4 mini) on five of seven benchmarks, ranging from 53% cheaper to 41% more expensive depending on task.
Implications for Australian agencies:
- [Consider] APS agencies evaluating AI model procurement or pilots could consider applying independent or held-out benchmarks rather than relying on vendor self-reported capability claims.
- [Monitor] Policy and security teams may want to monitor CAISI's ongoing evaluations for signal on PRC model capabilities, particularly in cyber and software engineering domains relevant to government use.
Retrieved from SIMS, 18 July 2026.