A startup claims it broke through a bottleneck that’s holding back LLMs
If sparse-attention approaches mature, they could lower the cost and energy footprint of LLM-based services that agencies procure or operate.
Key points
- Startup Subquadratic claims its sparse-attention architecture dramatically reduces LLM computation costs and latency.
- The quadratic scaling problem in transformer-based LLMs drives high costs that constrain Australian government AI procurement and deployment.
- Early-stage startup claim; no independent validation cited - relevance to APS practice is indirect and speculative for now.
Implications for Australian agencies
- Monitor Agencies tracking AI infrastructure costs or sustainability may want to monitor whether sparse-attention approaches gain independent validation and commercial adoption over the next 12-18 months.
Implications are AI-generated. Starting points, not advice — see methodology for how they're framed.
View original source
Copied.
"A startup claims it broke through a bottleneck that’s holding back LLMs"
Source: MIT Technology Review – AI
Published: 19 June 2026
URL: https://www.technologyreview.com/2026/06/19/1139313/a-startup-claims-it-broke-through-a-bottleneck-thats-holding-back-llms/
A startup called Subquadratic is claiming a breakthrough in LLM efficiency by replacing standard dense attention in transformer architectures with sparse attention, which selects only relevant token relationships rather than computing all pairwise combinations. This addresses the quadratic scaling problem - where doubling text length roughly quadruples computation - that makes current LLMs expensive and energy-intensive. The company claims significant speed and cost improvements for certain tasks, and its CEO suggests transformers may be displaced within a few years. No independent validation of the claims is cited in the article.
Implications for Australian agencies:
- [Monitor] Agencies tracking AI infrastructure costs or sustainability may want to monitor whether sparse-attention approaches gain independent validation and commercial adoption over the next 12-18 months.
Retrieved from SIMS, 18 July 2026.