Cloudflare Targets Mixed Crawlers as UK Adds Google AI Controls
Emerging publisher controls over AI crawler access signal a shifting regulatory and technical landscape for web content used in AI systems — worth monitoring as analogous Australian policy debates develop.
Key points
- Cloudflare will disallow mixed search-and-AI crawlers by default on ad-supported pages from September 15.
- The UK CMA imposed a publisher conduct requirement on Google on June 3, giving publishers controls over AI-search content use.
- Limited direct relevance to Australian federal agencies; most operational impact falls on publishers and platform teams.
Implications for Australian agencies
- Monitor Policy teams working on AI and digital market regulation may want to monitor the UK CMA's conduct requirement as a potential model for Australian publisher-protection mechanisms.
Implications are AI-generated. Starting points, not advice — see methodology for how they're framed.
View original source
Copied.
"Cloudflare Targets Mixed Crawlers as UK Adds Google AI Controls"
Source: Let's Data Science – AI Governance
Published: 10 August 2026
URL: https://letsdatascience.com/news/cloudflare-challenges-googles-combined-search-and-ai-crawlin-e3ea45b7
Cloudflare is updating its managed robots.txt service on September 15 to disallow crawlers that combine conventional search and generative-AI purposes by default on ad-supported pages, unless the crawler separates those functions. This follows the UK Competition and Markets Authority's June 3 conduct requirement on Google, which allows eligible UK publishers to exclude their content from AI-generated search features while retaining ordinary search visibility. The article also clarifies the distinct scope of Google-Extended (governing Gemini model training) versus the CMA-specific publisher control (covering AI Overviews and AI Mode). The practical takeaway is that publisher and platform teams must map each crawler control to the specific use it governs.
Implications for Australian agencies:
- [Monitor] Policy teams working on AI and digital market regulation may want to monitor the UK CMA's conduct requirement as a potential model for Australian publisher-protection mechanisms.
Retrieved from SIMS, 16 September 2026.