Cloudflare expands AI crawler management to all customers, introducing category-based controls for s...
The AMW Read
Novelty 2: Cloudflare's tiered crawler control significantly updates the baseline for content-owner tools, moving beyond binary blocking. Significance 2: The move impacts the entire data-IP substrate (cross.§F) by attempting to create a market-based compensation mechanism, though it is still a produ
Cloudflare expands AI crawler management to all customers, introducing category-based controls for search, agent, and training purposes. On July 1, 2026, the company launched a new feature allowing website operators to manage AI crawlers by purpose — "search," "agent," and "training" — moving beyond the previous binary block-all toggle. By September 15, new domains will default to blocking "training" and "agent" crawlers while allowing "search" crawlers. Alongside this, Cloudflare unveiled BotBase, a new database for displaying all crawlers to enterprise customers, and added new fields to the Content Signals extension of robots.txt for specifying content reuse scope.
Why it matters: This move formalizes a structural shift in the content-ownership battle between AI model labs and publishers. Cloudflare is positioning itself as the infrastructure layer for what the substrate calls the "content-licensing moat" pattern — enabling granular, purpose-based blocking rather than blunt refusal. This is a direct response to the ongoing tension between AI crawlers (e.g., Perplexity, Google) and site operators, which the substrate tracks as a recurring pattern in data-IP disputes. By offering "Pay Per Use" pricing — where publishers are compensated based on how often AI uses content in responses — Cloudflare is attempting to build a market mechanism that resolves the impasse between content creators and AI labs, updating the open debate on whether licensing or technical enforcement will govern AI training data.
Grounded expert take: Cloudflare's category-based segmentation is a pragmatic middle ground between total blocking and open access, but the real test will be adoption. The company's earlier claim that Perplexity was "stealth crawling" despite robots.txt blocks underscores the enforcement challenge — technical controls only work if crawlers respect them. Cloudflare's announcement of a pilot with Ceramic.ai and You.com for "Pay Per Use" suggests the company is betting on a compensation model that aligns incentives, but scaling this to thousands of publishers and dozens of AI labs faces significant coordination hurdles. The move strengthens Cloudflare's role as a neutral infrastructure arbiter, but does not fundamentally alter the power asymmetry between hyperscale AI labs and individual publishers.
