Cloudflare will default to blocking AI crawlers that mix search indexing with model training on ad-supported websites starting September 15, 2026, according to the company's own announcement of the policy. The change forces AI developers to either license content through Cloudflare's Pay Per Crawl and Pay Per Use marketplaces or lose automated access to a large share of the web's publishers outright.
What exactly changes on September 15?
Cloudflare's July 2026 announcement separates two permissions that AI crawlers have historically bundled: indexing for search results and scraping for model training or agent use. Under the new default, "mixed-use" crawlers — bots that blend search, training, and agent functions in a single fetch — will be blocked automatically on pages that carry advertising, unless a site owner opts back in. The rule applies to new Cloudflare customers, newly created sites belonging to existing customers, and all free-tier accounts, according to the company's announcement, which Cloudflare CEO Matthew Prince framed around the volume of automated traffic: "Now that the majority of traffic on the Internet is non-human, we must go further and act faster," Prince said, as reported by TechCrunch.
How does Cloudflare's payment mechanism actually work?
The underlying infrastructure is Pay Per Crawl, which Cloudflare describes in its own technical documentation as a system built on HTTP status codes rather than a separate billing platform. A crawler that requests paid content without an accepted payment receives an HTTP 402 "Payment Required" response carrying a crawler-price header; a successful transaction returns a 200 response with a crawler-charged header. Crawlers authenticate using Web Bot Auth, an Ed25519-based signature scheme that requires bots to register a public key and sign each request. Publishers set one flat, per-request price across their domain and choose, crawler by crawler, to allow free access, charge for it, or block it entirely, per Cloudflare's documentation. That system launched in private beta in July 2025 and has since been folded into a broader Pay Per Use framework, under which publishers can also be compensated when their content is cited inside an AI-generated answer or purchased by an agent for premium information, according to Forbes's reporting on the rollout. Cloudflare has named two early participants, Ceramic.ai and You.com, and has not disclosed transaction volume or aggregate publisher revenue to date.
Why are publishers backing a mechanism like this now?
The economic case publishers are making rests on a traffic-referral gap that has widened over the past year. Forbes, citing industry data on crawler-to-referral ratios, reported that Anthropic's crawlers fetch roughly 11,122 pages for every referral sent back to the originating site, that AI chatbot referrals generate about 96% less traffic than a traditional search click, and that users click through to a cited source only about 1% of the time. The same reporting put publisher traffic declines attributable to AI answer surfaces at a range of 20% to 90% over the prior year, without specifying a single study or methodology behind the composite figure. That imbalance — heavy automated consumption of published content against a near-collapse in the traffic that used to fund it through advertising — is the underlying rationale Cloudflare has offered for treating crawler access as a metered, billable resource rather than a free input, according to its own announcement material.
What structural advantage does the policy leave in place?
The rule change does not treat all crawlers equally, and that asymmetry is the strongest objection to it. Google continues to operate Google-Extended, a crawler permission that lets a site opt out of AI training use without affecting its inclusion in Google Search — a separation TechCrunch's reporting noted gives Google a structural advantage unavailable to smaller AI developers, since Google's search index already carries what Prince, in comments reported by TechCrunch, described as access to "2x more information" than competing AI companies gather independently. A publisher that blocks Google's training crawler risks nothing in search visibility; a publisher that blocks a mixed-use crawler from a newer AI company risks losing that company's product entirely as a discovery channel, with no equivalent search fallback to protect. Cloudflare's framework narrows the free-riding problem for challengers to the search incumbent but does not eliminate the incumbent's separate, pre-existing indexing advantage. For a newer AI company without an existing search product, the calculus is different: it has no comparable index to fall back on, so its only paths to publisher content are paying for what Cloudflare enables, negotiating direct bilateral licensing deals of the kind larger publishers have already struck with major AI developers, or operating with a smaller, license-limited corpus than an entrenched competitor.
Will this translate into meaningful publisher revenue?
Expectations documented so far are directional rather than quantified at the individual-publisher level. Forbes reported that close to 70% of publishers surveyed anticipate AI licensing will generate some revenue within three years, though the underlying survey's sponsor, sample size, and methodology were not specified in that reporting. Cloudflare itself has not published pricing benchmarks, adoption figures, or projected revenue per domain for Pay Per Crawl or Pay Per Use. What is verifiable is the mechanism: a flat per-request price set unilaterally by each publisher, with no reported floor, ceiling, or market-clearing process disclosed publicly, and a September 15, 2026 deadline after which the default access AI companies have enjoyed narrows considerably for any crawler that cannot cleanly separate its search and training functions. That structure also means individual publishers bear the burden of price-setting with no benchmark to reference, a gap that leaves room for both underpricing by publishers eager for any revenue and overpricing that some AI companies may simply route around by dropping a domain from their crawl set entirely rather than paying.
What to watch next
The near-term signal will be whether major AI developers beyond Ceramic.ai and You.com sign onto Cloudflare's marketplace before the September deadline, or instead route around it by splitting their crawlers into distinct search and training identities — the compliance path Cloudflare's own policy explicitly permits. A second signal is whether rival infrastructure and licensing efforts — Forbes's reporting named TollBit, Microsoft, and ProRata alongside Cloudflare as competing standards-setters in this space — converge on compatible authentication and pricing conventions or fragment the market into incompatible metering systems that raise compliance costs for AI companies and administrative overhead for publishers alike.
For a related media news perspective, read British Royal music featured on Gossip Stone TV reality TV show by Debbie Wingham.