Cloudflare's September 15 Update: AI Crawlers to Be Blocked by Default

Starting September 15, 2026, Cloudflare is flipping the default settings for a huge chunk of the sites it runs. Every brand-new site, every site still on the free plan, and any new site added under an existing account will get affected. Cloudflare will now be sorting bots into three buckets: Search, Training, and Agent. If a page has ads on it, Search bots still get in by default, but Training and Agent bots get blocked unless the site owner flips them back on manually. And any bot that won't say which of the three it's actually doing? It just gets shut out of ad-supported pages altogether.

What Changes in Cloudflare on September 15, 2026?

Cloudflare replaces its single AI/non-AI bot toggle with three separate crawler categories, each carrying its own default. Search crawlers keep collecting content to answer questions later, so they stay allowed on ad-supported pages. Training and Agent crawlers, along with any crawler that won't declare its purpose, get blocked on those same pages by default.

Which Sites Get the New Defaults?

Three groups inherit the new defaults automatically: brand-new Cloudflare customers, new sites added under an existing account, and every site still on the free tier. Existing paying customers keep their current bot settings until they log in and change them.

Why Is Cloudflare Making This Change Now?

Bot traffic passed human traffic on the open web for the first time this year, months earlier than expected. Cloudflare points to mixed-use crawlers, bots that blend search, training, and agent behavior into one, as the real problem. Because these crawlers don't separate out what they're actually doing, site owners are stuck with an impossible choice: block them entirely and disappear from AI search results, or allow them and hand over their content for free training and reuse with zero compensation. Transparent AI companies lose out too, since they get lumped in with, and penalized alongside, bots that refuse to disclose their intent.

"Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge," said Cloudflare co-founder and CEO Matthew Prince, announcing the policy (Cloudflare, July 2026).

Cloudflare also argues the change levels a playing field it says favors the largest search engine, which it estimates has access to roughly twice the web content available to other AI companies, because staying discoverable in search has effectively required accepting AI training too (TechCrunch, July 2026).

What Happens to Multi-Purpose Crawlers Like Googlebot?

Multi-purpose crawlers get evaluated under every policy they touch, not just one. If a site blocks Training crawlers, crawlers such as Googlebot, Applebot, and Bingbot get blocked too, even on sites that still allow Search, because those bots currently perform more than one function at once (Help Net Security, July 2026).

Site owners who want Google Search traffic but not Google's AI training crawlers will need Google to ship a bot that separates the two before September 15 changes anything for them.

What Is Pay Per Use, and Why Does It Matter?

Pay Per Use is Cloudflare's answer to the compensation half of the problem. It replaces last year's Pay Per Crawl marketplace, which charged AI companies per fetch regardless of whether the content did anything for the answer that followed.

Under Pay Per Use, publishers get paid when their content actually shapes an AI answer, not just when a bot reads it. Launch partners Ceramic.ai and You.com pay publishers per query and per on-demand content access, respectively.

Agencies already running automated SEO monitoring on client sites are best placed to catch which pages start earning under the new model, since the payouts depend on citation, not crawl volume.

How Should Site Owners Prepare Before the Deadline?

Four steps cover most of what a site needs before September 15:

Check current AI Bot Control status. Log into Cloudflare's Security settings and see exactly where the account stands today, including whether the legacy "Block AI bots" toggle is still on, since it now enforces against the new categories too.

1. Decide crawler by crawler, based on business model. For each AI crawler, choose whether Search, Training, and Agent access should stay open on ad-supported pages. Sites that don't run on ad revenue have little reason to block, since discoverability matters more than protecting ad inventory.
2. Watch for the Googlebot trap. Multi-purpose crawlers like Googlebot, Applebot, and BingBot get judged by every behavior they declare, so blocking Training can also block regular search indexing under the most-restrictive-rule policy. Confirm this doesn't happen before locking in a setting.
3. Lock in current settings if they should stay untouched. If the existing configuration is exactly what's wanted, opt out of the new defaults before September 15, or they'll be overwritten automatically.
4. Audit any site that moved to the free tier this year. These sites inherit the new defaults regardless of what was configured before, so past settings won't carry over unless explicitly reconfirmed.

September 15 doesn't ask site owners to block AI. It asks them to decide, crawler by crawler, what they're comfortable giving away for free. The sites that check their settings this week keep that choice. The ones that don't will find Cloudflare has already made it for them.

Alpesh D.

Co-founder, PilotDeck

Alpesh builds the growth engine behind PilotDeck. He's spent years helping startups go from zero traffic to consistent organic growth, combining SEO, AI automation, and content strategy into a repeatable system.