Cloudflare wants to charge AI crawlers – but can the traffic actually be controlled?

From robots.txt to a toll booth for AI

For years, websites have relied heavily on robots.txt to tell search engines and other crawlers which parts of a site they are expected to access.

Cloudflare is trying to go much further.

With AI Crawl Control, site owners can manage individual AI crawlers and decide whether they should be allowed, blocked or — through Pay Per Crawl — charged for access.

Pay Per Crawl is still in closed beta.

When an AI crawler requests content, it can present payment intent through request headers for successful access, or receive an HTTP 402 Payment Required response containing pricing information.

Cloudflare is therefore not only building technical protection between a website and automated traffic. It is also building infrastructure for a potential economic relationship between content owners and AI crawlers.

But robots.txt is not a locked door

There is an important distinction that can easily get lost in the debate over AI crawlers.

robots.txt is not the same thing as technical access control.

RFC 9309, the standard for the Robots Exclusion Protocol, explicitly states that its rules are not a form of access authorization and that robots.txt is not a substitute for proper security measures.

In practical terms, a website can say:

“Do not visit this part of the site.”

But robots.txt is not, by itself, a locked door that technically prevents a client from attempting to retrieve the content.

That distinction becomes much more important when access to content starts carrying a financial price.

Cloudflare is trying to make the rules enforceable

Cloudflare's model does not rely solely on crawlers voluntarily respecting robots.txt.

When a site owner blocks selected AI crawlers through AI Crawl Control, Cloudflare uses a custom Web Application Firewall rule to enforce that decision.

Cloudflare documents the order of processing as:

AI Crawl Control crawler blocks through WAF → Cloudflare Bot Solutions → Pay Per Crawl.

That means the blocking layer operates before the payment layer.

It is a much more enforceable model than robots.txt on its own.

But it immediately raises another question.

First, Cloudflare has to know who is knocking

To say yes, no or “pay first,” the system needs to identify the traffic it is dealing with.

On the Free plan, Cloudflare says AI Crawl Control identifies AI crawlers through their user-agent strings. This allows it to recognize well-known crawlers that identify themselves.

More extensive detection is available through Cloudflare's Bot Management technology.

The distinction matters.

A crawler that clearly states who it is and what it is doing is easier to manage than automated traffic that does not identify itself as clearly or attempts to resemble normal browser traffic.

The control problem is therefore not only about which rules the website owner sets.

It is also about how reliably the traffic can be identified.

Web Bot Auth is designed to strengthen identity

Cloudflare has also developed a response to the identity problem.

Web Bot Auth uses cryptographic signatures in HTTP messages to verify that a request comes from an automated bot.

A bot can publish its public key material and sign its requests, allowing Cloudflare to verify them against the registered identity.

That provides a stronger technical identity than relying solely on a name in a user-agent string.

For paid crawler access, this becomes particularly important.

If access and payment are associated with a specific crawler, the system needs a way to make it harder for somebody else to simply pretend to be that crawler.

Web Bot Auth is part of that infrastructure.

Not everyone goes through the controlled route

Better identification does not make the wider technical problem disappear.

TollBit, which itself sells technology for measuring and controlling AI access, has tested commercial scraping tools against major websites.

Its Scraper Audit tested 14 scraping tools against 30 sites.

According to TollBit, only three of those sites successfully blocked every scraping attempt in the test.

This is TollBit's own research and should not be treated as a universal measurement of the entire web.

But the result illustrates a central problem:

Publishing a rule is not the same thing as being able to enforce it against every form of automated traffic.

Scraping is becoming harder to recognize

Modern scraping tools do not necessarily behave like the traditional crawlers websites have dealt with for years.

TollBit describes tools capable of mimicking normal browser behavior, rotating infrastructure and using other techniques to evade conventional bot detection.

That turns identification into an ongoing contest.

On one side, publishers and infrastructure providers are trying to determine whether traffic comes from a human, a legitimate crawler, an AI agent or an unwanted scraper.

On the other side are automated systems that can change the way their traffic appears.

That is one reason why the infrastructure underneath robots.txt is becoming increasingly important.

Not all AI traffic is the same

Since July 1, 2026, Cloudflare has allowed all customers to manage three major forms of AI-related behavior separately:

Search — crawlers that collect or index content so it can later be used to answer questions.

Agent — automated activity acting in real time on behalf of a person, such as chat-fetch bots and browser-use agents.

Training — crawlers accessing content to train or fine-tune AI models.

A single bot can have more than one behavior.

That distinction matters for publishers.

A search crawler that helps people discover a website is not necessarily the same thing as a crawler collecting material for model training.

And an AI agent retrieving a page at the direct request of a user represents yet another situation.

Treating all AI traffic as one category is therefore becoming increasingly inadequate.

It is no longer enough to ask whether AI may read a page

The old question was relatively simple:

Are robots allowed to visit this page?

The new question has several layers.

Who is the automated client?

Why is it retrieving the content?

Is it doing so for search, training or on behalf of a user?

Has its identity been verified?

Does it follow the publisher's rules?

And if the site owner requires payment, can the same content still be retrieved through another technical route?

That is the environment Cloudflare is trying to build a control system for.

Cloudflare's controls do not solve everything

The fact that automated traffic can be difficult to identify does not mean Cloudflare's model is meaningless.

Cloudflare is attempting to solve several problems that robots.txt cannot solve on its own.

WAF rules can enforce blocking.

Bot-detection systems can classify traffic using more than a simple crawler name.

Web Bot Auth can provide automated requests with cryptographically verifiable bot identity.

Pay Per Crawl can make payment a condition of access for identified crawlers that a site owner has chosen to charge.

But none of those mechanisms automatically mean that every form of AI-related scraping across the web can be identified or forced into the same economic model.

That is the important boundary.

From a technical rule to an economic market

Cloudflare's ambition is larger than simply blocking a few bots.

The company is building a model in which a content owner can effectively say:

Yes.

No.

or:

Yes — but you have to pay.

That is a significant change from a web where automated systems have long been largely expected to read robots.txt and respect the publisher's stated preferences.

Once access has a price, identification and enforcement also become economic questions.

If one identified crawler pays while other automated traffic can obtain the same material outside the controlled route, that creates an obvious weakness in the market model.

And the more sophisticated identification needs to become, the more important the infrastructure that determines who is who and which rules apply becomes.

The real power question on the new AI web

The debate over AI crawlers is therefore changing.

At first, much of the argument focused on whether AI companies should be able to use publishers' content.

Then came the question of compensation.

Now another question is becoming increasingly important:

Who can actually enforce the rules?

Cloudflare is building a system in which website owners can decide who gets access and on what terms.

At the same time, developments in scraping technology show why that control cannot simply be taken for granted.

That does not mean Cloudflare's model has failed.

In fact, the problem itself helps explain why companies such as Cloudflare are building this kind of control layer in the first place.

But the power struggle on the emerging AI web will not be decided only by the rules publishers write or the price they set.

It will also be decided by who can identify the traffic, enforce the rules — and control the path between the AI system and the content.

💬 What do you think?

Should publishers be able to charge when AI systems retrieve their content — and how far should the responsibility to respect a website owner's rules extend?

Share your view in the comments.