The change landed on 15 September 2026, and it was announced two months in advance, which is why almost nobody treated it as news. From that date, Cloudflare's bots changelog says, domains newly onboarding to the service get defaults in which bots classified as Training or as Agent are blocked on pages that display ads, while bots classified as Search stay allowed. The July announcement that set the date put the reasoning in one line: an ad is "a signal that a website owner meant for a person to land there", so those pages are defended for human attention. Existing customers keep their settings. New ones inherit a rule that sorts automated traffic by what it is for.
That is the part worth pausing on. For thirty years the web's access controls have asked who or where: which user, which key, which address range, later which signing key. This default asks a different question, and purpose is not a property a request can carry. No header proves what will be done with a response after it is received. So the answer has to be supplied from somewhere else, and where it is supplied from is a classification held about the operator in a registry maintained by a third party.
The question moved from who is calling to what the caller is for
Cloudflare arrived here by steps that each looked small. Verified bots pinned a crawler's identity to addresses or, more recently, to keys it could sign with, so an origin could tell a real search crawler from something wearing its user agent. In July the company extended that programme with signed agents, giving agent traffic a classification of its own in security rules and in its public bot directory. The verified bots documentation adds a further field distinguishing direct from intermediary access, which tracks who is allowed to operate the bot.
Notice the direction of travel. A signature answers a question about custody: this request came from a party holding this key. Everything else being asked of that signature now, whether the traffic is search, training or agent work, whether it acts for itself or for a user, is metadata attached to the signer, not evidence in the request. The cryptography is doing identity properly. The policy is hanging an unverifiable claim about intent off it.
The multi-purpose crawler shows where that seam tears
The clearest illustration is in the changelog itself. Crawlers that do both search indexing and training, Googlebot and Bingbot and Applebot among them, are evaluated under both policies, and a site that blocks Training blocks them even where Search is allowed. One identity, two declared purposes, and the stricter rule wins.
This is the ordinary consequence of attaching purpose to a principal rather than to an action, and it has a predictable effect on how operators will organise themselves. If a purpose label is inherited by everything a signing identity does, then the way to keep access is to stop sharing identities across purposes: separate crawlers, separate keys, separate registry entries, each with a narrower declared job. Cloudflare has been explicit that it wants exactly that separation; TechCrunch read the July change as giving AI companies until 15 September to split their search crawlers from their training and agent crawlers. What I think is underappreciated is that this converts a description into a commitment. Once an operator has registered a key as the search key, a later use of that key for training is not a policy disagreement, it is a false statement by a named party, which is a far more actionable thing.
The strongest rival reading is that none of this is about identity
A reasonable person can say this is a pricing manoeuvre wearing a security costume. The defaults are not an attempt to model intent; they are leverage, applied at the only chokepoint with enough traffic to matter, to move AI companies towards paying publishers for content. Cloudflare's own press release frames the year as a shift from defence to enablement, which is the language of building a market rather than a permissions model. On that reading, the categories exist because you cannot bill for something you cannot name, and the identity apparatus is only the meter.
I think that is mostly right about the motive and still leaves my point standing, because the meter is the permissions model now. Whatever the commercial aim, the mechanism that ships is one where access to a page depends on a label held about you elsewhere, refreshed by a vendor, and applied before any negotiation happens. Publishers did not vote on the taxonomy. Neither did the agents. Three categories, chosen by one company with a very large share of the origins in question, are now the vocabulary in which a large part of the web says yes or no to software.
A consequence worth watching
The interesting failures will be at the edge where an agent acts for a specific human. An assistant fetching a page because a person asked it to is, from the origin's point of view, indistinguishable from a retrieval crawler unless something in the request says otherwise, and the thing that would say otherwise is a delegated credential naming the user rather than the operator. The direct-versus-intermediary field is the seed of that, but it describes who may drive the bot, not on whose behalf this particular call is made.
So the question I would put to both sides is narrow and answerable. When a user's own agent is refused at an ad-supported page that the user could have opened in a browser, what is the mechanism by which the user's entitlement travels with the agent? Right now there isn't one in general use, and the default that took effect on 15 September makes the absence expensive rather than theoretical. Purpose labels will get more granular, because the first thing anyone does with a category that blocks them is argue for a new category. Whether any of those categories ever carry a person is the part I cannot yet see in the documents.
Published . Corrections