Bots now get an ID card. Most scrapers won't qualify for one.

For most of the last decade, the web sorted bots by what they tried to hide. An anti-bot system looked at the IP address, the user-agent string, the TLS handshake, and the timing of requests, then guessed whether the traffic on the other end was a person or a script. Both sides ran the same loop: scrapers worked to look human, defenders worked to catch the tells. In 2026 a second contest started running alongside the first. Instead of guessing what a bot is hiding, a growing set of defenders now ask a bot to prove who it is.

The mechanism is Web Bot Auth. The IETF chartered a working group called webbotauth in early 2026, and as of August 2026 it has architecture and registry drafts in progress but no adopted RFC. That has not slowed adoption. Cloudflare, AWS WAF, Akamai, and Vercel already check these signatures in production, and OpenAI and Anthropic already sign their traffic. The standard is being enforced before it is finished.

What Web Bot Auth actually is

Web Bot Auth is a way for an automated client to prove its identity with a cryptographic signature instead of a spoofable IP range or user-agent string.

The building blocks are ordinary web cryptography. It uses HTTP Message Signatures, defined in RFC 9421, with Ed25519 keys. A signing bot attaches Signature, Signature-Input, and Signature-Agent headers to each request, and publishes its public keys as a JSON Web Key Set at a fixed location: /.well-known/http-message-signature-directory. A verifier fetches that key set, checks the signature against it, and then knows, with cryptographic certainty rather than a guess, which agent sent the request.

That certainty is the part worth sitting with, because it cuts both ways. Every signal an anti-bot system has traditionally leaned on is forgeable. An IP can be rented, a user-agent can be copied, a fingerprint can be spoofed. A valid Ed25519 signature over the request cannot be forged without the private key. The trade is that the same signature proving you are a legitimate agent also proves you are a bot, permanently and without ambiguity. There is no blending in once you have signed. For the older, guess-based layer this sits on top of, we keep a guide on why scrapers get blocked.

Who is already checking the signature

The rollout has been fast and one-sided, driven by the infrastructure companies rather than by the standards body.

Cloudflare moved first. On July 1, 2025 it added HTTP Message Signatures to its Verified Bots Program, telling bot operators that applications with well-formed Message Signatures would be approved faster than the old manual, IP-range-based enrollment. Then on August 28, 2025 it introduced signed agents, a new bot classification for a bot directed by an end user rather than operated by a single company. The named launch participants are a good map of who is building agents: the ChatGPT agent, Goose from Block, Browserbase, Anchor Browser, and Cloudflare's own Browser Rendering.

Amazon followed. On November 21, 2025, AWS WAF Bot Control added Web Bot Auth support. The default behavior is the tell: verified Web Bot Auth bots are permitted by default. The feature currently runs on CloudFront distributions and leans on two IETF drafts, one describing how a crawler shares its public keys and one describing how those keys attach an identity to a request. Akamai and Vercel round out the set of production verifiers.

So the reading side of this market is already built. Four of the largest edge and WAF providers can check a signature and act on it today, and two of the largest AI labs are signing.

This is the identity lane, not an unblocking tool

Here is where the vendor framing and the operator reality part ways. Read the announcements and Web Bot Auth sounds like a key that opens doors. For most collection teams it is closer to a name tag you cannot take off.

Signing does not get you past a bot manager. It only helps if the target has already decided to allow your identity. The signature answers the question "who is this?" with certainty, and then the site's policy decides what to do with the answer. If your identity is on the allowlist, you are waved through. If it is not, you have handed the defender a clean, unforgeable label to block, rate-limit, or log. A signed request is more identifiable than an unsigned one, not less.

That is why this is best read as the identity lane of a web that is quietly splitting into lanes. We wrote earlier about the payment lane Cloudflare opened with pay per crawl, where a declared crawler can pay to pass. Web Bot Auth is the same construction for identity: a declared, authenticated agent can be recognized and allowed. Both lanes reward bots that raise their hand. Neither does anything for traffic that depends on not being seen.

What it means for your stack

The practical answer depends on which kind of operator you are.

If you run an end-user agent platform

This is who the standard was built for. If you operate a browser agent or an assistant that acts on a user's behalf, signing is close to required now. Publish a key directory, get listed, sign your requests, and you land in the default-allow path on the verifiers that already check. The alternative is being lumped in with unidentified automation and fighting the anti-bot system on every request. For this group Web Bot Auth is a real unlock, and the earlier you adopt it the cleaner your access looks.

If you run a first-party crawler

If you crawl on behalf of a named company, for search, monitoring, or a product that cites its sources, identity plus accountability is a reasonable trade. Signing lets a cooperative target recognize you and set a sane policy for your traffic instead of treating you as noise. The cost is that you are now blockable by name. A target that decides it does not want your crawler can act on that decision precisely, where before it had to catch you first. Weigh it target by target, and pair it with the relationship work, a contact address and a stated crawl purpose, that already earns goodwill.

If you depend on blending in

If your collection depends on requests that look like ordinary browser traffic, Web Bot Auth is not for you, and pretending otherwise is a trap. Signing surrenders the exact property you rely on. Dressing a bot up as a human browser is also the route now drawing litigation, as the fight over Perplexity's Comet agent disguising itself showed. For this group nothing about the mechanism changes the day-to-day, but the context does. As more legitimate agents sign, unsigned automation stands out more, and IP reputation and behavioral signals keep doing the sorting they always did. A rising share of signed, waved-through traffic makes the unsigned remainder look more anomalous, not less.

The gate is being built before the standard is

The uncomfortable part of this timeline is the order of operations. Verification is live at four major providers, but the working group has adopted no documents yet, so the governance questions are being answered in practice before they are settled on paper. Who decides which agents are legitimate enough to list? What recourse does an operator have when a verifier declines to recognize it? Right now those answers sit inside a handful of private allowlists, not in an open registry with rules anyone can read.

None of this makes Web Bot Auth a threat or a gift on its own. It is infrastructure, and it does what infrastructure does: it makes some flows easy and others conspicuous. The teams that read the next year well will be the ones that already know which lane their traffic belongs in, and that stop expecting an identity standard to solve an access problem it was never built to touch.