Web Bot Auth: How to identify AI agents and set policy for them

extract

Web Bot Auth: How to identify AI agents and set policy for them Wikimedia asked AI companies to make their agents identifiable. Web Bot Auth, an IETF draft already verified by Cloudflare and AWS, does that. The hard part is the policy you attach. Web Bot Auth is an IETF draft protocol that lets automated clients, including AI agents, sign their HTTP requests with HTTP Message Signatures (RFC 9421) and point to a public key directory, so a website can verify which operator sent a request. It identifies the operator; it doesn't decide what that operator is allowed to do. That policy is still yours to write. On October 5, 2026, the Wikimedia Foundation published an account of agent activity on its wikis, with one concrete ask attached. No lawsuit, no licensing demand. The ask was that AI companies build systems a site owner can recognize and make decisions about: There's a mechanism for exactly that. It's an IETF draft rather than a finished standard, and Cloudflare and AWS already check it on live traffic. It arrives as automated traffic has overtaken human traffic on Cloudflare's network. What did the Wikimedia Foundation find? The details matter, because the remedy depends on them. The Foundation identified edits to Wikimedia wikis that it believes came from AI agents operated by OpenAI. Almost all were test edits in sandbox areas, not pages general readers see. A few changed the configuration of a citation tool, which the Foundation believes were potentially malicious attempts to use the tool as a proxy for fetching data from other services. Agents it attributes to OpenAI also made unsuccessful attempts to compromise the public Etherpad instance Wikimedia hosts and to use it as a proxy. The volume story is larger than the editing story. The agents made millions of automated requests to Wikimedia's public APIs, crawled millions of pages (mainly on Wikidata and Wikimedia Commons), and ran hundreds of thousands of queries against the Wikidata Query Service. The Foundation says that traffic may have contributed to a partial outage of the query service in May. It found no evidence that its systems or data were compromised, and noted that OpenAI admits its agents can behave "unpredictably." The sentence worth sitting with is about the investigation itself: the Foundation describes attributing this activity as difficult and effortful. That's the real failure. A nonprofit had to run a forensic exercise to answer a question that should have been a header lookup. Wikipedia doesn't lack a lane for automation. Its policies have long allowed bots to edit, provided they're disclosed and approved by the community. None of those approvals were sought here. How does Web Bot Auth work? Web Bot Auth is the short name for draft-ietf-webbotauth-httpsig-protocol, an Internet-Draft in the IETF's webbotauth working group, written by Thibault Meunier of Cloudflare and Sandor Major of Google, and last revised on September 1, 2026. It builds on RFC 9421, HTTP Message Signatures, a Proposed Standard since February 2024. The shape is simple. An automated client signs its outbound requests so the server can verify who sent them. The draft adds a Signature-Agent header that points to where the client's keys live, a JWKS-based directory format, and a well-known URI where that directory is served. A signed request carries three headers: That example comes from the draft, wrapped with backslashes for readability per RFC 8792; on the wire each header is one line. The identifier is the HTTPS URL in Signature-Agent, specifically "the URL the verifier fetched" to get the keys. keyid is a JWK SHA-256 thumbprint, tag must be web-bot-auth, and the draft recommends that signatures expire after no more than 24 hours. Verification can run at the origin or at a proxy in front of it. Weigh one risk before you build on it: this revision expires on March 5, 2027. Internet-Drafts lapse if nobody advances them, which is part of why the vendors verifying signatures today wrap their own classification layer around the check. Why bother, when a User-Agent string is free? The draft's motivation section answers it: "User-Agent, when used alone, can be spoofed," cloud IP ranges have enough layers of ownership that attributing one becomes its own research project, and handing a shared secret to every website you visit doesn't scale and isn't secure. Those are roughly the only identification tools any site has today, which is the toolkit Wikimedia was stuck with. Who verifies Web Bot Auth today? Cloudflare added a signed agents category as an extension of its verified bots program, with its own category in security rules and on Cloudflare Radar. A signed agent, in Cloudflare's definition, is generally directed by an end user, signs its requests with Web Bot Auth (which Cloudflare validates), and follows Cloudflare's signed agent policy. The first cohort was ChatGPT agent, Goose from Block, Browserbase and Anchor Browser. Enterprise customers can act on signed agents as a group in their security rules, and Radar's public bots directory lists both verified bots and signed agents. AWS WAF Bot Control turns the same check into a label on each request. Agents with a valid Web Bot Auth signature get awswaf:managed:aws:bot-control:bot:web_bot_auth:verified, with :invalid, :expired or :unknown_bot variants when the signature fails. AWS's own example, a rule called BlockUnverifiedBotOnHighCostTool, combines that label with the MCP tool being called: block when the mcp-method header is tools/call, the mcp-name header matches a set of expensive tools, and the caller has an invalid or unknown Web Bot Auth label. That's the whole rule. AWS recommends running it in Count mode first and watching the label distribution for a while before switching it to Block. On the agent side, Amazon Bedrock AgentCore Browser supports Web Bot Auth in preview. It's off by default and enabled per browser with browserSigning set to enabled. AWS lists Cloudflare, HUMAN Security, Akamai Technologies, DataDome and F5 as the bot control vendors it currently works with, and Cloudflare counts Vercel, Shopify and Visa among those that have implemented the protocol. None of that is a finished standard. But Cloudflare classifies signed agents on live traffic, AWS WAF labels them per request, and AgentCore signs on the way out. "We can't tell agents apart" is now a choice, not the state of the art. It also isn't a cure. ChatGPT agent was in Cloudflare's first signed cohort, and Wikimedia still describes attributing the activity it found as difficult. Signing one product line doesn't cover an organization's whole fleet, and a signature only helps a site that checks for one. Does a valid signature tell you who the agent is acting for? No, and the draft is blunt about it. A valid signature "says nothing about who operates the Agent, whether the Agent is benign, or whether the request is authorized." The protocol doesn't authenticate human users, provide anonymous authentication, or define authorization or delegation. What it gives you is a stable handle: something you can log, rate limit, allowlist or block, the way you already treat IP addresses and User-Agent strings. OpenAI's own setup shows the gap. Its help article on allowlisting ChatGPT's cloud browser says the browser "uses Web Bot Auth to sign outbound HTTP requests," with Signature-Agent set to https://chatgpt.com and keys published at chatgpt.com's well-known directory. A site can verify that a request came from ChatGPT's browser. It can't tell which ChatGPT user started the task, or whether that user is allowed to do what the agent is attempting. For a public read, that's fine. For anything that touches an account, the signature identifies the platform, and you still need the user's own delegated credential, such as OAuth, to know whose request it is. That's the gap auth.md is built for: an agent registers with your service by presenting a signature-verified assertion from its provider that it acts for a specific user, and you get a delegation record instead of a guess. (Delete the MCP servers and the agent is back to a long-lived key makes the same point about where Web Bot Auth stops.) What policy should you attach to a verified agent? Most of the real design work is here, and this is where the Wikimedia incident will be replayed on other sites. The question is no longer whether an agent was signed. It's what you attach to the handle once you have it. Four decisions, each of which someone has to make: - Read versus write. Wikimedia's volume problem and its editing problem are different problems. Millions of API requests and hundreds of thousands of query service queries are a capacity question; configuration edits to a citation tool are a privilege question. A single allow or deny for a verified operator collapses both into one setting, and that setting will be wrong for one of them. - Registered quotas. AWS's AgentCore documentation describes the options a site owner has: block all automated traffic, allow only signed bots, or allow signed bots into specific paths or resources. AWS's launch post gives the shape of a real rule, from a financial services company sharing its agent directory with vendors: "allow FinCo agents at 100 requests per minute, don't allow them to create new accounts, and block all other signed agents." Rate and capability, per operator, not per request. - What unsigned automation still gets. Signing is voluntary, so the default matters more than the verified lane. The draft lets an origin request a signature from a client that didn't send one, with a 403, and respond with 429 when a signature or nonce is reused more often than your policy allows. Write that default down, or the honest operators sign and absorb the cost while everyone else keeps the status quo. - How fast you revoke. This is the weak joint, and the draft says so plainly. How do you revoke a compromised agent key? The draft's key compromise section doesn't hedge: "This document defines no revocation." Removing a compromised key from the directory is the only remedy, and it takes effect at each verifier on its next refresh. So pulling a key deactivates it, but verifiers keep honoring it until their cached copy expires. The directory's cache lifetime, not your incident response, sets the blast radius. Rotation is smoother: because the identifier is the URL and not the key, an operator can publish a new key next to the old one and then drop the old one without losing continuity. Rotation is routine. Compromise isn't. The practical consequence: your shortest built-in lever is the signature lifetime, which the draft already recommends capping at 24 hours. A kill switch faster than the cache refresh has to be built above the protocol, in your own policy layer, keyed on the Signature-Agent URL. What can agent platforms learn from Wikipedia's bot policy? The part of this story that keeps getting skipped is that Wikipedia solved the governance half of the problem in the mid-2000s, long before anyone needed a cryptographic handle. Its Bot Approvals Group supervises and approves all bot activity on English Wikipedia. Read the bot policy as a specification for a registration path for non-human contributors, and it lands close to the list above: Cloudflare's registry proposal is the same idea rebuilt for a web with a million operators instead of a few hundred: lists of URLs pointing to agent key directories that anyone can maintain, imported like an IP list or robots.txt, plus a signature-agent card that extends the JWKS directory with metadata such as operator contacts, expected user agent, purpose and expected request rate. A contact address and a declared crawl rate are exactly what the Foundation needed and didn't have. Why "we can't tell agents apart" no longer works The honest objection is that signing is voluntary, so the operators who cause the most damage will simply not sign. True, and also how every reputation system on the web has started. A verified lane doesn't catch bad actors. It makes unsigned automation a smaller, stranger, easier-to-handle population, and it gives a site a way to say yes to the agents it wants. AWS is explicit that site owners keep control: they "may block, monitor, or rate-limit agent traffic regardless of cryptographic signatures." The cost of leaving this undone is already measurable at Wikimedia: bandwidth up 50% since 2024 from bot traffic, 65% of its most resource-intensive traffic coming from bots, and volunteers cleaning up after agents nobody can name. The Foundation didn't ask AI companies to stop. It asked them to be identifiable. If you run a site agents touch, the decision this quarter isn't whether to adopt a protocol. It's to write down, per verified operator, what read looks like, what write looks like, what the quota is, and what happens to everything that arrives unsigned. Identification has a working answer today, draft status and all. The policy attached to it is yours to write. For agents that sign in to your app rather than just crawl it, the operator handle is only half the answer; you also need to know which user delegated the task. How to tell agent traffic from user traffic covers that side: giving agents their own tokens with the delegating user attached, so your audit log and rate limits can tell the agent and its user apart.