Most teams made the same decision quietly, some time in the last eighteen months. AI crawlers started appearing in the logs, someone wrote a rule, and the rule matched on a string. Allow ClaudeBot. Allow GPTBot. Rate-limit everything else. It was a five-minute change at the edge and it worked.
That string is now an attack surface. The Agentic Web Index, which fingerprints automated traffic across the sites it monitors, reported this week that it is observing a widespread campaign impersonating AI bots to scan websites for vulnerabilities. The requests carry the user agents of well-known crawlers. They are not those crawlers.
TL;DR
- An active campaign is spoofing AI crawler user agents to run vulnerability scans. Googlebot is the most impersonated identity at roughly 0.5% of observed traffic, with ChatGPT-User at 0.1%.
- The probed paths target AI-assisted development artefacts:
/.env.local,/.env.production,/.aws/credentials,/service-account.jsonand/firebase-adminsdk.json. - A user agent header is a claim, not an identity. Anyone can send one, and it costs nothing to do so.
- Two verification methods work today: forward-confirmed reverse DNS and vendor-published IP range files. A third, Web Bot Auth, uses RFC 9421 HTTP Message Signatures and is already enforced by Cloudflare, AWS WAF, Akamai, HUMAN and Vercel despite not yet being an adopted IETF standard.
- If your WAF or rate limiter grants a concession to a bot user agent, that concession is currently available to attackers for free.
What the campaign is actually looking for
The interesting part is not the disguise. It is the wordlist.
This is not the familiar sweep for /wp-admin and forgotten .git directories. The paths being probed are the artefacts of how software is built in 2026. .env.local and .env.production are Next.js and Vite conventions. service-account.json and firebase-adminsdk.json are the exact filenames the Firebase console hands you, and the exact filenames an AI coding agent drops into a project root when asked to wire up authentication. .aws/credentials has no business being inside a web root at all, and gets there when somebody copies an entire working directory onto a server.
As the Index puts it, the attacker “appears to be targeting credential and configuration paths used by AI coding tools”. The scanner is not sophisticated. It is current.
The disguise matters because of what many sites now do for crawlers. In the scramble to stay visible in AI-assisted search, a lot of organisations carved out a permissive path: bot user agents excluded from rate limits, waved past JavaScript challenges, exempted from managed WAF rules. Spoofing the header converts that carefully engineered concession into a free pass, and it poisons your triage, because crawler traffic is the traffic your team has been trained to ignore.
A user agent is a claim, not an identity
Nothing authenticates the User-Agent header. It is a string the client chooses. curl -A "ClaudeBot/1.0" is the entire exploit chain.
Googlebot spoofing is as old as SEO itself. What has changed is the size of the prize. Automated traffic is now around half of what hits a typical site, and a great many organisations have written explicit allow rules for a dozen agent identities they had never heard of two years ago. We have attached meaningful access decisions to an unauthenticated header, at scale, in about eighteen months. The correct mental model is email: From: is also a claim, which is why SPF, DKIM and DMARC exist.
Three ways to actually verify a bot
1. Forward-confirmed reverse DNS
The oldest method and still a good one. Take the source IP, run a reverse DNS lookup to get the PTR hostname, confirm that hostname sits under a domain the vendor documents, then run a forward lookup on that hostname and confirm it resolves back to the same source IP. Both directions have to agree, which is why a half-implementation that only checks the PTR record is worse than useless. Cache the verdict per IP rather than paying two DNS round trips per request. It fails when a vendor does not maintain PTR records for its fleet, which some do not.
2. Published IP range files
Most major operators now publish machine-readable ranges. Google maintains common-crawlers.json; OpenAI publishes separate JSON files for GPTBot, OAI-SearchBot and ChatGPT-User. Verification collapses to a prefix lookup, cheap enough to do inline.
Two things teams get wrong. First, they hardcode the ranges and forget them; fetch on a schedule and fail safe. Second, they treat “verified operator” as “wanted traffic”. Those OpenAI files are separate for a reason: GPTBot is a training crawler, OAI-SearchBot builds a search index, and ChatGPT-User is a fetch initiated by a human in a chat window. Same company, three entirely different commercial propositions. Verify the operator, then decide the policy.
3. Web Bot Auth
The direction of travel is cryptographic. Web Bot Auth has bots sign their requests using RFC 9421 HTTP Message Signatures with an asymmetric keypair, typically Ed25519. The signature covers @authority, created and expires, carries a keyid derived as a JWK SHA-256 thumbprint, and is tagged web-bot-auth. An optional Signature-Agent header points at the operator’s key directory so the origin can discover the public key, and must itself be signed. A nonce provides replay protection, and signature headers should be refused over unencrypted connections.
Status matters here, because the marketing has run ahead of the specification. The IETF working group was chartered in 2026 but has adopted no documents; what exists are individual drafts, principally Cloudflare’s. That has not stopped deployment: Cloudflare, AWS WAF, Akamai, HUMAN and Vercel already verify these signatures in production, with Amazon and OpenAI backing the effort.
Signatures are the only one of the three methods that survives the thing that is obviously coming next: agents running on an end user’s device, from a residential IP, on behalf of a person. No IP range file describes that traffic. A key does.
The signal you should actually be keeping
Verification is usually framed as a gate, and that framing loses most of its value. The Index defines spoofing as “visits that claim the identity of a known agent but fail a supported authentication method”. Read that as a detection rule rather than an access rule. A request claiming to be ClaudeBot that fails verification is not ambiguous traffic; nobody sets that header by accident. It is one of the highest-confidence hostile-intent signals in a web log, and almost nobody records it.
What to change this week
- Grep your edge configuration for user agent strings. Every rule that matches one and grants something (a rate limit exemption, a WAF bypass, a paywall concession) is a finding. Convert allow-by-user-agent into verify-then-allow.
- Log verification failures and alert on the rate. Blocking is optional. Losing the signal is not.
- Test the exact path list. Confirm every credential path returns 404 rather than 200 or a directory listing. Check your build output and container images too, because
.env.productionhas a habit of being copied into an image by aCOPY . .that nobody has read in two years. - Get the credentials off the origin entirely. A service account JSON file in a project directory is one misconfigured static handler away from being public. Secrets belong in a platform secret manager or injected at runtime, and anything ever deployed as a file should be rotated on the assumption it is already known.
- Separate crawler policy from crawler identity. Training crawlers, search indexers and human-initiated agent fetches are three different business decisions. Attach each to a verified identity rather than a string.
- Plan for signatures. If you sit behind Cloudflare, Akamai, AWS WAF or Vercel, find out whether signature verification is exposed to you and turn it on in reporting mode. If you operate a bot yourself, start signing before origins start demanding it.
The bigger shift
robots.txt was a politeness protocol, designed for a web where crawlers were a rounding error run by people you could email. It has no enforcement model because it never needed one. That web is gone. Bot identity now carries real commercial weight, and where identity carries weight, identity gets forged. The economics are stable: as long as claiming to be ClaudeBot is free and gets you a rate limit exemption, somebody will claim to be ClaudeBot.
The web will end up where email ended up, with cryptographic sender verification, a long messy transition and a decade of partial deployment. Web Bot Auth is that transition starting. In the meantime, the fix is not exotic. Verify before you concede.
At REPTILEHAUS we build and harden the edge layer for clients running everything from content platforms to AI-powered SaaS. Bot policy has moved from an SEO conversation to a security one faster than most teams have updated their WAF rules. If you want a second pair of eyes on what your infrastructure currently trusts, get in touch.
Related reading: AI crawlers are eating your bandwidth, DMARCbis and the 68% enforcement gap, and the vibe-coding data leak crisis.
📷 Photo by Taylor Vick (@tvick) on Unsplash
