For the first time in internet history, machines are requesting more web pages than humans. Cloudflare’s June 2026 data confirms it: 57.5% of HTML traffic now comes from automated requests, with AI and LLM crawlers quadrupling their share in just eight months — from 2.6% to 10.1% of all bot traffic.
If your hosting bill has crept up without a corresponding jump in real visitors, you are almost certainly paying to serve content to bots that will never convert, never subscribe, and never buy.
Here is what your development team needs to know — and do — right now.
TL;DR
- AI crawlers now account for over 57% of web traffic — the first time bots have outnumbered humans on the internet.
- A single training crawler like GPTBot can consume up to 40% of your bandwidth during a deep crawl, costing SMEs thousands per month in wasted egress.
- The critical distinction is between training crawlers (GPTBot, CCBot, anthropic-ai) and search crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot) — block one, keep the other.
- robots.txt alone is not enough: 72% of AI crawlers violate it. You need WAF rules, rate limiting, and active monitoring as enforcement layers.
- AI referral traffic grew 975% year-on-year — blocking everything means becoming invisible in AI-powered search experiences.
The Scale of the Problem
Traditional search engine crawlers — Googlebot, Bingbot — have always consumed bandwidth. But they were predictable, well-behaved, and directly tied to discoverability. AI crawlers are a different animal entirely.
Training bots like GPTBot, CCBot, and Bytespider are designed to scrape entire datasets for model weights. They bypass CDN caches, hammer unoptimised endpoints, and increase P99 latency for your actual users. This is what the industry calls “Shadow Crawl” — a massive drain on resources that yields zero immediate referral traffic, zero conversions, and zero revenue.
The numbers are stark. A standard training bot can consume up to 40% of a site’s bandwidth during a deep crawl cycle. For larger websites, AI crawler bandwidth can reach 1–10 TB per month, translating to €900–€9,000 in monthly hosting costs with nothing to show for it.
One organisation documented cutting daily outbound bandwidth by 75% — from 800 GB to 200 GB — simply by blocking aggressive AI crawlers at the edge. That saved roughly €1,400 per month.
Training Crawlers vs Search Crawlers: The Distinction That Matters
The single most important thing to understand in 2026 is that major AI companies now run separate crawlers for training and search. This changes everything about how you should approach bot management.
Here is the breakdown:
| Company | Training Crawler (Block) | Search Crawler (Consider Allowing) |
|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot, ChatGPT-User |
| Anthropic | anthropic-ai | Claude-SearchBot, ClaudeBot |
| Google-Extended | Googlebot (unchanged) | |
| Common Crawl | CCBot | N/A |
| ByteDance | Bytespider | N/A |
| Perplexity | — | PerplexityBot |
Blocking training crawlers does not affect your Google search rankings. These bots are entirely separate from Googlebot. What it does affect is whether your content ends up in future LLM training datasets — a decision that is yours to make.
The smart 2026 strategy: opt out of training, stay visible in AI search.
A Practical robots.txt Strategy
Start with your robots.txt. It is the polite layer — well-behaved crawlers honour it, though it has no enforcement mechanism against those that do not.
# Block AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: cohere-ai
Disallow: /
# Allow AI search crawlers
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Standard search engines — business as usual
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /This configuration lets your content appear in AI-powered search results — with attribution and potential referral traffic — whilst keeping it out of future training runs.
robots.txt Is Not Enough
Here is the uncomfortable reality: studies show that up to 72% of AI crawlers violate robots.txt rules. The file is a gentleman’s agreement, and not all crawlers are gentlemen.
You need enforcement layers:
1. WAF Rules (First Line of Defence)
Web Application Firewall rules are evaluated before robots.txt is ever read. If your site sits behind Cloudflare, AWS CloudFront, or a similar CDN, you can block by user-agent string at the edge — meaning the request never reaches your origin server and never consumes your bandwidth.
Configure your WAF to block known training crawler user-agent strings. This is more reliable than robots.txt and has the added benefit of reducing origin load.
2. Rate Limiting
For crawlers you want to allow (like AI search bots), implement rate limiting. A search bot does not need to make 500 requests per second. Set sensible thresholds — perhaps 10 requests per second per user-agent — and let anything above that hit a 429 response.
3. Active Monitoring
New AI crawlers appear regularly. Monitor your access logs or CDN analytics for unknown user-agent strings making high-volume requests. Classify and respond accordingly. Set up alerts for sudden bandwidth spikes — they are often the first sign of a new or misbehaving crawler.
The SEO Balancing Act
The temptation is to block everything. Resist it.
AI referral traffic grew 975% year-on-year from January 2025 to January 2026. When someone asks ChatGPT, Claude, or Perplexity a question and your site appears as a source, that is qualified traffic — often more qualified than a generic search result click.
Completely blocking AI crawlers means your content becomes invisible in AI-powered search experiences. Given that a growing share of users now start their research in AI tools rather than traditional search engines, this invisibility carries a real business cost.
The right approach is selective:
- Block training crawlers that consume bandwidth without direct benefit
- Allow search crawlers that drive referral traffic and citations
- Rate-limit everything to prevent abuse
- Monitor continuously and adjust as the landscape evolves
Beyond robots.txt: The llms.txt Standard
A newer initiative gaining traction is llms.txt — a proposed standard that lets you provide structured, AI-readable information about your site. Think of it as a curated summary layer: instead of letting crawlers scrape your entire site indiscriminately, you offer a concise, structured document that tells AI systems what your business does, what content matters, and how to cite you.
It is still early days, but forward-thinking teams are already adding llms.txt alongside their robots.txt. If AI-powered search is going to cite your site, you might as well control the narrative.
What This Means for Your Budget
For an SME running a site that serves 100,000 requests per month, the maths is straightforward. If 57.5% of that traffic is bots, you are paying server costs for roughly 57,500 requests that generate zero revenue. On a €200/month hosting plan, that is approximately €115 per month wasted on bot traffic.
Scale that up. A site serving 1 million requests per month on a cloud hosting provider with pay-as-you-go bandwidth pricing can easily see €500–€2,000 per month in unnecessary costs. And if you are on shared hosting, those bot requests are eating into your allocated resources, potentially slowing your site for real users and hurting your Core Web Vitals scores.
This is not a theoretical problem. It is a line item on your infrastructure bill.
An Action Checklist
- Audit your current bot traffic — check your server logs or CDN analytics for the percentage of AI crawler traffic
- Update your robots.txt — implement the training-vs-search split described above
- Configure WAF rules — block training crawlers at the edge, before they reach your origin
- Set up rate limiting — cap request frequency for allowed AI crawlers
- Add monitoring alerts — flag unexpected bandwidth spikes and new unknown user-agents
- Consider llms.txt — give AI search tools a structured summary of your business
- Review quarterly — the AI crawler landscape changes rapidly; your policy should too
How REPTILEHAUS Can Help
Bot traffic management sits at the intersection of DevOps, security, and SEO — exactly the kind of cross-cutting problem that falls through the cracks when teams are focused on shipping features. Our team helps businesses audit their traffic, configure edge-level defences, and build monitoring that keeps bot costs under control without sacrificing AI search visibility.
If your hosting bill does not make sense, or if your site performance has degraded without an obvious cause, get in touch. We will help you work out what is real traffic and what is just feeding the machines.
