A July 2026 study tested the top 1,000 websites and found 40.9% unreadable to GPTBot, the crawler that feeds ChatGPT.1
Among news sites carrying a robots.txt, 82.4% restrict at least one AI crawler - the most defensive category measured. GPTBot appears in more disallow rules than any other AI bot, followed by CCBot, ClaudeBot and Google-Extended.1
Two-fifths of the most visited sites on the internet have made themselves invisible to the system answering the questions their content used to answer.
The split that most robots.txt files miss
The important technical development is not that blocking happens. It is that training and search are now served by separate crawlers, which makes the decision considerably finer than a single allow-or-deny.2
You can block the crawlers that ingest content for model training while remaining eligible for citation in AI search results. Blocking GPTBot, Google-Extended and ClaudeBot restricts training. Permitting OAI-SearchBot, Claude-SearchBot and PerplexityBot keeps you eligible to be cited in the answers those systems generate.2
This matters because a great many robots.txt files were written before the split existed, or were copied from a template that blocks by keyword. An organisation that blocked "AI bots" in 2024 may have removed itself from AI search citation without intending to, having meant only to withhold training data.
That is a configuration error with commercial consequence, and it is invisible unless someone checks.
Why news blocked hardest
The 82.4% figure for news is the highest of any category, and the reason is structural rather than ideological.
News has the shortest half-life and the most complete substitutability. A generated summary of a news event delivers essentially the whole product - there is no residual reason to visit, no deeper layer the summary omits. For a news publisher, being read by a model and being read by a person are close to mutually exclusive.
Compare that with documentation, research or reference material, where the summary answers one question and the source holds a hundred more. Those categories have far less reason to block, and the blocking data reflects it.
The useful generalisation for anyone else: the case for blocking is strongest where your content is completely substitutable by a summary of it. The Content desk's Feature 21 sets out which categories those are.
What blocking actually buys, and over what horizon
The case for blocking is usually argued as compensation - they take our content, we get nothing, so we withhold it. That framing is emotionally satisfying and analytically incomplete, because the two crawler classes operate on completely different time horizons.
Training is a long game with diffuse returns. A model trained on your material in 2026 may surface your expertise, your product category and your framing for years, in answers that never cite you and never send a visit. You cannot measure that and you cannot invoice for it. You also cannot get it back if you opt out and the model simply learns the category from a competitor instead.
Search is immediate and attributable. A search crawler fetching your page to answer a live question either names you or does not, and the outcome happens today.
This asymmetry explains why the two decisions should be taken separately and why collapsing them is expensive. The cheap decision to defend is blocking training. The costly decision, usually taken by accident, is blocking search.
It also explains the news pattern. A news publisher's content has almost no long-tail training value to protect and enormous immediate substitutability to lose, which makes aggressive blocking rational for them and a poor template for anyone else copying their robots.txt.
Blocking is now a business decision, not a technical one
Cloudflare offers one-click blocking of AI crawlers and a pay-per-crawl model that lets publishers charge AI companies for access.2
That second item is the significant one. It converts a binary - allow or deny - into a price, and a priced input behaves very differently from a free one.
Whether pay-per-crawl produces meaningful revenue for anyone below the largest publishers is an open question and we have no data on realised rates. But the mechanism changes who should be making the decision. A file that determines whether your content is available to the systems answering your market's questions is not an engineering artefact.
Whose interest this serves
The crawler-control guidance in this feature comes largely from infrastructure vendors and SEO consultancies - parties who sell the tooling and advice for managing exactly this. The blocking-prevalence study is reported through the same channels. Marketing Legendary publishes original research and wants to be cited, which makes us structurally biased toward permitting search crawlers and against the blocking case. We have tried to state the case for blocking at its strongest anyway.
What to do about it
Read your own robots.txt this week. Not the version in the repository - the one being served. Confirm which of the six crawler classes above you currently permit, and whether that reflects a decision anyone actually made.
Separate the two decisions explicitly. Training and search are different bargains with different returns. Treating them as one question guarantees getting one of them wrong.
Default to permitting search crawlers unless you are news. If your content is not completely substitutable by a summary, blocking search crawlers removes you from the answer without protecting anything. The Content desk's citation data is the argument.
Do not assume compliance. robots.txt is voluntary and user agents are self-declared. If exclusion genuinely matters commercially, it needs enforcement at network level, not a directive in a text file.
Move the decision out of engineering. Whoever owns commercial licensing should own this, because that is now what it is.
| Measure | Value | Grade |
|---|---|---|
| Prevalence of blocking | ||
| Top 1,000 sites unreadable to GPTBot | 40.9% | Reported, July 2026 |
| News sites with robots.txt restricting an AI crawler | 82.4% | Reported |
| Most-blocked crawler | GPTBot, then CCBot, ClaudeBot, Google-Extended | |
| The crawler split | ||
| Training crawlers | GPTBot, Google-Extended, ClaudeBot, CCBot | |
| Search crawlers | OAI-SearchBot, Claude-SearchBot, PerplexityBot | |
| Blocking training while remaining citable | Possible | Reported |
| Commercial layer | ||
| One-click blocking | Available at CDN level | |
| Pay-per-crawl | Available; realised rates not disclosed | |
| Not established | ||
| Definition of "unreadable" in the study | - | Method not obtained |
| Rate of crawler compliance with robots.txt | - | No data located |
| Revenue realised from pay-per-crawl | - | Not disclosed |
How we did this
What this doesn't prove
- That blocked content is actually excluded. robots.txt is voluntary. We found no data measuring compliance rates.
- Whether blocking helps or harms the blocker. No study we found compares commercial outcomes for sites that block against those that do not.
- What "unreadable" means in the source study. The 40.9% could describe policy, network blocking or fetch failures, and these have different implications.
- That the crawler split is stable. Crawler names and their separation are provider decisions and can change without notice.
- Whether pay-per-crawl produces meaningful revenue. The mechanism exists. No realised rates or volumes are published.
- That news publishers are right to block. We explain why the incentive is strongest there. Whether it is the correct commercial call is untested.
Sources for this feature
- AI crawler blocking prevalence, July 2026. technologychecker.io, ustechautomations.com A named study, reported by someone else - study via secondary coverage
- Crawler classes, robots.txt guidance and CDN-level controls, 2026. digitalapplied.com, witscode.com From a company that sells into this market - vendor and consultancy guidance
- Features 21 and 22 of this edition. Another feature in this edition