Measured 17 Oct 2026
How much of the web turns AI crawlers away.
Every morning we read the robots.txt of a slice of the most visited sites and count which AI crawlers each one shuts out. A site that does keeps its pages out of the open crawls AI models are trained on, so a licence is the only way a developer can use them.
Crawler by crawler
- GPTBotWho runs itOpenAI's training crawlerSites that shut it out
- ClaudeBotWho runs itAnthropic's crawlerSites that shut it out
- CCBotWho runs itCommon Crawl, the open web archive most models start fromSites that shut it out
- Google-ExtendedWho runs itGoogle's switch for training GeminiSites that shut it out
- Applebot-ExtendedWho runs itApple's switch for training its modelsSites that shut it out
- meta-externalagentWho runs itMeta's crawlerSites that shut it out
- BytespiderWho runs itByteDance's crawlerSites that shut it out
- PerplexityBotWho runs itPerplexity's crawlerSites that shut it out
How it is measured
The list is the Tranco ranking of the most visited sites (the copy of 15 Oct 2026): the top 300,000, keeping those whose address ends in .com, .net, .org, .io, .co, .ai, .app, .dev, .info, .us or .uk. Each morning reads 4,000 of them, one slice of the list, so a full pass takes 46 mornings; the figures above add up the newest pass so far. A site counts as shutting a crawler out when its robots.txt names that crawler and disallows the whole site. A site with no robots.txt, or one that did not answer, is not counted.
Each morning's sweep is its own record, and each share above links to the sweeps it adds up. No site is named on this page.
What it means for a site that does
Turning crawlers away protects an archive; it does not price it. The largest publishers have been paid for theirs (every disclosed licence is on the deal ledger), while smaller archives rarely get a call because nobody puts them in front of the buyers. What text sells for, per word and per work, is on the page for text and archives.