Records / AI crawler census sweeps
AI crawler census, 2026-10-18: the robots.txt of 4,000 of the most visited sites (part 45 of 46 of the Tranco list of 2026-10-15)
260 of 2,141 sites that answered shut at least one AI crawler out
Dated 18 Oct 2026Source checked 18 Oct 2026Basis: observed
Where it was read
Also read from the same source
- AI crawler census, 2026-10-15: the robots.txt of 4,000 of the most visited sites (part 42 of 46 of the Tranco list of 2026-10-15)
- AI crawler census, 2026-10-16: the robots.txt of 4,000 of the most visited sites (part 43 of 46 of the Tranco list of 2026-10-15)
- AI crawler census, 2026-10-17: the robots.txt of 4,000 of the most visited sites (part 44 of 46 of the Tranco list of 2026-10-15)
What it is about
Figures it counts in
- 300,000Most visited sites the census reads
- 4,000Robots.txt files the census reads each morning
- 11.6%Of them shut at least one AI crawler out of the whole site
- 8.5%Shut out two or more, one of them a training crawler
- 7.6%Of sites shut out GPTBot (OpenAI's training crawler)
- 6.3%Of sites shut out ClaudeBot (Anthropic's crawler)
Cited in this guide