Sources / Press and newsletters

Tranco list of the most visited sites (2026-10-15), and each site's own robots.txt

3 records were read from this source, dated 15 Oct 2026 to 17 Oct 2026.

Open it on tranco-list.eu

Its permanent copy in the Wayback Machine, 9 Oct 2026

Last checked 17 Oct 2026

What was read from it

Figures it feeds
  • 300,000Most visited sites the census reads
  • 4,000Robots.txt files the census reads each morning
  • 11.4%Of them shut at least one AI crawler out of the whole site
  • 8.4%Shut out two or more, one of them a training crawler
  • 7.5%Of sites shut out GPTBot (OpenAI's training crawler)
  • 6.0%Of sites shut out ClaudeBot (Anthropic's crawler)
  • 7.2%Of sites shut out CCBot (Common Crawl, the open web archive most models start from)
  • 5.0%Of sites shut out Google-Extended (Google's switch for training Gemini)
  • 4.4%Of sites shut out Applebot-Extended (Apple's switch for training its models)
  • 5.4%Of sites shut out meta-externalagent (Meta's crawler)
  • 7.0%Of sites shut out Bytespider (ByteDance's crawler)
  • 3.1%Of sites shut out PerplexityBot (Perplexity's crawler)
Tranco list of the most visited sites (2026-10-15), and each site's own robots.txt: what… | AIData.Broker