Figures / Holders
11.4%
Of them shut at least one AI crawler out of the whole site
696 of the 6,094 robots.txt files read in the current pass name at least one AI crawler and disallow it from the whole site ("Disallow: /" in a group that names it).
Updated 17 Oct 2026, from its newest record.
The 3 records it is computed from, newest first
- As stated219 of 1,898 sites that answered shut at least one AI crawler outDated17 Oct 2026SourceTranco list of the most visited sites…
- As stated244 of 2,089 sites that answered shut at least one AI crawler outDated16 Oct 2026SourceTranco list of the most visited sites…
- As stated233 of 2,107 sites that answered shut at least one AI crawler outDated15 Oct 2026SourceTranco list of the most visited sites…