Figures / Holders
8.4%
Shut out two or more, one of them a training crawler
514 of the 6,094 robots.txt files read in the current pass shut two or more AI crawlers out of the whole site, at least one of them a crawler that gathers training data (GPTBot, ClaudeBot, CCBot, Google-Extended and the like): the holder watch's test for a holder that guards its data.
Updated 17 Oct 2026, from its newest record.
The 3 records it is computed from, newest first
- As stated219 of 1,898 sites that answered shut at least one AI crawler outDated17 Oct 2026SourceTranco list of the most visited sites…
- As stated244 of 2,089 sites that answered shut at least one AI crawler outDated16 Oct 2026SourceTranco list of the most visited sites…
- As stated233 of 2,107 sites that answered shut at least one AI crawler outDated15 Oct 2026SourceTranco list of the most visited sites…