AI crawler census sweeps: 3

Each morning's reading of the robots.txt of a slice of the most visited sites: how many turn AI crawlers away, crawler by crawler.

AI crawler census sweeps
Record	As stated	Dated	Source
AI crawler census, 2026-10-17: the robots.txt of 4,000 of the most visited sites (part 44 of 46 of the Tranco list of…	219 of 1,898 sites that answered shut at least one AI crawler out	17 Oct 2026	Tranco list of the most visited sites (2026-10-15), and…
AI crawler census, 2026-10-16: the robots.txt of 4,000 of the most visited sites (part 43 of 46 of the Tranco list of…	244 of 2,089 sites that answered shut at least one AI crawler out	16 Oct 2026	Tranco list of the most visited sites (2026-10-15), and…
AI crawler census, 2026-10-15: the robots.txt of 4,000 of the most visited sites (part 42 of 46 of the Tranco list of…	233 of 2,107 sites that answered shut at least one AI crawler out	15 Oct 2026	Tranco list of the most visited sites (2026-10-15), and…