Which websites block AI crawlers: a census of robots.txt rules on 6,094 lower-ranked sites, October 2026

Abstract

Website operators who object to the use of their pages for training AI models usually say so in robots.txt, the plain-text file that tells automated clients what they may fetch. This paper reports a census of those files. A reader takes the Tranco ranking of popular sites, keeps registrable domains with eleven address endings within the top 300,000 (181,046 sites), and reads one slice of 4,000 sites each morning in rank order, so that a full pass takes 46 mornings. The current pass began on 15 October 2026 at slice 42, set by date arithmetic, and by 17 October had read slices 42 to 44: Tranco ranks 270,775 to 291,236, well below the most visited sites. Of 12,000 requests, 6,094 returned a usable robots.txt. Of these, 696 (11.4%) disallow at least one of 15 named AI user-agent tokens from the whole site, and 514 (8.4%) disallow two or more, one of them a token used for training. GPTBot is refused most often (457 sites, 7.5%), followed by CCBot (7.2%) and Bytespider (7.0%). Sites refuse training tokens more often than the search and user-request tokens of the same operator: GPTBot is refused 3.7 times as often as OAI-SearchBot. Refusing sites name 6.4 tokens on average, including names that the operator's current documentation does not list, which suggests maintained or copied block lists. We set out the sampling frame, fetch protocol and parser, and the limits of a stated-refusal measure taken on three slices of the ranking.

Keywords: robots.txt, Robots Exclusion Protocol, web crawling, AI crawlers, web measurement, Tranco, training data, opt-out signals, RFC 9309