Figures / Holders
5.4%
Of sites shut out meta-externalagent (Meta's crawler)
327 of the 6,094 robots.txt files read in the current pass disallow meta-externalagent from the whole site.
Updated 17 Oct 2026, from its newest record.
The 3 records it is computed from, newest first
- As stated219 of 1,898 sites that answered shut at least one AI crawler outDated17 Oct 2026SourceTranco list of the most visited sites…
- As stated244 of 2,089 sites that answered shut at least one AI crawler outDated16 Oct 2026SourceTranco list of the most visited sites…
- As stated233 of 2,107 sites that answered shut at least one AI crawler outDated15 Oct 2026SourceTranco list of the most visited sites…