How AI developers acquire training data: six channels and the shift away from the open web

Abstract

Developers of large AI models obtain training data through six channels: collection from the open web without a licence, licences from publishers and platforms, commissioned human and expert work, the purchase or licence of organisations' internal records, purpose-built environments and synthetic data, and data from their own users and devices. This paper describes each channel from a public-record compilation read on 18 October 2026: 122 quantified supply facts, 111 disclosed licensing deals, 83 lawsuits, rulings and regulatory actions, 227 financial records of data vendors, 111 funding events, 110 brokerage records and 1,022 price observations, with a robots.txt census of 6,094 websites. The open web remains the largest source by volume, but access is narrowing: 11.4 percent of the census sites shut at least one AI crawler out of the whole site, and an earlier audit found restrictions on 28 percent or more of the most actively maintained sources of a standard web corpus. Of the 83 legal records, 70 concern content taken without a licence and 12 concern developers' use of their own users' data; none concerns the channels in which developers pay for data. Money is moving to those paid channels: disclosed licences, data vendors whose newest reported revenues or run rates exceed one billion dollars, and, since 2026, published programmes that buy organisations' records. We set out the scale, prices and legal exposure of each channel (not legal advice), the limits of the record, and what the shift changes for organisations that hold data.

Keywords: artificial intelligence, training data, data governance, web crawling, robots.txt, data licensing, platform labour, privacy, copyright, synthetic data