News Behind the News: Spillover Effects of AI Crawler Blocking Among News Publishers
Using 44 months of archived robots.txt files for major news publishers in the United States, Europe and the rest of the world, we show that the arrival of OpenAI's GPTBot in August 2023 triggered a sharp, front-loaded wave of AI-crawler blocking, strongest among paywalled outlets, and that the blocking spilled over onto crawlers with no AI function at all: archives, research tools, accessibility services and news monitors.