Artificial intelligence companies continue to consume vast amounts of online content while sending relatively little traffic back to the websites that produce it, according to new data from Cloudflare.
The development has bolstered growing concerns over the sustainability of the internet’s long-standing economic model.
The latest figures, covering the week of July 1 to July 7, show that Anthropic remains the largest outlier among major AI developers, with its web crawlers requesting pages roughly 2,800 times for every single referral its services sent back to publishers.
Register for Tekedia Mini-MBA edition 20 (June 8 – Sept 5, 2026).
Register for Tekedia AI in Business Masterclass.
Join Tekedia Capital Syndicate and co-invest in great global startups.
While that marks a substantial improvement from early April, when Cloudflare estimated Anthropic’s crawl-to-referral ratio at around 8,800-to-1, the broader trend suggests the company’s AI systems continue to collect online content at a pace that far exceeds the amount of traffic they return.
Cloudflare’s data also indicates that the improvement may not represent a sustained change.
During the first week of May, Anthropic’s bots reportedly crawled websites approximately 24,700 times for every referral, illustrating how dramatically the ratio can fluctuate over relatively short periods.
Among the major AI companies tracked by Cloudflare, OpenAI ranked second behind Anthropic, followed by AI search startup Perplexity. Microsoft and Google recorded lower crawl-to-referral ratios, while search engine DuckDuckGo stood out as one of the few companies approaching a more balanced exchange.
According to Cloudflare, DuckDuckGo generated roughly one referral for every three webpage crawls, a ratio far closer to the traditional relationship that has historically existed between search engines and publishers.
The data comes as AI-powered search tools increasingly replace conventional web searches.
For decades, website owners generally accepted search engines crawling their content because indexing generated traffic that could be monetized through advertising, subscriptions, or product sales.
The arrangement created a mutually beneficial ecosystem.
Search engines gained access to information, while publishers received visitors who generated revenue.
Generative AI is fundamentally changing that relationship.
Instead of directing users to external websites, AI chatbots and AI-powered search engines increasingly provide complete answers within their own interfaces. As a result, users often obtain the information they need without ever visiting the original source.
That shift has become one of the publishing industry’s biggest concerns as AI adoption accelerates. This is because if AI systems continue extracting content while reducing referral traffic, publishers could face declining advertising revenue and fewer incentives to invest in producing original journalism, research and educational material.
Cloudflare’s crawl-to-referral metric has therefore become an increasingly watched indicator of how AI companies interact with the broader web. Rather than measuring the absolute amount of data collected, the metric compares how frequently AI crawlers request webpages against how often users are sent back to those same sites.
Although the ratio does not capture every aspect of publisher traffic, it offers a proxy for whether AI companies are maintaining the economic exchange that has historically supported the open internet.
Anthropic’s position is particularly notable because the company has frequently emphasized AI safety and responsible development as core elements of its business strategy. The latest figures have therefore renewed debate over what constitutes responsible behavior when AI systems rely heavily on publicly available web content for search, retrieval and other services.
At the same time, Anthropic has recently taken a firm stance against the unauthorized use of its own AI technology. The company has criticized competitors for allegedly using outputs generated by Anthropic’s models to improve their own systems through a practice commonly referred to as model distillation. Anthropic argues that such activities violate its terms of service because they involve using its AI-generated outputs to train competing models.
Many companies in the AI industry now object to competitors using their proprietary AI outputs for commercial purposes while simultaneously relying on vast quantities of online content created by publishers, writers, researchers and other content producers.
Publishers have argued that AI companies should compensate content creators when their material contributes to commercial AI products, particularly as chatbot-generated answers reduce visits to original websites. Anthropic is currently facing a lawsuit for unlawful use of publishers contents.
Anthropic has previously challenged Cloudflare’s analysis.
The company said it could not independently verify Cloudflare’s methodology for calculating crawl-to-referral ratios and argued that recently introduced search features are increasing the number of users directed back to publishers. That suggests that Anthropic believes the reported figures may not fully reflect how its services generate referral traffic.