The State of AI Traffic: What Publishers Are Seeing in Crawls, Clicks, and Costs

By Raluca Mitchell.

First in a series from the Prebid LLM Taskforce.

For about twenty years, publishers had a simple bargain with crawlers: a crawler scraped your page, and a search engine sent you traffic. Nobody signed anything, because both sides wanted the same outcome. That bargain is now broken. The crawling has continued; the traffic, and the revenue that came with it, have not.

At Duplex Technologies we run five properties, Quartz among them. Seven days of Cloudflare edge data across all of them tells the same story publishers everywhere are reporting: the content is still valuable, it is monetized somewhere else, and we pay the bill for the machines that collect it.

The traffic is leaving through two doors

The first is readers who never reach a search box because they ask an assistant instead. The Reuters Institute’s Digital News Report 2026 puts weekly AI chatbot use for news at 10% globally, up from 7% in a year, and at 17% in the youngest cohort. Tomorrow’s audience is moving fastest.

The second is readers who do search and are handed an AI Overview. In the first four months of 2026, 68% of US Google searches ended without a click, up from 60% in 2024. Where an AI Overview appears, Similarweb puts the zero-click rate at 80% to 83%, and news site traffic down 26% in the twelve months after Overviews launched.

The first door is underestimated because it leaves no record. A lost search click is a decline you can chart; a reader who asks a chat interface leaves no referrer, no session, no log line. The only trace is the crawl that fetched your page to answer the question. Neither door is a ranking problem. You cannot optimize your way back to a click that was never going to happen.

What comes back does not pay

Pass-through is low. Only 4% of readers say they always or often follow the sources behind an AI answer, against 19% for search and 17% for social. Volume is insignificant. AI referrals are under 1% of publisher pageviews by Similarweb’s measure. Ahmed Karim at Longitude Ads puts them at 0.016%, about one in 6,000, across nearly three billion pageviews.

The “premium” is hard to find. Adobe has AI-referred visitors converting 42% better with 48% more time on site. Longitude’s publisher data does not bear it out: LLM-referred traffic monetizes at roughly the same RPM as everything else. Even a premium on one pageview in six thousand would not change the narrative. And most publishers cannot measure this traffic at all: assistant visits carry no recognizable referrer, so analytics files them under organic search. Duplex only separated them by hand-building a custom channel group on the referrer hostnames assistants do send.

Some of it is not human. Nepomuk Seiler at highfivve found a single crawler that slipped past Cloudflare, executed their tracking JavaScript, never consented to advertising, and for a period accounted for around half their pageviews. Duplex drops anomaly days from reporting by hand for the same reason.

What the automated traffic costs

Five costs, none on a rate card.

Serving. Humans read hot, cached pages; scrapers sweep the cold long tail straight to origin. Fastly measured fewer than 9% of human requests requiring an origin fetch, against more than 51% of automated AI requests.

Outages. In July, one Duplex property degraded for about fifty minutes after a roughly tenfold surge in unverified scraper traffic on spoofed, rotating user agents flooded the cache and forced live rendering. Nobody was attacking us; something wanted our content quickly. highfivve saw an AI agent hammer their internal search, an uncacheable operation nobody designs for at machine speed.

Defense, which scales with the value of your content. qz.com logged 2.5 million classified threats in seven days, about 3.25% of requests; our two syndication zones logged 84 and 110. Blocking is not free either: cut too broadly and brand safety crawlers go out with the scrapers, taking their bids with them, and publishers that blocked LLM crawlers via robots.txt saw roughly a 7% decline in weekly visits within six weeks.

Revenue already earned, then lost. Invalid traffic scrubbing has removed up to 80% of a day’s impressions on our properties when the mix looked automated. Kean Graham at MonetizeMore, whose Traffic Cop has analyzed nearly a trillion impressions, sees 8.5% invalid and a further 14.8% “suspicious” traffic after Cloudflare-class filtering, both trending up. The impact arrives abruptly: Google’s automated detection lets much through silently, and when a manual review finally triggers, it claws back ninety days. Terminations are effectively unappealable, and an AdSense termination takes AdX with it.

Trust in your own numbers. As Longitude Ads put it, bot traffic is no longer a fraud problem but a measurement problem. When automated traffic swings, viewability, CTR and RPM swing with it, and every investigation now starts with “what changed in the traffic mix.”

The content still earns. It just does not earn for you.

Alphabet’s Q1 2026 shows both halves of the story. Google Search revenue was $60.4 billion, up 19%; Network advertising, the line that pays third-party publishers, was $7.0 billion, down 4%. Same queries, same content, opposite directions.

The advertising is following the answers. OpenAI opened self-serve advertising inside ChatGPT in July 2026, with projected ad revenue of $2.5 billion this year and $11 billion in 2027. Asked whether publishers would share in that revenue, OpenAI’s Varun Shetty answered “Not at this point.”

Licensing pays a fixed fee for training and retrieval rights, not a share of what the answer earns, so the gap widens by design. Perplexity’s Comet Plus is the exception, a $42.5 million pool behind an 80/20 split with more than 2,400 publishers. The structure is right; beside a $2.5 billion ad projection, the gap is one of scale.

The exchange rate keeps moving one way: TollBit puts the scrape-to-referral ratio at 227 to 1 in Q2 2026, up from 150 to 1 in Q1. And what flows back is concentrated: across 31 million AI citations, Goodie found the top five news publishers holding 66% of news citations, Forbes alone 33%. Most publishers, ourselves included, are outside both the citations and the licensing deals.

We will explore the actions publishers can take and the evolving ecosystem in the coming articles.

Sources

Duplex Technologies, Cloudflare edge analytics via Datadog. Production degradation incident report and internal latency alerting.

Nepomuk Seiler, highfivve.com

Ahmed Karim, Longitude Ads

Kean Graham, MonetizeMore

Public research