Cloudflare launched a new AI content marketplace called Pay per Crawl on Tuesday. The private beta lets website owners charge AI companies micropayments for every AI crawl of their site. Publishers can also allow free crawling or outright block AI bots.
Cloudflare serves 20% of the web and has spent the last year rolling out tools to monitor and block AI crawlers. CEO Matthew Prince told TechCrunch earlier this year these tools would set up a marketplace where publishers control and monetize AI access.
Pay per Crawl gives publishers rate-setting power over AI scrapers. It tracks the purpose behind the crawl—whether for AI training data, search results, or other uses. Cloudflare acts as the middleman, charging AI firms and paying website owners accordingly. 
The launch comes as publishers wrestle with declining Google Search traffic and rising AI chatbot use. Some news outlets have sued over AI using their content without consent, while others have licensing deals with AI companies. But those deals mostly benefit large publishers. Cloudflare aims for a wider, more flexible system.
New Cloudflare-hosted sites will now block AI crawlers by default unless permissions are granted. Major publishers including Conde Nast, TIME, The Associated Press, and The Atlantic have joined the initiative.
Cloudflare’s own data shows AI bots scrape sites vastly more than Google’s crawler—for every Google referral, OpenAI’s crawler scrapes 17,000 times, Anthropic 73,000 times. AI agents that crawl websites to answer user queries directly threaten traditional traffic and ad revenue models.
Cloudflare imagines Pay per Crawl powering an “agentic” future where AI agents use paid budgets to gather top content programmatically.
Cloudflare spokesperson Ripley Park confirmed no cryptocurrencies or stablecoins are involved at launch.
“What if an agentic paywall could operate at the network edge, entirely programmatically? Imagine asking your favorite deep research program to help you synthesize the latest cancer research or a legal brief, or just help you find the best restaurant in Soho — and then giving that agent a budget to spend to acquire the best and most relevant content,” Cloudflare said in a blog post.
Both AI companies and publishers must have Cloudflare accounts to join. Each side sets their price per crawl and Cloudflare handles payments and payouts.
The marketplace’s future hinges on publishers and AI firms agreeing on terms. Convincing AI companies to pay is a challenge—they currently scrape content for free. But Cloudflare’s scale and new default-blocking approach put it in a rare spot to make this work.
