Cloudflare just launched Pay Per Crawl, a new way for content creators to charge AI crawlers for accessing their content.
The launch breaks the binary of either letting AI freely scrape everything or locking up content behind walls. Instead, creators can now set prices for access using an old HTTP code—402 Payment Required.
Pay Per Crawl is in private beta. It uses HTTP status codes to allow or block crawlers, or require payment at a set flat rate across a domain. Publishers get three choices: allow free access, charge a fee, or block entirely. 
Cloudflare handles billing and tech infrastructure. Crawlers prove their identity via Web Bot Auth protocols using cryptographic signatures to avoid spoofing.
There are two payment flows:
- Reactive: Crawler tries to access content, receives 402 with price, then retries with price acceptance header.
- Proactive: Crawler includes max price upfront; if acceptable, content is served with HTTP 200 and charged.
Billing collects all charges, bills the crawler, and pays content owners accordingly.
Pay Per Crawl taps into the emerging future where AI agents could negotiate content access automatically, setting budgets for access to research, news, or specialty info.
Cloudflare invites interested crawlers and publishers to sign up for the private beta at cloudflare.com/paypercrawl-signup.
This is a key step toward programmatic, scalable content monetization for AI-driven web crawling.
