Cloudflare is launching a new way for publishers to make AI companies pay every time their bots scrape a website. The feature is called Pay per Crawl. It’s designed to charge fees to AI firms that collect data by crawling web content.
Cloudflare powers about 20% of the web, so this move could reshape how online content is accessed and monetized. The company has spent the last year building bot-blocking tools, and this feels like the next step.
The new experiment puts Cloudflare at the center of a pay-for-content model, aiming to control how AI bots interact with publishers’ sites. It could spark major changes in the AI training data landscape if it sticks. 
This was discussed on TechCrunch’s Equity podcast with hosts Kirsten Korosec and Max Zeff, who debated whether Cloudflare’s Pay per Crawl is a smart move or unrealistic.
Cloudflare’s launch follows rising tensions over AI companies scraping online content without permission. Watch how publishers and AI firms respond.
Equity also covered how ICEBlock, an app for reporting ICE sightings, went viral after backlash to former prosecutor Pam Bondi, and Figma’s S-1 filing pointing to a big IPO ahead with 48% revenue growth.
Equity airs Wednesdays and Fridays. Listen on Apple Podcasts, Overcast, Spotify, or follow on X, and Threads at @EquityPod.
