Cloudflare is blocking AI web crawlers by default for new customers. The move stops unauthorized AI scrapers from accessing content without permission or payment.
New domain owners will be asked if they want to allow AI scrapers. Some big publishers can set a “Pay Per Crawl” fee for AI companies to access their content. AI firms see pricing, decide to pay or walk away. This is currently limited to “a group of some of the leading publishers and content creators.”
Cloudflare has fought AI crawlers since 2023, first letting sites block those that respect robots.txt rules. Then they allowed blocking “all” AI bots, even if they ignore robots.txt. Now, blocking is the default for new customers. The company identifies scrapers using a list of known AI bots. 
In March, Cloudflare introduced an “AI Labyrinth” feature to trap unauthorized web crawlers and stop scraping.
Major outlets like The Associated Press, The Atlantic, Fortune, Stack Overflow, and Quora are on board. Cloudflare CEO Matthew Prince said last week at Axios Live that people trust AI more, so they read less original content online.
Cloudflare is also working with AI companies to verify their crawlers and disclose what they use content for — training, search, or inference. Website owners get to approve the crawlers.
Matthew Prince stated:
Original content is what makes the Internet one of the greatest inventions in the last century, and we have to come together to protect it.
AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate.
