Cloudflare is blocking AI crawlers by default on new domains starting Tuesday. The move lets website owners control or charge AI bots scraping their content.
Cloudflare, a major content delivery network handling about 16% of global internet traffic, now asks every new web domain if they allow AI crawlers. Publishers can block bots outright or use a new "pay per crawl" system to monetize AI data access.
This targets AI bots that scrape massive text and images to train models like OpenAI’s and Google’s. Cloudflare says these crawlers disrupt the old web model by replacing direct visits with AI-generated answers, cutting traffic and ad revenue for creators. 
Cloudflare CEO Matthew Prince said this move is about "putting the power back in the hands of creators" and "safeguarding the future of a free and vibrant Internet."
"AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate," Matthew Prince stated.
"This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone," he added.
OpenAI declined to participate, arguing Cloudflare adds a middleman. The company insists its crawlers respect site rules via robots.txt. Legal experts warn this could disrupt AI training short term and threaten model viability long term.
"AI crawlers are typically seen as more invasive and selective when it comes to the data they consume. They have been accused of overwhelming websites and significantly impacting user experience," said Matthew Holman, partner at UK law firm Cripps.
"If effective, the development would hinder AI chatbots’ ability to harvest data for training and search purposes. This is likely to lead to a short term impact on AI model training and could, over the long term, affect the viability of models."
Cloudflare first gave publishers blocking tools last September. Now it’s forcing the change by default. AI developers and site owners are bracing for the impact.
