Cloudflare is rolling out a new system letting millions of websites block AI bots from scraping their content without permission. Big names like Sky News, The Associated Press, and Buzzfeed are now covered.
The company, which hosts roughly 20% of the internet, says the tech is already live on a million sites. The plan: eventually let sites charge AI firms for access.
The move targets AI “crawlers” — automated programs that browse and collect web data for training AI models. Cloudflare argues these bots break an old “agreement” by grabbing content to build AI without sending traffic back to the source, cutting creators out of revenue. 
Condé Nast CEO Roger Lynch called it “a game-changer” in protecting quality journalism and holding AI firms accountable.
But experts warn legal fixes are still needed.
The system activates by default for new Cloudflare users and those who joined a previous crawler-blocking effort. Publishers want AI firms to pay for training on their content, especially after the BBC threatened legal action over the use of its material by US AI startup Perplexity.
Cloudflare’s CEO Matthew Prince said on the issue:
"If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone."
To handle bad actors, Cloudflare first set up a “Labyrinth” trick sending rogue bots into junk-filled pages. Now, it’s moving toward a “Pay Per Crawl” billing system for AI companies.
AI crawler activity has exploded. Cloudflare reported over 50 billion daily AI bot requests in March alone.
Still, critics like Fairly Trained founder Ed Newton-Rex say this patch only works on sites using Cloudflare and won’t fix the broader problem.
"This is really only a sticking plaster when what’s required is major surgery," Newton-Rex told the BBC.
"It will only offer protection for people on websites they control – it’s like having body armour that stops working when you leave your house."
"The only real way to protect people’s content from theft by AI companies is through the law."
Filmmaker Baroness Beeban Kidron praised Cloudflare’s action as leadership in a heated UK debate over AI copyright and creative rights.
"Cloudflare sits at the heart of the digital world and it is exciting to see them take decisive action," she said.
"If we want a vibrant public sphere we need AI companies to contribute to the communities in which they operate, that means paying their fair share of tax, settling with those whose work they have stolen to build their products, and, as Cloudflare has just shown, using tech creatively to ensure equity between digital and human creators on an ongoing basis."
