Perplexity pushes back on Cloudflare’s claim of stealth crawling. The AI assistant maker says it’s not a traditional crawler and doesn’t break robots.txt rules.
The issue started when Cloudflare flagged Perplexity’s traffic, calling it stealthy and disrespectful of web scraping rules. Perplexity fired back, saying its AI assistants fetch pages on demand — only when users ask questions. No pre-indexing or wide-scale crawling happens here.
Perplexity points out that fetching on user request is different from how regular crawlers index content blindly. They even compared their fetching to Google’s user-triggered page fetches, which also circumvent robots.txt in limited ways like reading text aloud or site verification. 
The company insists it doesn’t keep or train its models on fetched data. The AI acts like a direct extension of a user’s request, not an autonomous bot scooping content endlessly.
Perplexity also slammed Cloudflare’s bot management. It says Cloudflare can’t tell responsible AI assistants apart from bad actors and ends up overblocking legitimate traffic. They argue Cloudflare misunderstood their tech and blocked them unfairly.
Perplexity published a detailed response on their site, pushing back hard.
Perplexity stated:
Agents or Bots? Making Sense of AI on the Open Web
https://www.perplexity.ai/hub/blog/agents-or-bots-making-sense-of-ai-on-the-open-web
