This week, the internet’s AI drama spotlight turned to a heated spat between Cloudflare, the internet’s neighborhood watch, and Perplexity AI, the rising AI search engine. The accusation? Perplexity was sneaking around the block—scraping websites that explicitly told it “No Visitors Allowed” via robots.txt.
Cloudflare’s “sting operation” was almost cinematic. They built a brand-new website, locked the front door to Perplexity’s known bots, then asked Perplexity about the site’s content. Shockingly, Perplexity still served up answers—this time, by using a generic browser disguise pretending to be Google Chrome on macOS. Cloudflare CEO Matthew Prince went full action-movie mode on X, likening some AI companies to “North Korean hackers” and urging site owners to name, shame, and block.
But here’s where it gets messy: not everyone thinks Perplexity’s move was “breaking and entering.” Supporters argue that if a human can visit a public page, an AI fetching it on behalf of a human shouldn’t be treated differently. One Hacker News commenter put it bluntly: “If I ask for the page, why should it matter whether I use Firefox or an AI assistant?”
Perplexity’s official stance? “It wasn’t us — it was a third-party service.” But in their counterattack blog post, they claimed the bigger issue is that Cloudflare can’t tell the difference between a legit AI helper and a bad bot.
For Cloudflare, the example of OpenAI is the gold standard—they play by the rules, respect robots.txt, and sign their AI’s web requests using Web Bot Auth, a new open standard designed to identify AI agents.
This fight is more than internet gossip — it’s a peek into the future of how AI agents and websites will coexist. Bots now make up over 50% of all web traffic (Imperva’s latest Bad Bot report), and a big chunk of that is AI-powered scraping. Meanwhile, Gartner predicts search volumes could drop 25% by 2026 as people shift to AI agents for tasks like booking travel or shopping.
The dilemma? Websites want traffic (and ad revenue) from humans, but AI agents are poised to become the middlemen. Will sites risk blocking them and losing customers? Or will they adapt to feed the bots that, in turn, feed the humans?
As one X user summed it up:
- “I want Perplexity to fetch any public content for me.”
- “But what if the site owners don’t want to give it away without you visiting?”
Welcome to the messy reality of Agentic Browsing — the battle for the open web has officially entered its next phase.
About the Author
