According to the Electronic Frontier Foundation, the open internet faces a threat from inside the IETF—the standards body that shapes how the web works. Publishers and tech companies are pushing new standards to block automated access to their sites, citing concerns about AI training and lost revenue. Their worries are understandable. AI crawlers can strain infrastructure and disrupt business models built on advertising. But the proposed solutions risk closing off the entire internet. Two IETF working groups are advancing standards that would let websites cryptographically authenticate bots and deny access to whoever they choose—not just malicious actors, but researchers, competitors, and startups without deep pockets. The stakes are enormous. Journalists, archivists, accessibility researchers, and watchdog organizations rely on web scraping for investigations, preservation, and fact-checking. If these standards pass, automated tools would require corporate permission slips. The EFF argues this would effectively commercialize the internet, turning what's supposed to be public infrastructure into a series of paywalled fiefdoms.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton