Grammarly bought email app Superhuman for an undisclosed sum, part of its plan to build an AI productivity empire. With $1 billion in fresh funding, the grammar company wants to put AI agents at the center of your workday.
Good Morning from San Francisco, Cloudflare just declared war on freeloading AI bots. The company that handles 20% of internet
Cloudflare now blocks AI bots by default and lets publishers charge per scrape. With AI companies taking 17,000+ crawls per referral while Google takes just 14, the internet's biggest traffic handler is reshaping data collection rules.
Cloudflare now blocks AI bots by default and lets publishers charge per scrape. With AI companies taking 17,000+ crawls per referral while Google takes just 14, the internet's biggest traffic handler is reshaping data collection rules.
💡 TL;DR - The 30 Seconds Version
🚫 Cloudflare now blocks AI crawlers by default for new websites, affecting 20% of internet traffic and forcing AI companies to get permission before scraping content.
📊 Data shows the scraping imbalance: OpenAI crawls sites 17,000 times per referral while Google crawls just 14 times, leaving publishers without traffic or revenue.
💰 Publishers can now charge AI companies per crawl through Cloudflare's new marketplace, with major outlets like AP, Time, and Fortune already signed up.
🤖 The system uses HTTP 402 "Payment Required" responses and requires AI crawlers to authenticate themselves and state their purpose for accessing content.
⚡ OpenAI refused to participate, calling it an unnecessary middleman, while other major AI companies remain silent on whether they'll pay publishers.
🌍 Success depends on AI companies choosing to pay rather than finding workarounds, potentially reshaping how AI systems access training data across the web.
Cloudflare just flipped the script on AI web scraping. The company that handles 20% of internet traffic now blocks AI crawlers by default and launched a marketplace where publishers can charge bots for each visit.
New websites joining Cloudflare must now grant permission before AI crawlers can access their content. The change affects millions of sites and could reshape how AI companies gather training data. Several major publishers, including The Associated Press, Time, and Fortune, have already signed up.
The move comes as publishers watch their traffic plummet while AI companies scrape their content freely. Traditional search engines like Google send users back to original sources. AI chatbots serve up answers without crediting where they found the information.
The numbers tell the story
Cloudflare's data reveals the stark difference between old and new web scrapers. Google's crawler visits sites 14 times for every user it sends back. OpenAI's crawler? It scrapes 17,000 times per referral. Anthropic's crawler hits an astounding 73,000 scrapes per referral.
"We're changing the rules of the internet," said Matthew Prince, Cloudflare's CEO. "If you're a robot, now you have to go on the toll road."
How the pay-per-crawl system works
Publishers set a flat rate across their entire site. When an AI crawler requests content, it either pays the price or gets blocked with an HTTP 402 "Payment Required" response. Cloudflare handles the billing and distributes earnings to publishers.
The system uses established web protocols. Crawlers must authenticate themselves with digital signatures and specify what they plan to do with the content - whether it's for training, search, or other purposes. Publishers can then decide which crawlers to allow, block, or charge.
Publisher controls get granular
Website owners have three options for each crawler: allow free access, require payment, or block entirely. They can also bypass charges for specific crawlers or negotiate separate deals outside the system.
The technology builds on HTTP response code 402, a rarely used status code originally designed for payment systems. Cloudflare dusted it off to create what it calls "the forgotten piece of the web."
Industry backing grows
Media companies like Condé Nast, The Atlantic, and ADWEEK support the initiative. Stack Overflow and Quora have also joined. "Community platforms that fuel LLMs should be compensated for their contributions," said Stack Overflow CEO Prashanth Chandrasekar.
The system addresses a fundamental shift in how people consume information. Instead of clicking through to original sources, users increasingly get answers directly from AI chatbots. This leaves publishers without the traffic they need to generate ad revenue.
Technical challenges remain
Success depends on AI companies choosing to participate. Some may simply ignore the payment system and continue scraping for free. Cloudflare plans to use its bot detection expertise to stop unauthorized crawlers, but the cat-and-mouse game continues.
The company must also accurately identify AI crawlers, which currently relies on user reports and traffic pattern analysis. Sophisticated crawlers might find ways to mask their identity or purpose.
Future possibilities
Cloudflare envisions a more complex marketplace ahead. Publishers might charge different rates for different content types or implement dynamic pricing based on demand. The system could support AI agents that automatically negotiate content deals across the web.
The company sees potential in an "agentic" future where AI assistants could receive budgets to buy access to premium content. Imagine asking an AI to research cancer treatments or find the best restaurant in a city, then having it pay for access to the most relevant sources.
AI companies push back
Not everyone embraces the change. OpenAI declined to participate in Cloudflare's preview, arguing the system adds an unnecessary middleman. The company stressed its commitment to respecting robots.txt files and publisher preferences.
Other AI companies have remained silent on the announcement. Their participation will determine whether Cloudflare's marketplace succeeds or becomes another failed attempt to monetize web content.
Why this matters:
Q: How much does Cloudflare charge publishers to use this system?
A: Cloudflare acts as the payment processor and takes a cut of each transaction, similar to how credit card companies work. The exact percentage wasn't disclosed, but publishers keep most of the money they charge AI companies.
Q: Which AI companies have actually agreed to pay for content?
A: Cloudflare says it has "AI partners" but won't name them. OpenAI publicly refused to join, calling it an unnecessary middleman. The system's success depends on getting major players like Google, Anthropic, and Microsoft onboard.
Q: Can small websites use this or just big publishers?
A: Any website using Cloudflare can block AI crawlers by default. But the pay-per-crawl marketplace is currently limited to "leading publishers and content creators" in private beta. Smaller sites must wait for wider rollout.
Q: What stops AI companies from just ignoring this system?
A: Cloudflare uses the same bot detection methods it employs against cyberattacks. However, sophisticated AI companies could potentially mask their crawlers or find workarounds. It's an ongoing technical arms race.
Q: How does Cloudflare know which bots are AI crawlers?
A: The system relies on user reports and analysis of traffic patterns. Legitimate AI companies can register their crawlers and identify their purpose. Unregistered bots get blocked if they show suspicious mass-scraping behavior.
Q: Will this make websites load slower for regular users?
A: No. The system only affects automated crawlers, not human visitors. Regular users browsing websites won't notice any difference in loading speeds. The bot detection happens at Cloudflare's network level.
Q: How much money could publishers realistically make from this?
A: Unknown. Publishers set their own prices, but market forces will determine actual rates. If AI companies pay a few cents per thousand crawls, large publishers might earn thousands monthly. Small sites would make much less.
Q: When will this be available to all websites?
A: Blocking AI crawlers by default starts immediately for new Cloudflare customers. The pay-per-crawl marketplace is in private beta with select publishers. Cloudflare hasn't announced a timeline for public availability.
Apple considers ditching its own AI technology for Siri, eyeing Anthropic's Claude or OpenAI's ChatGPT instead. The potential reversal exposes Apple's struggle in the AI race and internal talent exodus.
Meta hired four more OpenAI researchers this week, escalating Zuckerberg's talent war with $100M packages. The exodus follows Meta's disappointing Llama 4 launch as the CEO personally hunts AI stars to close the innovation gap.
Meta poaches three OpenAI researchers with $100 million signing bonuses as Zuckerberg builds a "superintelligence" team. Sam Altman dismisses the blitz, but departures suggest money talks in AI's talent war.
Meta won a major AI copyright case but the judge warned this victory won't last. He ruled the company's use of stolen books was legal while predicting future cases will go the other way—essentially giving authors a roadmap to win.
Get tomorrow's intel today. Join the clever kids' club. Free forever.
