Cloudflare argues that publishers increasingly want finer control over how their content is used as AI-powered search and answer engines replace visits to original websites. Rather than treating search discovery and AI usage as a single permission, the company is separating those activities into independent choices.
The company is also expanding its experimental publisher marketplace beyond Pay Per Crawl. Its new Pay Per Use model is designed to compensate publishers when their content creates value inside AI systems, not only when it is fetched by a crawler. Under the model, publishers can receive payment when their content appears in AI-generated results or when an AI agent accesses premium information for a specific task.
Ceramic.ai and You.com are the first companies participating in the updated payment model. Cloudflare said publishers who opt in can be paid when their content appears in Ceramic.ai’s AI search results or when You.com retrieves premium pages, while allowing other AI companies to implement arrangements tailored to their own products.
Cloudflare is pairing those controls with new visibility tools. A planned Attribution Business Insights dashboard will show how AI bots access content, where content is cited, and how much referral traffic individual AI platforms generate. The company said the dashboard is intended to give publishers measurable data around Answer Engine Optimization, allowing them to evaluate how content performs across AI-powered discovery systems.
In discussing the initiative, Stephanie Cohen, Chief Strategy Officer at Cloudflare, said, “The vast majority of our customers want AI to engage with their content. However, for those who rely on advertising and subscriptions, the challenge is distinct: they want to remain discoverable without being forced to give their work away for free. As an infrastructure provider, we can help bridge that gap. That’s why we are focused on delivering the tools necessary to support all types of site owners as the agentic Internet continues to evolve.”
Cloudflare said the policy change reflects broader shifts in internet traffic. According to the company, bots now account for the majority of activity on the internet, a milestone it previously expected to reach later. Co-founder and CEO Matthew Prince said the growing volume of automated traffic requires clearer disclosure from AI companies about how their crawlers use publisher content.
“Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” Prince said.
The company also said more than half of AI crawler traffic currently consists of repeatedly retrieving pages that have not changed. It argues that allowing publishers to signal content freshness could reduce unnecessary bandwidth costs while lowering computing expenses for AI companies.
The default settings primarily affect operators that rely on blended crawlers. Cloudflare noted that AI companies using separate, purpose-specific user agents can continue accessing content if they clearly declare each crawler’s function. The changes are expected to require additional engineering work for companies that currently use a single crawler for search indexing, AI retrieval, and model training.
The new defaults do not apply universally across Cloudflare’s customer base. Sites on paid Cloudflare plans that have already customized crawler settings will not automatically receive the updated configuration, and the controls only affect websites using Cloudflare’s infrastructure.
Cloudflare’s latest changes highlight a broader effort to give publishers greater control over AI access while introducing commercial mechanisms for content usage. By separating search indexing from AI training and retrieval, and by shifting its marketplace from crawl-based payments to value-based compensation, the company is positioning content permissions, attribution, and monetization as core components of what it calls the “Agentic Internet.”
This analysis is based on reporting from Forbes.
Image courtesy of Victory Digital.
This article was generated with AI assistance and reviewed for accuracy and quality.
