Cloudflare announced plans on Monday to create a marketplace where website owners can sell access to their content for AI model training. This move is part of a broader initiative to give publishers more control over how AI bots interact with their websites.
Matthew Prince, Cloudflare's CEO, explained the rationale behind this decision: "We think that sites of any size should be fairly compensated for the use of their content." The marketplace, set to launch within the next year, will allow content creators to set prices for AI companies to access and scrape their sites.
This announcement comes alongside the immediate release of AI Audit, a free tool for all Cloudflare customers. AI Audit provides website owners with detailed analytics on AI bot activity, including information on which companies are scraping their content, how frequently, and which sections of their sites are most popular among AI crawlers.
The new tool categorizes AI-related bots into three types: AI Data Scrapers, AI Search Crawlers, and Archivers. This distinction is crucial, as each type can impact a website differently. For instance, AI search engines might drive traffic to a site, while data scrapers primarily collect information for model training without necessarily attributing the source.
Cloudflare has also introduced a one-click option to block all AI crawlers, giving site owners a "pause" to assess their strategy regarding AI content usage. For more granular control, administrators can now create custom rules to allow or block specific AI providers or bot types.
Prince highlighted the current imbalance in the AI ecosystem, where many websites are scraped without compensation or even awareness. "If you don't compensate creators one way or another, then they stop creating, and that's the bit which has to get solved," he said in an interview with TechCrunch.
The new tools aim to address growing concerns about the impact of AI on online content creation. Some website owners have reported that excessive AI scraping has led to server issues resembling DDoS attacks. Cloudflare's solution allows site owners to block specific AI bots while permitting others, giving them granular control over their content's use.
For publishers who have already negotiated deals with AI companies, Cloudflare now offers tools to audit these arrangements. Site owners can generate reports to verify if AI providers are adhering to agreed-upon terms regarding scanning frequency and content access.
While the marketplace concept is still in development, it could potentially democratize content licensing for AI training. Currently, only large publishers have been able to negotiate deals with major AI companies like OpenAI. Cloudflare's platform could extend this opportunity to smaller content creators, handling the pricing, access control, and value capture for content scanning.
However, the big question is how AI model developers will respond to potentially paying for content they currently access for free. Prince believes this approach is necessary for the long-term sustainability of the AI ecosystem and the broader internet.
Cloudflare customers can begin using the AI Audit tools immediately through their dashboards and are invited to join the waitlist for the marketplace’s beta version at Cloudflare AI Value Tool Waitlist. The company will further showcase these advancements during its Builder Day Live Stream on September 26.