DIGITAL LIFE

Cloudflare blocks AI crawlers, here's why

Cloudflare blocks AI crawlers by default on all websites that use its network, in a change that redefines the rules for accessing online content.

Now, website owners can explicitly authorize the collection of data by artificial intelligence systems, instead of having to manually prohibit it.

This decision, announced yesterday (July 1), responds to the massive consumption of data by AI crawlers, which access content to train models without generating return traffic or revenue for creators.

The goal is to rebalance the relationship between content producers and technology companies, returning control and the possibility of financial compensation to creators.

New rules for controlling access to websites...Cloudflare has introduced an active permission model (opt-in) for AI crawlers.

The main features available to website owners include:

Blocking by default all known AI crawlers on new domains.

Ability to view and manage, from the control panel, which bots are trying to access the content.

Create specific rules to allow or deny access, based on the crawler’s stated purpose (model training, inference, research).

Distinguish between legitimate bots and “shadow scrapers” that try to bypass restrictions.

Define permissions for each phase of the AI ​​lifecycle (training, tuning, inference).

Ability to define fees for access, through the “Pay Per Crawl” model.

A new AI content monetization model...Cloudflare has launched the experimental “Pay Per Crawl” marketplace, allowing publishers to set a price for each authorized access of an AI crawler.

Options for website owners include:

Charge micropayments for each authorized crawler access.

Grant free access to certain bots.

Maintain full blocking of AI crawlers.

The process is simple: each time a crawler requests access, it must express an intent to pay, Cloudflare processes the payment and distributes the revenue to the publisher. Site owners can set fixed prices per request and differentiate access between crawlers.

Broad support from media and technology groups...Cloudflare’s initiative has received support from dozens of leading media, technology and digital platform groups, including:

ADWEEK

The Arena Group

The Associated Press

The Atlantic

Atlas Obscura

BuzzFeed, Inc.

Condé Nast

Digital Content Next

Dotdash Meredith

Drupal & Acquia

Fortune

Gannett Media | USA Today Network

Independent Media

News/Media Alliance

O’Reilly Media

Pinterest

PMC

Quora

Reddit

Sky News Group

Snopes.com

Stack Overflow

TIME

Universal Music Group

Webflow

Ziff Davis

These groups advocate for a more sustainable digital ecosystem, where access to AI content depends on permission and fair compensation.

Context and impact for the industry...Cloudflare CEO Matthew Prince points out that AI crawlers have been extracting content without limits, putting the sustainability of creators at risk. The new model aims to return power to content producers, promoting responsible innovation and protecting the viability of the open internet.

Publishers now have:

-Full control over who accesses their content and for what purposes.

-Transparency about the crawlers' intentions (training, research, inference).

-The ability to monetize access to content, creating a new revenue stream.

Cloudflare hopes that this approach will encourage direct negotiations and licensing agreements, fostering an environment of greater transparency and sustainability for all stakeholders in the digital ecosystem.

Conclusion...Cloudflare’s decision to block AI crawlers by default represents a fundamental shift in the internet economy. By shifting decision-making power to content creators and facilitating financial compensation mechanisms, the company is setting a new precedent for how data is accessed and valued in the age of artificial intelligence. This approach forces a much-needed dialogue about the sustainability of the digital ecosystem and the value of original content.

For more information, read Cloudflare’s full press release...https://www.cloudflare.com/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/

mundophone