startups: The web is now mostly bots. Cloudflare is rebuilding its

In a significant shift marking a new era for the internet, automated bots now account for over half of all web traffic, surpassing human users. This pivotal milestone, with bots generating approximately 57% of all web requests, has prompted cybersecurity giant Cloudflare to launch an entirely new defense mechanism: Cloudflare Precursor. Unveiled on Monday, July 13, 2026, Precursor represents a fundamental overhaul in how websites distinguish between human visitors and sophisticated automated agents, preparing the web for an increasingly AI-driven future.

Rethinking Bot Detection with Cloudflare Precursor

Traditional web security measures, like CAPTCHAs, typically rely on a single, momentary check to verify a user's humanity. However, modern bots have become adept at bypassing these superficial gatekeepers. Cloudflare Precursor moves beyond this approach, embedding itself within the browser to monitor an entire user session.

Instead of a one-time ID check, Precursor observes a visitor's comprehensive behavior, including subtle cues like mouse movements, scrolling patterns, typing cadence, clipboard usage, and how long a page remains in view. This holistic, continuous monitoring makes it significantly harder for malicious bots to mimic genuine human interaction, transforming what was once an easy task for sophisticated automation into a complex engineering challenge.

Dane Knecht, Cloudflare’s chief technology officer, emphasized this shift: "Traditional security checks look at a single moment in time, but modern bots have gotten smart enough to fake their way through the front door." He noted that the crucial period between a user logging in and making a purchase has historically been a "black box," a gap Precursor aims to close. The tool is designed with privacy at its core, logging only behavioral patterns such as typing rhythm and cadence, never the actual keystrokes or content. Activation is streamlined, requiring just one click with no code modifications needed for websites.

Navigating the New Landscape of AI Agents

Precursor is part of a broader strategic re-evaluation by Cloudflare to manage the diverse and growing array of automated traffic. Not all bots are malicious; many perform essential functions. To address this, Cloudflare is now categorizing AI traffic into three distinct types: Search bots, which index content for later queries; Agent bots, designed to act in real time on behalf of a person; and Training bots, which absorb web content to train AI models.

Beginning September 15, new websites utilizing Cloudflare's services will automatically block Training and Agent bots on pages that display advertisements, while still allowing Search bots to operate freely. This policy change is driven by economic considerations, as Search bots typically direct traffic back to sites, unlike Training and Agent bots, which often do not. This initiative builds on Cloudflare’s previous stance that AI crawlers should compensate publishers for content usage or face blocking.

To further empower publishers and maintain control over their content, Cloudflare is introducing new options for sites to specify how bots may interact with their data: whether to store nothing, index and link back, or summarize and reproduce content. A new comprehensive database, dubbed "BotBase," will catalog every known web crawler, enhancing transparency and accountability.

Building Trust and Reshaping Web Infrastructure

Crucially, Cloudflare is pushing for greater accountability among bot operators. The company advocates for these operators to openly declare their identity using existing web headers, allowing websites to grant or deny access to specific entities like "OpenAI," with this choice persisting across complex network layers. Cloudflare suggests that the risk of losing trusted status across the more than 20 percent of web domains it protects will serve as a powerful deterrent against misuse.

This move aligns with broader industry discussions, such as Estonia's proposal to assign unique ID numbers to AI agents, and efforts to enable publishers to opt out of AI training without disappearing from search results. The implications become increasingly significant as AI agents evolve to perform tasks like shopping and making payments on behalf of users.

The foundational infrastructure of the web itself is also undergoing quiet but profound changes. On the same day as Precursor's launch, the Internet Engineering Task Force (IETF) published a new HTTP method called QUERY. Co-authored by engineers from Cloudflare and Akamai, this standard introduces a dedicated, safe, and cacheable verb for complex search requests. This eliminates the need for such searches to masquerade as data-altering POST requests, streamlining web communication built for a machine-dominated environment. These intertwined developments underscore a fundamental transformation of the internet's core machinery, shifting its design from human clicks to the intricate interactions of an "agentic" web.

FAQ

Q: What is Cloudflare Precursor and how does it differ from existing bot detection methods?

A: Cloudflare Precursor is a new tool launched by Cloudflare that detects sophisticated bots by monitoring a visitor's entire session within the browser, rather than relying on single-point checks like CAPTCHAs. It analyzes behavioral patterns such as mouse movement, scrolling rhythm, and typing cadence, making it far more difficult for modern bots to mimic human activity.

Q: Why is Cloudflare changing its approach to bot detection now?

A: Cloudflare is adapting because bots now generate approximately 57% of all web traffic, surpassing human users. The existing web infrastructure and traditional security methods were primarily designed for human interactions and are no longer sufficient to secure against the advanced capabilities of modern AI agents and other automated traffic.

Q: How does Cloudflare plan to manage different types of AI bots?

A: Cloudflare is categorizing AI traffic into Search bots, Agent bots, and Training bots. Starting September 15, new sites using Cloudflare will block Training and Agent bots by default on ad-supported pages, while allowing Search bots. They are also introducing tools for publishers to control how bots can reuse their content and creating a "BotBase" database to identify known crawlers.