Cloudflare CEO Matthew Prince announced a significant projection regarding internet traffic patterns during an interview at the SXSW conference in Austin. He stated that artificial intelligence bot traffic will surpass human web traffic by 2027. This prediction highlights the accelerating pace of generative AI adoption across the digital ecosystem and signals a major shift in how the web operates fundamentally.
Prince explained that bots’ web usage has been increasing alongside the growth of generative AI technology because agents visit far more sites to answer queries. If a human were shopping for a digital camera, they might visit five websites, but a bot will often go to 5,000 sites. These are real transactions creating real load that infrastructure providers must manage daily.
Before the current generative AI era, the internet was only about 20% bot traffic, with Google’s web crawler being the largest component. According to Prince, whose infrastructure and security company is used by one-fifth of all websites, reputable crawlers dominated the space. Beyond some other reputable crawlers, the only other bots were typically those used by scammers and bad actors who sought to exploit vulnerabilities.
With the rise of generative AI and its insatiable need for data, the industry expects a dramatic rise in automated requests from software agents. Prince said the amount of bot traffic online will exceed the amount of human traffic that is online within the next year. This trend represents a structural change rather than a temporary spike in network usage affecting global connectivity.
The executive noted that this change to the web would require the development of new technologies like sandboxes for AI agents to handle the load safely. These sandboxes can be spun up on the fly and then torn down when their task has finished to optimize resource usage. They could come into play when consumers ask AI agents to perform certain tasks on their behalf without human intervention.
Prince imagines there will soon be a time when millions of these sandboxes for agents would be created every second to support the growing demand. Of course, bots’ use of the internet at this scale would require physical infrastructure in the form of data centers and servers. He pointed out that this growth is more gradual than the spikes seen during the pandemic, allowing for better planning.
Internet traffic increased so quickly during Covid, particularly among video streamers, that some parts of the internet were nearly buckling under the strain. Prince added that unlike COVID, where traffic plateaued at a new high, we are seeing internet traffic grow and grow without anything to slow it down. This continuous growth requires robust planning for server capacity and bandwidth to prevent outages.
All these concerns about overload are great marketing for Cloudflare, a company whose services focus on helping websites stay highly available and load quickly. Its offerings include a content delivery network, a series of security and DDoS protections, and Always Online technology. It also provides businesses with tools to block the AI bot traffic they do not want from slowing down their applications.
Cloudflare’s scale gives it the advantage of being able to view the internet’s ongoing evolution and the quickly arising challenges facing the generative AI era. Prince said people do not appreciate that AI is a platform shift similar to the move from desktop to mobile. The way that you are going to consume information is completely different and requires architectural changes.
The executive recalled the web’s earlier platform shifts to contextualize the current transformation in digital consumption and infrastructure needs. Infrastructure providers must prepare for a future where machines outnumber people in terms of active requests across the network. Organizations need to adapt their security and load balancing strategies to handle the new reality effectively.
