AI Bots - a new risk and opportunity for CIOs to manage

A new risk from bots crawling the web estate is beginning to impact digital leaders. Bot crawling is as old as the internet. But now AI is creating a tsunami of bot activity and adding costs to CIOs’ budgets. Digital leaders are worried that, unlike the previous era of bots, they will not benefit from trade and revenue. 

The new wave of AI firms, however, will benefit from increased utility at your expense.

CIOs and CTOs in our network are seeing an increase in automated traffic to their infrastructure and are worried. These digital leaders represent a diverse set of sectors such as property, financial and professional services, tech firms, academia, manufacturing, media, and charities. With one reporting a 400 percent increase in AI-generated website crawling. Distributed cloud computing platform provider Akamai reported in its recent State of the Internet report that AI-driven bot traffic had surged 300% in the last 12 months, adding:

Creating an environment where automated abuse is a primary driver of fraud, infrastructure cost overruns, traffic volatility, and customer friction.

For CIOs, this can mean an increase in operational costs as automated requests increase API and cloud demands. That can mean higher bills from service providers and a decrease in performance and experience for customers and end users. Content delivery network (CDN) provider Cloudflare reports that Googlebot was responsible for the highest volumes of traffic on its network in 2025, and that internet traffic grew by 19 percent in the same year. It goes on to state that AI bots were responsible for 4.2% of HTML request traffic, with Googlebot accounting for 4.5% alone. Henrique Teixeira, SVP for strategy at Saviynt, a provider of an identity security platform, says:

The challenge of dealing with traffic from bots versus humans is nothing new; the majority of web traffic is API driven, so machine-to-machine.

The bottom line

This increased demand on the CIO’s estate has an impact on the bottom line. Tom Howe,  Director of Field Engineering at Hydrolix, a real-time data platform business, and a former Director of Software Engineering at media firm Disney, says distribution bills have increased in the last six months, and that ISP bills had a six-figure increase from unwanted traffic. He says of one business he has dealt with:

What was discovered is that bots were crawling sites in a novel way and found a set of high-resolution images on a web server, and due to the non-deterministic behavior of the bots, it was then downloading the images. The reality is, even if bots are not malicious, they can lead to very unexpected consequences.

This is a case of poor asset management and increased bot activity combining to damage the bottom line. The lesson for this business and all CIOs is that it is important to understand and plan for the behavior of AI bots. As the Akamai report states, AI bots repeatedly fetch the same content, driving up egress, compute, and storage costs, with no clear value for business. 

Angel Maldonado, founder and CEO of Empathy AI, a retail technology provider, says major European retailers have experienced these spikes and have found that attempts to negotiate with the likes of Google have fallen on deaf ears. He adds that CIOs need to consider that as they introduce AI into the business, they are creating dependencies, and so too are all other CIOs, straining the social contract that underpins the web.

If businesses do not understand and manage the activity of AI bots on their estate in a mutually beneficial way, they face major risks. Bot activities, to date, have been largely beneficial, leading search engines and, therefore, consumers to your company. Some of this behavior and benefit will continue in the AI era. However, it will become essential for digital leaders to track and manage bot activity to ensure the business gets that benefit. If they don't, costs will increase, and there is the potential for revenue streams to decrease.

CIOs and CMOs need to understand page views to ensure the sales funnel is flowing towards income. They need to know that APIs are benefiting their business and partners, and not AI firms that do not deliver revenue back to their employees and shareholders. Consider that OpenAI is reported to be delivering revenue of $12 billion per annum.

These are not just business issues, for health and public sector organizations, who are typically content-rich, citizens are funding AI firms that do not contribute to the upkeep of their civic society and health resources.

Rethinking bots management

AI is requiring digital and business leaders to rethink swathes of business operations, and how the firm deals with bots is no different. Former Disney tech leader Howe at Hydrolix says:

The challenge of the rise of AI bots is knowledge about what they are doing and how to identify what they are doing. Then you can extract their intent and then behave accordingly as a business.

He adds that digital leaders need to gather information on AI bot behavior and monitor patterns of activity, then firms can make real-time decisions on managing the bots. Hydrolix, of course, has a tool on offer to manage this, but the right tools will be essential, as Howe says:

Bot developers are smart; they are going to figure out that if they hit a site with regularity, they will get blocked, so they will adapt.

Which means CIOs and their firms must continually adapt to protect the business from both malicious and uneconomic AI bot activity. Teixeira agrees and adds:

You need to have onion layers to detect the bots and identify those that are machines and those that are people and act accordingly.

One CTO in our network says this needs explaining to the business and then put into practice:

A balanced approach is needed and then put measures in place to allow AI bots to crawl a paragraph of each article that’s paywalled - just enough to provide context and basic information, but all the juicy bits remain behind the paywall.

Observability

As ever with new technology waves, existing technologies and methodologies get a new lease of life, and AI is no different. Observability and content delivery network technologies will increase in importance as they secure revenues, intellectual property, and the cybersecurity posture of the business. Howe says:

Traffic log technology is the forgotten middle child, but it has become increasingly important because of these new interaction patterns from AI, which are a source of risk and opportunity.

CIOs will need data from these tools turned into insight that will inform business and AI bot decision-making. Observability and bot insight tools are going to become an essential part of the IT estate; we are already seeing examples such as at Virgin Atlantic. Howe adds:

We will have to rethink a lot of assumptions on authentication and authorization, and what we have done in the past will not cut it.

My take

The AI era has to be fair. If a CIO pays for the infrastructure, the firm invests in content and a supply chain, and that then informs an AI business, then a reasonable transaction has to take place. This has to be a transaction that correlates to the costs, investments, skills, creativity, and experience of the business, its partners and suppliers. If the balance continues to tilt equitably towards the AI-firms and their bots, then that is not a fair trade.

In addition, as digital leaders create agents and send them out to work, they will need to consider the impact of their agents on the wider ecosystem of companies, suppliers, and society.