Your browser doesn’t support HTML5 audio
One compelling narrative being shaped by Big AI has been that achieving AI dominance – whether against China or competitors – requires ever larger AI data factories. Powering these requires ever-larger power plants and wires, with the costs and environmental impacts likely passed on to ratepayers and neighbors. Indeed, even the current Trump administration recently invited Big AI to a White House energy pledge photoshoot.
Against this backdrop, Qoob CEO AJ Javan believes that smaller is a better bet in terms of predicted market size and systems engineering.
The shift toward smaller, distributed data centers near population centers is driven by a fundamental change in AI consumption, which is the transition from training-heavy workloads to an inference-dominant market. Inference is projected to account for the majority of all AI compute needs by the end of this year. Large remote GPU farms face increasing power grid bottlenecks with long interconnection queue times. Smaller distributed data centers process information at the source, thus reducing latency and significantly reducing operational costs. A network of smaller data centers also provide redundancy and ensure any outages remain localized and contained.
In the short term, large remote GPU farms continue to dominate AI capacity growth in terms of mindshare and GPUs deployed. But Javan says this growth is slowing down due to data scarcity and diminishing returns. Meanwhile, the demand for smaller-scale distributed AI-based data centers is surging. One big driver is inference, which is projected to account for the majority of all AI compute needs by the end of this year. Also on the training side, many enterprise use cases could be served just as well by training domain-specific models using thousands of regional training nodes rather than a few centralized gigawatt-scale campuses.
How big is enough?
All of the Big AI and cloud vendors have announced ambitious plans, backed by dubious financing arrangements, to outdo each other in what feels like an AI infrastructure arms race. A new 1+ gigawatt data center sounds impressive on paper, until you work through all the system details, like unrealistic nuclear power schemes, current grid bottlenecks, water consumption, and ratepayers who seem to be growing a little more cynical about how they are going to be left with the bill.
There have also been some obvious economies of scale in infrastructure as the industry refines the supporting architectures and processes for cluster orchestration, memory management, and data pipelines to support bigger and longer training runs. But now, the balance between the benefits of larger data centers and the system-engineering complexity is starting to favor smaller data centers. Javan explains:
The engineering sweet spot has shifted toward smaller 50 MW (megawatt) to 150 MW modular campuses that prioritize rack density, allowing for faster deployment than gigawatt-scale campuses. Constraints to be addressed include balancing 100 kW+ rack densities with direct-to-chip (DTC) liquid cooling while meeting the structural demands of heavy racks, requiring specialized reinforced flooring. Managing and orchestrating these clusters requires integrated management software that synchronizes power, thermal loads, and hardware health in real-time.
The primary constraint that is significantly harder to scale is utility-level power grid interconnection, as the mismatch between the 18-month construction cycle for a data center and the 6+ year timeline for grid expansion has created a massive bottleneck. Water scarcity is also a critical bottleneck, but Javan believes it's more localized and can be addressed by innovating in DTC liquid cooling. There are also far more potential locations with access to 50 to 150 MW of power than to gigawatts. Javan says:
From a systems engineering perspective, the practical limit for a cost-efficient AI factory today lies between 50 MW and 150 MW modular campuses. While gigawatt-scale mega-farms are possible, they face grid bottlenecks, making smaller sites a more viable and rapidly deployable alternative.
Dynamic micro-grids
One of the biggest gaps between new energy pledges and the felt experience of local ratepayers seems likely to lie in the details. For example, they might be constructed with sufficient local power to achieve a nominal capacity that minimizes grid energy purchases while still passing grid upgrade costs on to consumers for resilience and stability. This could arise from fluctuations in operations associated with large training runs or site power outages.
The traditional approach to data centers has been relatively static local direct current (DC) infrastructure fed by alternating current (AC) grids or by power equipment such as gas turbines or diesel generators. Even with renewable power sources like wind and solar, the default has been relatively simple power transformers to shape the raw energy into the appropriate form for the AI chips. And the local grids are also evolving from 48 volts in legacy IT equipment towards 100 volts in new AI systems, with predictions of 400- or even 800-volt wiring in the near future. The benefits are improved power efficiency, denser racks, and lower copper requirements.
An ancillary issue is that the electrical currents in traditional AC infrastructure must be kept tightly synchronized for safe operation, which can be challenging when starting with DC sources like solar. They need a substantial source of reactive power, such as a traditional power plant or specialized equipment, to restart after a blackout.
Innovations in new energy infrastructure to support more dynamic microgrids can help mitigate these challenges. These can independently disconnect from the main utility grid when needed to balance supply and demand in real time intelligently. Unlike traditional microgrids, which mostly serve as static backup systems, dynamic microgrids could act as active grid participants, using AI to optimize energy flows from various on-site energy sources. Javan believes:
Dynamic solar microgrids will transform the needs of the data center from a static energy consumer into a proactive energy and demand partner. By utilizing on-site solar generation, operators can reduce capital and operating expenses, since solar, as a permanent power source, is one of the lowest-cost and fastest to deploy energy sources. AI-driven management systems can achieve further cost reductions through real-time energy optimization and peak shaving, using a combination of solar plus battery storage.
Despite the current rhetoric about nuclear, Javan argues that solar and battery combined are still lower in cost than building a new nuclear power plant. Also, solar and wind can provide cost advantages by coupling with battery storage to achieve a levelized cost of energy significantly cheaper than traditional utility power, especially in power-constrained markets. These advantages are also likely to survive the loss of federal support, driven by declining installation costs and massive load growth.
My take
I think Javan’s theory that the market favors more numerous, smaller data centers over bigger ones is more realistic than the current "bigger is better" paradigm championed by Big AI. After all, the way to actually make money with these models is to run them in production, not just build bigger or better ones. And that is going to mean a lot more inference and fine-tuning of existing models.
Also, the shift toward dynamic microgrids sounds promising and necessary, but it is still early. In the long run, this approach is likely to improve the economics of solar and wind in general, aligning their physical and management characteristics with those of traditional power sources. The rush to build more AI data centers, whether big or small, could also help lower the costs of integrating renewables into the grid.
Water is likely to be less of an issue for renewable energy sources like solar and wind. However, it's also likely to be an emerging issue for other energy sources that are prominent in the energy dominance strategy, including nuclear, coal, and, to a lesser extent, gas. One assessment I came across suggested that nuclear power, for example, requires almost four times as much water as traditional data center evaporative cooling, and gas-fired power adds about twice as much water overhead.
This is likely to become an issue in areas already experiencing water stress, whether due to increased water costs associated with infrastructure or to the depletion of shared aquifers. Water is not mentioned at all in Trump’s Ratepayer Pledge Protection Proclamation. I was even surprised to recently read that even Trump is starting to push back on some aspects of the Energy Dominance meme. Last Wednesday, Trump recalled to an amused audience, chiding US Energy Secretary Chris Wright for being a bit overzealous in talking about “clean, beautiful coal,” instead of just mentioning coal, the Guardian reported.
It’s a minor detail in Trump's rapidly evolving meme train but it also comes in the midst of spiking energy costs associated with events in Iran. I sensed an opportunity for a shift towards combining two of Trump’s favorite things to talk about dominating into ‘AI-energy’ dominance. This could provide an interesting middle ground between Trump’s previous hostility towards the “green scam” and broader aspirations of a more sustainable future, not to mention cheaper bills.