Chips

AI infrastructure hits infrastructure limits: power and cooling now the real bottleneck

As enterprises scale AI deployments, securing GPUs has become trivial compared to the challenge of powering and cooling them. The industry is reckoning with a fundamental shift from rack-scale thinking to data-center-scale operations measured in megawatts.

3 min read
Enterprise AI hits a wall of power and cooling, not GPU supply

Acquiring graphics processing units has become the straightforward part of building AI infrastructure. The genuine constraints now lie in power delivery, thermal management and network capacity—factors that determine whether expensive systems actually generate value or remain dormant.

This infrastructure reality dominated discussions at SC26, where commercial AI systems increasingly competed for attention alongside traditional scientific computing. Rack power consumption, which historically peaked around 10 kilowatts, is now trending toward a megawatt—a transformation that demands enterprises examine every layer of their systems, from the network down through storage and compute, according to Jack Hogan, vice president of advanced growth technologies at SHI International Corp.

When we look at the full stack, we recognize that customers are getting tripped up. They can go get the GPUs, but they can't power them. They can't cool them. These are not cheap systems either. Procuring the infrastructure is just the start.

Jack Hogan, SHI International Corp

Edge inference and the megawatt as AI's new yardstick

The financial model underlying AI infrastructure is undergoing transformation alongside the hardware itself. Enterprises previously evaluated deployments by individual rack units or complete racks, but contemporary data centers now function as unified systems with shared memory pools, where every GPU must remain productive to justify its capital expenditure. This shift has reshaped how SHI measures deployment success.

Now we're talking about row and now data center scale. … We're measuring that now in megawatts. So the megawatt now starts to become a measurement that people start to grasp and understand. How many flops can you get through that megawatt? How fast can you process your tokens?

Jack Hogan

SHI's operations span the entire lifecycle of these systems. Drawing on 37 years of industry experience, the company assembles customized racks within its global integration facilities and conducts pre-deployment assessments to ensure systems operate immediately upon arrival. In one recent engagement in New Jersey, SHI is both hosting and managing the customer's infrastructure within its own data center. Edge deployments introduce their own geographic considerations, particularly for workloads like video processing that generate data requiring local analysis.

You can't backhaul that. The networks are not necessarily the right way to move that. We're creating the scenario of both the mega hyperscaler as well as the edge inference and be able to tie those two things together.

Jack Hogan

High GPU utilization does not automatically translate to high productivity. SHI begins each engagement by understanding the customer's business objectives, then works backward to identify the appropriate model—whether that involves deploying cutting-edge large language models or running smaller models on edge devices. Hardware selection follows the same principle, with SHI's AI and Cyber Lab validating GPUs, newer CPUs, inference-optimized processors and neural processing units.

https://www.youtube.com/embed/xB6BBID6qX4?feature=oembed

It's not about the cost of the token or reducing the number of tokens or even finding cheaper tokens. It's about optimizing and ensuring that those tokens are turning into business value.

Jack Hogan

Source: SiliconANGLE · Reporting supplemented by The Silicon Ledger staff.