Chips

Inference chip makers pivot to existing data centers as new builds lag behind demand

Construction delays and capital constraints are creating a shortage of AI data center capacity, prompting chipmakers to develop air-cooled systems that can operate in older telecommunications and colocation facilities without expensive retrofits.

2 min read
Stalled data center buildouts send inference chips hunting for space in telco and colo sites

The pace of inference workload growth is outstripping the ability of the industry to construct purpose-built AI data centers, as power availability, financing constraints and regulatory obstacles extend timelines for new facilities. This mismatch is driving chipmakers to identify usable space within existing infrastructure.

Older telecommunications and colocation providers possess both floor space and electrical capacity, yet most lack the infrastructure to support the dense, liquid-cooled GPU configurations that have become standard. This limitation is creating an opening for air-cooled inference processors, according to Anton McGonnell, vice president of core product at SambaNova Systems Inc.

We as an industry are just encumbered by the velocity of data center buildouts, and that's been slowed down by just the physics of how long this takes, by the access to capital, and more and more by legislative hurdles — particularly in the U.S.

Anton McGonnell, SambaNova Systems

McGonnell shared these observations during an exclusive interview at SC26 with theCUBE Research's John Furrier and Gemma Allen, broadcast on theCUBE, SiliconANGLE Media's livestreaming platform. The discussion centered on AI infrastructure constraints and the emerging market for air-cooled inference systems.

Reviving brownfield capacity for AI data centers

SambaNova's fifth-generation processor, the SN50, is engineered for the decode phase of inference operations, where the system generates output tokens and data movement becomes the limiting factor. The company's rack design maintains air cooling and remains lightweight enough to deploy in existing facilities without structural modifications. In contrast, contemporary GPU racks often require floor reinforcement and between 120 to 250 kilowatts per rack, McGonnell noted.

On all of this stuff — the power and the cooling gear — there's 18- to 24-month lead times to get this equipment into the data centers. Never mind all of the construction costs.

Anton McGonnell, SambaNova Systems

SambaNova, which reached a $11 billion valuation in July, combines its decode-focused chips with partner solutions that manage prefill, the computationally intensive initial phase of inference. At SC26, the company is showcasing Intel's D-Series GPUs handling prefill workloads alongside its SambaRack SN50 for decode operations in an air-cooled heterogeneous configuration.

https://www.youtube.com/embed/fP4zfctTS7c?start=26&feature=oembed

They have developed a chip that has much, much better cost performance for the prefill phase of inference. They've sacrificed HBM, which isn't needed for prefill because it's a compute-bound problem.

Anton McGonnell, SambaNova Systems

Source: SiliconANGLE · Reporting supplemented by The Silicon Ledger staff.