Cloudflare Cuts Server Hash Tables by 90%, Freeing Up 100 TB of RAM

Cloudflare has reclaimed another 100 TB of RAM, this time through optimization of its hash-mapping algorithm. The company's primary business involves caching data—serving URLs directly from memory or disk rather than retrieving them from origin servers, a process that can consume significant time. Cloudflare relies on its own open-source Pingora framework to handle the task of routing arbitrary URLs to its network of cache servers using the Ketama algorithm. What appears straightforward in theory becomes complex at Cloudflare's scale, especially when infrastructure constantly shifts as servers come online and go offline.

The process of directing URLs to cached servers demands substantial memory allocation for lookup tables. When a URL arrives, it undergoes hashing—similar to how files are processed with CRC-32—and the resulting number determines which server handles the request. A straightforward approach would assign each URL to a single server sequentially, but this creates an immediate problem: identical URLs must consistently reach the same servers. The solution involves creating hashes for each server based on identifiers like IP addresses and hostnames, then matching URL hashes to server hashes by numerical proximity.

This approach functions well initially, since URL hash distribution tends to be effectively random, and requests distribute evenly across servers. However, a second challenge emerges: imagine four servers each handling 25 percent of traffic, and server #2 goes offline due to a hardware failure. Requests that server #2 previously handled get redirected to the nearest server in the hash map—typically server #3—creating an overload situation while servers #1 and #4 remain underutilized.

The standard solution involves assigning multiple hashes per server and randomizing their placement. With many hashes from each server distributed randomly across the hash space, a server failure causes its traffic to spread evenly among survivors. However, this approach demands substantial memory, particularly when implementing weighted layers so larger servers absorb more requests, and when architectural constraints prevent certain servers from handling all request types. Cloudflare was maintaining as many as 100,000 server hashes per machine, creating significant RAM consumption.

Through mathematical analysis and statistical modeling, Cloudflare's engineering team determined that 100,000 hashes far exceeded the practical threshold for performance gains. Their calculations showed that just 10 percent of that number—10,000 hashes—delivered nearly identical results, with error rates showing minimal improvement beyond that point. The team also optimized Rust data structures to save 2 bytes per entry in the hash-server mapping. While this sounds negligible, the savings compound dramatically across billions of records.

The combined effect of these changes freed approximately 100 TB of RAM across Cloudflare's infrastructure. To minimize deployment risk, engineers implemented the new algorithm as a separate code path rather than replacing the existing one outright, enabling quick rollback if issues surfaced. The approach demonstrates how careful resource management remains possible in modern software systems.