Five Database Engineers Reveal Their Performance Trade-offs at P99 CONF

The tension between read speed, write speed, and storage efficiency defines modern database architecture. At P99 CONF, a virtual gathering focused on low-latency, high-performance applications, engineers from leading firms walked through the specific trade-offs they made when building systems for their workloads. The conference, which ran October 21–22, featured more than 60 sessions; recordings are available on demand at p99conf.io after registration.

Database designers must navigate what researchers call the RUM conjecture: the relationship between read amplification (data read per result), write amplification (data written per logical write), and space amplification (storage used relative to live data). Optimizing one dimension typically increases costs in another. The five presentations below illustrate how different teams made these choices across their systems.

Can Everything be an LSM?

Almog Gavra, Co-Founder at Responsive (now Databricks)

Gavra structured his discussion around the core trade-offs: logs enable rapid writes but burden reads, while sorted arrays do the opposite. He characterized LSM trees—log-structured merge trees—as tools for navigating this spectrum. "I think about LSM trees as a mechanism for tuning between these two extremes," Gavra explained.

LSM trees were originally designed for local disk storage, but their immutable file structure translates well to object storage platforms. Gavra highlighted Amazon S3's advantages in cost, durability, and availability, then enumerated the costs: objects require replacement rather than in-place modification, each GET and PUT incurs charges, and cold reads suffer latency penalties. A local NVMe cache miss takes roughly 150 microseconds, he noted, compared to 20–30 milliseconds for S3 Standard—a difference of two orders of magnitude. LSMs batch writes into immutable files, and caching keeps object-store reads from becoming a critical path bottleneck.

Gavra's open-source project SlateDB implements an object-store-native LSM library. It presents a key-value interface over S3, GCS, and R2, enabling developers to construct data systems without building their own LSM implementation.

How ScyllaDB Eliminates Read Amplification at Scale

Michael Litvak (Senior Software Engineer) and Felipe Cardeneti Mendes (Technical Director), ScyllaDB

ScyllaDB introduced Logstor, an optional storage engine designed for read-heavy key-value workloads that coexists with its existing LSM engine rather than replacing it. Each key maintains an in-memory index containing a hash and the record's flash location. A read operation either retrieves a cached value or locates the record with a single disk I/O, bypassing the LSM search across SSTables, Bloom Filters, and indexed structures. Both engines can operate within the same cluster using ScyllaDB's standard drivers and consistency guarantees.

In preliminary read-dominant testing, the system sustained 1.1 million reads per second with p99.9 latency under one millisecond and p99.99 latency around 1.5 milliseconds. The test employed 2 KB values; earlier LSM examples in the presentation used 1 KB values, so direct comparison is not possible. The team also demonstrated similar throughput and sub-millisecond p99.9 latency in an 80% read, 20% write scenario.

Logstor's read performance comes at a cost: larger RAM requirements for the index and additional write overhead. Even minor updates write the entire new value to a log, leaving the old record as garbage; segment compaction later recovers this space. Mendes positioned the engine for bounded key sets, frequent overwrites, and point lookups where the memory trade-off is justified. During the presentation, Mendes described the compromise as "optimized for time at the sacrifice of space."

The Vertical Scaling Wall, and How Valkey Gets Over It

Madelyn Olson, Principal Software Engineer at AWS

Olson shifted focus from storage to CPU bottlenecks. She described Amazon ElastiCache customers experiencing sudden p50 latency spikes and benchmarks revealing that Valkey's main execution thread had reached saturation.

In Olson's unpipelined GET benchmark with approximately three million keys and 650 clients, p50 latency climbed sharply near 150,000 requests per second. Valkey already possessed I/O threads, but they remained idle while the main thread processed commands. The team deployed lockless queues to keep network I/O and command parsing active on those threads, while command execution remained on the main thread.

The new ceiling reached approximately 1.2 million requests per second in that workload—roughly eight times the earlier limit. Olson emphasized this as a workload-specific result rather than a general throughput guarantee.

For horizontal scaling, Valkey 9.0 introduced atomic slot migration. The source shard retains ownership while streaming a snapshot and subsequent changes to a target shard, then transfers ownership once the target catches up. In a roughly 40 GB, three-to-four-shard test, Olson reported approximately 10 times faster migration without load and about five times faster under heavy GET workload. If migration fails before handoff, the staged transfer can be abandoned without affecting end-user data.

I/O threads can be added or removed as workload demands shift. Olson recommended using horizontal scaling for baseline capacity and vertical scaling for short bursts, while keeping command execution single-threaded for now. Her principle: "Efficiency trumps headline numbers."

Searching 100B Vectors on Object Storage @ 200ms P99

Nathan VanBenschoten, Chief Architect at turbopuffer

VanBenschoten presented turbopuffer's vector search index operating at 100 billion vectors, representing more than 200 TiB of dense vector data. The system reports 200 ms p99 latency at more than 1,000 queries per second with 92% recall in early results. Object storage serves as the durable backing store, but this high-query-rate workload pins data in local SSD caches. Hierarchical clustering groups nearby vectors; RaBitQ binary quantization compresses a shadow index to one bit per dimension, allowing the search to filter candidates before reranking a small subset using full-precision vectors.

Uncompressed search faced bandwidth constraints moving vectors from storage into memory. Quantization shifts computational load into higher cache tiers but also increases arithmetic intensity, making the system CPU-bound. "Often when you push down one bottleneck, a new one emerges," VanBenschoten observed, directing the team toward more efficient vector instructions as the next optimization target.

To fit an index of this scale on SSDs, turbopuffer distributes it across machines. VanBenschoten noted that its separated compute-and-storage architecture favors fewer, larger shards: a replacement machine can point to object storage and populate its local cache instead of receiving a full shard from another node. Queries fan out to the shards, and a coordinator merges their top results.

The Fastest Object Storage Client in the World*

Georg Kreuzmayr, Software Engineer at TigerBeetle

TigerBeetle, a financial transactions database, implements what Kreuzmayr calls "diagonal scaling": keeping hot data on local NVMe while extending the storage hierarchy to shared object storage for colder data.

A failed machine avoids reconstructing data already stored in object storage. Kreuzmayr illustrated the recovery challenge: recovering 128 TB over a 100 Gbps network takes approximately three hours at line rate; if recovery exceeds the interval between failures, a cluster risks entering a failure spiral.

This design requires an object storage client compatible with TigerBeetle's single-core transaction processing. Kreuzmayr identified mismatches in the AWS C++ SDK, particularly its blocking getaddrinfo call: invoking it on the transaction core would halt transaction work. SDK thread pools, runtime allocation, and dependencies also conflicted with the project's design principles.

TigerBeetle built its own multi-cloud client instead of adapting official SDKs. It supports S3, Azure, and GCP and can be tested with simulated I/O and injected faults.

Written in Zig, the client includes custom DNS, TLS, and HTTP components, uses io_uring and zero-copy sends, and allocates its working memory at startup. Kreuzmayr reported the implementation spans under 40,000 lines of code with no dependencies beyond its toolchain. These choices reflect TigerStyle's three principles: safety, performance, and developer experience.

In the talk's S3 PutObject test—8 MB requests over plain HTTP without checksums on an AWS c7gn.metal-24xlarge instance—Kreuzmayr reported uploads approximately 20 times faster than official SDKs and five times faster than a research implementation. For GetObject, he reported about three times the SDK throughput and roughly 25% more than the research implementation. Kreuzmayr attributed the smaller download advantage to the cloud TCP receive path. These results apply to the stated test setup; the conference summarizes the broader client comparison as 4–10 times faster than official GCP and AWS SDKs.

Kreuzmayr's conclusion: building the client required more upfront effort but made the code easier to test and tune. "Code stops being technical debt and ends up being an asset," he said.

Across these five presentations, engineers identified different bottlenecks in storage, search, networking, and scaling. Their designs optimize for specific workloads while introducing new costs or complexity in other dimensions.

More than 200 P99 CONF talks are available on demand at p99conf.io. Registration is required. P99 CONF is a ScyllaDB community event.