The Hidden Cost of Splitting Real-Time Processing and Storage

Architectures for real-time systems frequently duplicate state across multiple platforms—a pattern that carries expenses not always visible in infrastructure costs. The typical design chains together Kafka for event ingestion, Flink for stream processing, Redis for serving current values, and a database as the authoritative record. Vector search workloads may introduce additional components into this stack.

This separation of concerns provides operational advantages: each tier can scale independently, and the architecture offers flexibility in deployment and technology choices. Yet the arrangement creates friction when data must traverse between components or when the same information exists in multiple places simultaneously.

Not all duplication deserves scrutiny. Replicas safeguard against outages, checkpoints enable recovery, and immutable event logs preserve transaction history—these are justified expenses. The problematic copies are those that exist primarily because one system performs calculations while another serves the results to applications.

Consider a fraud-detection system that tracks transaction velocity on a payment card over a five-minute window. Flink maintains the internal state needed for this calculation, but a fraud-scoring service must retrieve the current count. Since Flink's internal state storage wasn't designed for general application queries and the company deprecated its Queryable State feature in version 1.18, the standard approach publishes the count to Redis. This introduces a serving layer with its own scaling requirements, but each update demands serialization, network transmission, and processing before consumers access it. The state in Flink and the value in Redis may diverge, creating a gap between what the processor knows and what applications can read.

Where consistency gets complicated

Synchronizing state across systems becomes more intricate when applications span multiple availability zones.

Imagine a payment system that distributes read traffic to asynchronous replicas across several zones to minimize cross-zone communication. A refund transaction completes, but the update hasn't yet reached all replicas. A service querying one of those lagging replicas might see the original charge still active and process a second refund. Without proper coordination between refund operations, the customer receives duplicate payments.

Without coordination between the refund paths, the customer could be paid twice.

This challenge exists wherever application logic makes decisions based on asynchronously replicated data, regardless of whether processing and storage run on the same platform. The core problem combines replica lag—the delay before updates propagate to all copies—with application code that doesn't atomically check and modify the state governing the refund. If the refund contacts an external payment processor, that service must also support idempotent requests to prevent duplicate charges from retries.

Separating processing from serving introduces another timing issue: export lag, the interval between when the processor calculates a new value and when the serving store receives it. This differs from replica lag within the serving tier itself. During export lag, a wave of fraudulent transactions might be evaluated against an outdated count. Both export lag and replica lag can cause applications to read stale information, but neither should be conflated with the check-then-act race condition that could trigger the duplicate refund.

Hazelcast offers one solution for certain workloads: entry processors that atomically modify individual map entries on the cluster member storing that key, eliminating the separate publication step. This approach doesn't guarantee strong consistency during network partitions. Applications requiring stronger guarantees can employ data structures within Hazelcast's Raft-based CP subsystem, though this doesn't automatically make ordinary IMaps strongly consistent or make map updates atomic with external payment services. Other architectures combining separate processing and storage systems can also implement strongly consistent coordination.

The cost of moving state

Discussions of in-memory systems typically emphasize memory consumption, but CPU and network expenses often prove equally significant.

A 2025 study from UC Berkeley and Databricks titled "Rethinking the Cost of Distributed Caches for Datacenter Services" analyzed caching economics using synthetic workloads, traces from Meta, and production data from Databricks' Unity Catalog. Using Google Cloud pricing of approximately $17 per vCPU-month and $2 per gigabyte-month of memory, the researchers determined that adding in-memory caching reduced operating costs by a factor of three to four. CPU savings exceeded the expense of additional RAM.

This finding doesn't directly compare Hazelcast against a Flink-plus-Redis combination; rather, it contrasts caching with no caching. The study's comparison of remote caches to co-located caches does support the narrower point that serialization and network communication consume CPU. The research also showed that per-read consistency checks could eliminate much of the cost advantage, meaning valid comparisons must account for equivalent consistency models.

A 2019 paper by Google researchers, "Fast Key-Value Stores: An Idea Whose Time Has Come and Gone," documented a 40% reduction in p99.9 latency for ProtoCache, a Google application component, after migrating from a remote key-value store to an architecture colocating application logic with in-memory state.

A straightforward calculation illustrates the memory cost of maintaining duplicate working sets. At the pricing assumptions in the Berkeley paper, 400 GB of memory costs roughly $800 monthly before accounting for backups, checkpoints, and supporting infrastructure. Duplicating an equivalent amount across both processing and serving tiers would double this figure, though Flink's intermediate state and published Redis values often differ substantially.

Beyond memory, the operational burden of keeping systems in sync adds overhead. Every update requires serialization, network transmission, and processing, and both platforms demand monitoring, capacity management, and maintenance.

Tracing a single read-modify-write operation reveals this overhead. With separate processing and serving systems, an event flows through ingestion, keyed state updates, result serialization, publication, and finally storage in the serving tier before an application reads it—potentially four or five serialization or network steps for one change. Running computation where state resides can compress this to a single step. While not a precise cost measure, mapping an update's path through the architecture helps pinpoint where CPU and network resources disappear.

A complete cost comparison requires accounting for node counts, instance types, licensing, negotiated discounts, and whether deployments are managed or self-hosted. The $800 figure represents memory arithmetic alone, not a full total-cost analysis or measured Hazelcast savings.

Bringing computation closer to the data

Hazelcast provides a mechanism for moving processing nearer to stored data. Rather than shuttling information between separate systems, it distributes data across cluster members by key and executes computations on the member holding that key. Backup copies protect against member failures.

Hazelcast supports two approaches depending on processing requirements. The Jet engine maintains internal state for windowed stream calculations and publishes results to a distributed map that applications can query. This intermediate state and result-publication step remain on the same platform, though processing, map writes, and backups may still generate traffic between members. Kafka can continue serving as the durable event log, and the authoritative database can remain in place.

For workloads that need to modify stored values directly, entry processors execute code against the map entry itself rather than computing results elsewhere and copying them across. Returning to the transaction counter example: an entry processor could maintain per-minute transaction counts in a distributed map, and the application could sum the five most recent buckets to derive the total. This eliminates the need for a timer to expire old transactions, though results would be accurate to the minute rather than the second. The application must still handle late or duplicate events.

Entry processors should remain lightweight, as long-running operations can starve other work on the same partition. Network traffic still flows between clients and the cluster, and Hazelcast still maintains backup copies. Serialization costs depend on the map's in-memory format: the default BINARY format stores serialized values, while OBJECT can reduce conversion overhead for entry processing. For workloads suited to this pattern, the advantage is eliminating a separate processing tier that continuously publishes updates to the serving store.

When separate systems make sense

Decoupling processing from storage offers one major benefit: independent scaling of each tier.

Consider an application executing complex decision logic that yields only a binary result. It might demand substantial CPU resources but minimal storage. A separate processing tier lets teams add compute capacity without expanding the serving infrastructure. Conversely, other applications maintain large state but perform minimal processing. Combining both workloads on a single cluster may yield limited benefits, especially if CPU-intensive tasks contend with latency-sensitive reads.

Separate platforms also enable independent deployments, workload isolation, and access to specialized processing capabilities. Hazelcast can provide some of this flexibility within a single cluster through lite members, which handle compute tasks without storing data, allowing teams to dedicate regular members to storage and add compute capacity separately.

For certain workloads, maintaining entirely separate processing and storage platforms remains the right choice. But when applications frequently read and update the same state, Hazelcast can reduce the data movement penalty between systems.

What to measure before making the switch

Begin with a profile of the current deployment. How much CPU time goes toward application logic versus serialization and network communication? How much state is genuinely duplicated between processing and serving?

Average response times provide incomplete information. A system might perform acceptably most of the time but degrade sharply on certain requests. Examine p99 and p99.9 latency percentiles alongside throughput and end-to-end latency—the elapsed time from event arrival to result readability, including export lag.

Run the comparison using identical workloads, the same end-to-end latency targets, and equivalent durability guarantees. Failure scenarios matter: measure recovery time, replay behavior, duplicate handling, and correctness during node failures and network partitions. Backup copies and recovery mechanisms add to infrastructure costs.

Some applications will continue to benefit from independently scaling processing and storage. Others may discover they're spending resources to move data between systems when they could perform the same work directly against application-accessible state. Not every copy is wasteful. Replicas guard against failures, checkpoints enable recovery, and stream processors require intermediate state. The copies worth examining are those that exist mainly because processing and serving operate in different systems.