Caching LLM Responses: A Three-Tier Strategy to Cut Inference Costs

Every time an LLM answers the same question, you pay again. The economics of token-based billing mean that duplicate requests become duplicate expenses unless you intercept them first. The challenge lies not in recognizing the problem but in building a framework to decide whether a previously cached response remains valid and safe to return.

This pattern mirrors a longstanding issue in data engineering. Production pipelines often recalculate aggregations that have not changed since the previous run, burning compute resources while producing identical results. The waste becomes visible only during cost reviews, when someone notices that significant upstream processing repeats work whose inputs never shifted. The solution—change detection through input hashing and dependency fingerprinting—applies directly to LLM workloads, where repeated requests trigger repeated charges.

Duplicate LLM requests arrive through multiple channels. Upstream users converge on similar questions. Batch jobs invoke the same boilerplate repeatedly. Prompt-engineering experiments in development and continuous integration runs repeat identical prompts. Tool-calling agents may hit the same knowledge-base tool many times in a single work session. Understanding the distinction between native prompt caching and response caching matters here: prompt caching lets providers reuse cached computation at reduced rates while output generation remains billable, whereas response caching attempts to skip the model call entirely when an answer already exists in your own infrastructure.

Tier 1: exact match

The simplest approach normalizes the model request body, runs it through a cryptographic hash like SHA-256, then looks up the hash in an in-memory store such as Redis. A matching hash means returning the cached answer without waiting for model inference. Exact-match caching works best when model requests are bounded and predictable—a description that fits most batch pipelines, CI runs, and boilerplate summarization tasks.

Tier 2: semantic match

Exact matching falls short for many workloads. When you need to look up a response for a query that is close but not identical, take the user's query, run it through an embedding model, and store the resulting vector in a vector database. When a new query arrives, embed it the same way and search for close matches by cosine similarity.

The threshold for "close enough" requires tuning. A common starting point sits in the [0.90, 0.95] range, but treat that as a number to validate against your embedding model and data, not as a default. Vector stores differ in their reporting: some return cosine similarity rising toward 1 for closer matches, while others report distance falling toward 0. Confirm which your threshold compares against. A looser threshold raises the risk of wrong matches, where the system answers one query while the user asked about another—conflating "What's the weather in my town?" with the same question about a different town, for instance, just because cosine similarity is high.

Tier 3: hybrid

A common approach runs both tiers in sequence: check the exact-match store first, and run semantic search only on a miss. When semantic search returns a close-enough match, promote the result back into the exact-match store under the hash of the new query that triggered it. This way, the paraphrase and its answer become an exact hit next time, favoring cheap exact matches on repeat traffic.

Both tiers key on more than query text alone. The context and documents in the prompt, the model and its settings, the version of any retrieved source, and the caller's access scope all factor in. Two identical questions asked against different documents, or by users with different permissions, must not share a cache entry.

def cached_completion(query, ctx):

    # ctx bundles everything that changes what the correct answer is:

    # the context/documents in the prompt, the model and its settings,

    # the source-version of any retrieved content, and the caller's access scope.

    key = sha256(normalize(query, ctx))

    # Tier 1: exact-key lookup on Redis (O(1)).

    # Correctness still depends on cache contents, request scope, and freshness.

    if (hit := redis.get(key)):

        return hit

    # Tier 2: semantic search, restricted to the same scope as the request.

    emb = embed(query)

    match = vector_db.search(emb, top_k=1, filter=scope_of(ctx))

    if match and same_scope(match, ctx) \

            and match.score >= threshold_for(category(query)):

        # Promote, but preserve the original freshness deadline.

        remaining = match.expires_at - now()

        if remaining > 0:

            redis.set(key, match.response, ttl=remaining)

            return match.response

    # Miss on both tiers: call the model, validate before writing back.

    resp = llm(query, ctx)

    if is_valid(resp):  # no errors, no empty payloads, no malformed JSON

        ttl = ttl_for(category(query))

        redis.set(key, resp, ttl=ttl)

        vector_db.insert(emb, resp, ttl=ttl, scope=scope_of(ctx))

    return resp

One threshold does not fit all categories. Code-like queries often need stricter thresholds, around 0.95 or higher, because small wording changes can produce entirely different results. Conversational queries can tolerate looser thresholds, in the 0.85 to 0.90 range. These numbers serve as starting points, not settled values—validate them for your own workload and embedding model before relying on them. Cache freshness follows the same principle, and the right time-to-live (TTL) depends on how much staleness the use case can tolerate, not on the data type alone.

A cached market-data answer might be acceptable for only a minute or two, because a stale price can be actively misleading. An internal HR policy answer can often be reused for weeks, because the underlying document rarely changes and a slightly old answer is usually still correct. The interval represents a judgment about acceptable staleness, not a fixed property of the content.

The math

Consider a workload of 1,000,000 calls per month at $0.006 per call, roughly $6,000 with no caching. A hybrid cache delivering about a 60% hit rate avoids 600,000 calls to the model, while embedding and vector-store costs come to about $150. Monthly spend drops closer to $2,550, a 57.5% reduction, plus the latency win of answering many questions without waiting on the model. One essential caveat: measure your hit rate before projecting any savings.

The decisions

Beyond the tiered framework, tune your TTLs to the freshness each data type actually needs, and invalidate entries when you update the content behind them. A fine-grained approach assigns a per-category TTL based on how quickly each answer goes stale: a news summary might hold up for an hour, while a live sports score is worthless within seconds and should not be cached at all during a game.

Live scores require a freshness policy matched to the application. Verified final scores can support much longer caching, with invalidation for corrections. The distinction hinges on whether the underlying value is still moving. A blunter approach skips per-category tuning entirely and purges the whole cache whenever the source content changes. Either way, run the cache in shadow mode first, logging what you would have returned without changing behavior. Evaluate cached answers against verified reference answers or expert review. A fresh model response can help identify differences, but it is not ground truth.

Warm the cache from a historical set of common queries before you rely on it, and validate answers before writing them back, so you do not poison the cache with errors, empty responses, or malformed content.

When should you skip caching? Avoid it for requests with personal or account-specific data, to prevent leaking one user's cached output into another's request. Skip it for creative tasks, where you want a different answer each run. And skip it for genuinely real-time data like stock prices and live inventory, where an answer even a minute old may be too stale for the application.

The takeaway

The principle predates the web: Donald Michie described memo functions in 1968. When you can, fingerprint the question and store the hashed exact form alongside the semantic-variant form, so you avoid repeated model calls while a valid cached answer remains available.