Engineering Token Efficiency: How to Optimize LLM Systems at Scale

As enterprise AI deployments expand, engineering teams frequently encounter a critical inflection point. The initial symptom often appears as a financial one: monthly API bills spiraling beyond budget. Yet reducing token consumption to a cost-accounting exercise overlooks the deeper architectural reality. Token optimization is a distributed systems and hardware utilization challenge, not merely an accounting exercise.

This analysis examines two production systems—Concierge, a latency-sensitive synchronous customer support agent, and Pathfinder, an asynchronous multi-step autonomous CI debugging agent—to illustrate how autoregressive bottlenecks emerge during scaling and the engineering solutions that resolve them.

Understanding token economics

A token is not equivalent to a word. Major LLM providers employ byte-pair encoding (BPE) to tokenize text, fragmenting words into subword units. Common words remain intact, but uncommon words and punctuation split into multiple fragments. The practical conversion rate: 1 token approximates 4 characters or 0.75 words in standard English.

Production budgeting must account for the structural pricing differential between input and output tokens. Providers charge output tokens at significantly higher rates—typically 4-5X more expensive than input tokens. A baseline mid-tier frontier model costs approximately $3 per million input tokens and $15 per million output tokens.

The quadratic cost accumulation problem

LLM provider APIs maintain no state. To simulate memory of prior interactions, the entire session history and context must be resent with each API call. This means the model's previous outputs become re-billed input tokens on every subsequent step, creating a compounding cost structure that affects both test systems differently.

Let S represent static system context (instructions and schemas), u represent incoming data per step, and r represent the model's response. The input cost for turn k compounds across execution steps. Summing across a complete N-step run produces O(N²) accumulation of history—the exact mechanism driving cost and latency explosions.

Real-world measurements from both systems illustrate the impact:

• 3,100 tokens (Full returns/shipping policies and brand guidelines)
• 1,200 tokens (Tool definitions, system constraints, CI environment data)
• 900 tokens (Massive raw text payloads: log excerpts, file reads, shell outputs)
• 220 tokens (Polite customer-facing answers)
• 300 tokens (Internal monologue + JSON Tool Arguments)
• 10 turns (Average support thread length)
• 15 steps (Average agent troubleshooting loop length)

Applying the quadratic formula to actual usage patterns:

• Concierge: 45,300 tokens per 10-turn ticket
• Pathfinder: 150,000 tokens per 15-step run

Pathfinder's step increment was 4X larger than Concierge, producing a drastically steeper cost curve. A single Pathfinder run stuck in an infinite tool-use loop at 30 steps could consume 570,000 tokens.

Optimization strategies

Reducing per-call token consumption

Prompt hygiene: Hardcoding static reference documentation into the system prompt forces re-parsing of identical text on every turn. The solution: strip static text from the prompt and implement dynamic injection instead.

Concierge implemented a retrieval-augmented generation (RAG) step to fetch only the 2-3 policy snippets relevant to each ticket, reducing the prompt from 3,100 tokens to 380—a 60% reduction across a 10-turn thread.

Pathfinder applied automated prompt compression using LLMLingua-2 to compress verbose CI log files before sending them to the model. Filtering non-essential log lines reduced incoming tool observations by 3X without sacrificing debugging accuracy.

from llmlingua import PromptCompressor

compressor = PromptCompressor(
model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
use_llmlingua2=True
)

try:
compressed_result = compressor.compress_prompt(
raw_ci_log_text,
rate=0.33,
force_tokens=["Error", "Exception", "Failed", "Traceback", "FATAL"]
)
# Pass high-density payload to the frontier model
compact_prompt = compressed_result["compressed_prompt"]
except Exception as e:
print(f"Compression failed, falling back to raw log text: {e}")
# Graceful degradation: pass the raw (or truncated) log if compression fails
compact_prompt = raw_ci_log_text

Eliminating retries: Open-ended prose instructions like "return JSON" caused malformation. Failed parsing triggered synchronous retries, resending the entire accumulated context. The fix: replace natural language formatting requests with strict structural contracts via forced schema validation.

Both systems converted output formats to strict Pydantic schemas for tool-calling mode and tool-execution payloads. Malformed outputs dropped below 0.5%, eliminating tail latency spikes from cascading queues.

# Unified Schema Enforcement for Concierge Responses & Pathfinder Tool Execution
from pydantic import BaseModel
from typing import Literal

class TicketResponse(BaseModel):
    reply: str
    category: Literal["shipping", "returns", "billing", "product", "other"]
    escalate: bool
    confidence: float

# The API is structurally locked into emitting validated JSON matching the schema
response = client.messages.create(
    model="claude-opus-4",
    system=SYSTEM_PROMPT,
    messages=messages,
    tools=[
    {
    "name": "respond_to_ticket",
    "description": "Formulate a response and classify the support ticket.",
    "input_schema": TicketResponse.model_json_schema()
    }
    ],
    tool_choice={"type": "tool", "name": "respond_to_ticket"},
)

Output token bounding: Models naturally generate verbose reasoning chains and conversational filler, inflating expensive output tokens. Where providers expose logit bias, suppress every token outside the valid set at decode time. Where unavailable, constrained decoding libraries (Outlines, Guidance) or forced tool calls with enum-typed schemas provide the same guarantee.

class ClassifyOnly(BaseModel):
    category: Literal["shipping", "returns", "billing", "product", "other"]
    priority: Literal["low", "medium", "high", "urgent"]

State management and caching

The stateless nature of LLM APIs required parsing the static prompt prefix and historical steps on every turn. The solution: introduce explicit cache breakpoints allowing the inference engine to reuse states of static blocks. Both systems flagged stable, historical segments for caching. Under standard vendor pricing, cache reads receive a 90% discount. Verify caching availability with your provider.

# Caching the stable history prefix for a multi-turn session
response = client.messages.create(
    model="claude-sonnet-4",
    max_tokens=4096,
    system=[{
        "type": "text",
        "text": SYSTEM_PROMPT,
        "cache_control": {"type": "ephemeral"} # Cache hits drop prefix costs by 90%
    }],
    tools=TOOL_SCHEMAS,
    messages=session_history + [{"role": "user", "content": current_step_input}],
)

For a 10-turn Concierge chat, this approach reduced input costs by approximately 70%. For a 15-step Pathfinder trajectory, it achieved a 76% cost reduction.

Semantic caching across sessions

Duplicate queries across separate sessions triggered redundant frontier model invocations. A vector similarity cache layer upstream of the LLM using Redis intercepted these requests. Concierge analysis revealed that 34% of customer support tickets were semantic duplicates of common FAQs. Intercepting these requests reduced latency to sub-50ms for cache hits. CI pipeline logs proved unsuitable for Pathfinder caching due to their unique nature.

import os
import json
import redis
from redis.commands.search.query import Query

# Configure connection via environment variable for environment portability
redis_url = os.environ.get("REDIS_URL", "redis://localhost:6379")
r = redis.Redis.from_url(redis_url)

def get_cached_response(tenant_id, query_text, threshold=0.92):
    try:
results = r.ft(f"cache_idx:{tenant_id}").search( # scoped by tenant -- see below
Query("*=>[KNN 1 @vector $vec AS score]").sort_by("score").dialect(2),
query_params={"vec": query_vec.tobytes()},
)
except redis.RedisError as e:
print(f"Redis cache error: {e}")
return None # Fail-open: gracefully fall back to a cache miss)
  query_vec = embed(query_text) # small, fast bi-encoder -- not the frontier model

try:
results = r.ft(f"cache_idx:{tenant_id}").search( # scoped by tenant -- see below
Query("*=>[KNN 1 @vector $vec AS score]").sort_by("score").dialect(2),
query_params={"vec": query_vec.tobytes()},
)
except redis.RedisError as e:
print(f"Redis cache error: {e}")
return None # Fail-open: gracefully fall back to a cache miss

Semantic caching introduces security risks. If a global cache is used, Customer A's account-specific answer could be served to Customer B due to embedding similarity. Mitigation: split the cache into two tiers—a global cache for tenant-agnostic content and a per-tenant, per-user namespace keyed with the tenant ID for anything touching account state.

Cache poisoning represents another risk. Mitigations include writing to cache only from responses passing schema validation and injection-pattern classification, stamping every cache entry with its source traceId, and encouraging routine purging of unknown caches.

Context compaction

Uncapped conversation or agent trajectories allowed N to grow continuously, expanding the cost curve and degrading latency. The solution: cap N by implementing a sliding window that summarizes historical context via a small, ultra-cheap model. Concierge retained the last 3 turns verbatim while condensing older turns into a rolling metadata block.

Pathfinder trimmed and summarized the oldest tool execution outputs into a compact chronological timeline when debugging steps exceeded 4 runs, transforming the open-ended quadratic cost explosion into a predictable, bounded window.

def compact_session_history(history_steps: List[Dict[str, Any]], keep_recent: int = 3) ->     List[Dict[str, Any]];
    """Flattens older history into a cheap summary block, preserving recent context."""
    if len(history_steps) 

Model cascading and routing

"Directing every single operation to an expensive frontier model represents massive overprovisioning for mundane tasks." LiteLLM was integrated as an internal routing gateway to implement model cascading, routing each request to the lowest-cost model capable of completing the task.

# litellm_config.yaml
model_list:
  - model_name: fast-path
    litellm_params:
      model: openai/mistral-support-ft
      api_base: http://vllm-internal:8000/v1
  - model_name: frontier-path
    litellm_params:
      model: anthropic/claude-opus-4

Simple, repetitive tasks route to lower-cost models, offloading 70% of Concierge chats from the frontier model. Pathfinder's agent loop broke into separate sub-tasks: high-level planning, tool selection, and code-patch synthesis remained with the frontier model, while mechanical, text-heavy operations—log parsing, regex extraction, error-string formatting—offloaded to lower models. This hybrid orchestration reduced Pathfinder's token costs by more than 50%.

Lessons for production AI systems

The transformation of Concierge and Pathfinder demonstrates a fundamental principle about production AI: scaling cannot rely solely on the natural language capabilities of a frontier model. The system must be engineered around it. By shifting focus from naive token reduction to maximizing system resource utilization, absolute control over infrastructure was reclaimed.

Efficiency in the era of gen AI is not defined by how cheaply you can operate but by how densely you can pack information.

Efficiency in the era of generative AI depends on how densely information can be packed, how quickly it can be served, and how reliably output can be parsed. The architectural decisions detailed here transcend token optimization strategy; they form a required foundation for building high-throughput, battle-tested, and resilient AI systems at scale.

Source: The New Stack