Making AI Agent Testing a Core Part of Product Development

Informal testing through chat interfaces and recorded demonstrations can mask serious problems in AI agent behavior. A team might validate an agent against a handful of representative prompts, see satisfactory results, and move forward with deployment. But when retrieval systems change, models upgrade, or configurations shift, previously working functionality can break in ways that remain invisible until users encounter them or monitoring systems flag anomalies.

A successful demo proves only that an agent functioned under specific conditions at a particular moment. It provides no assurance that subsequent versions will maintain the reliability and consistency users and operators require. Evaluation must transition from a one-time validation step to an ongoing component of the delivery pipeline.

If it can't reproduce a run or a material regression in a high-risk workflow, the product isn't ready to pass the release gate.

A robust evaluation framework executes predetermined scenarios along the agent's actual execution path and generates sufficient documentation to determine whether a release should proceed. The system must exercise the code responsible for assembling context, the tools available to the agent, and the permissions enforced by the runtime environment.

Establish requirements before building tests

Vague criteria like "the answer was good" cannot be tested consistently. Before selecting an evaluation tool, teams should document the specific functions the agent performs, the constraints on each function, and the outcomes that fall outside acceptable operating parameters.

Consider a support agent tasked with answering billing inquiries. A valid job might involve retrieving information from the correct customer account and citing the applicable policy. The agent should be prohibited from modifying service plans or accessing another customer's information. When a policy cannot be located, the agent should acknowledge this limitation rather than presenting unsupported information as fact. The agent may also need to request an account number before proceeding or escalate complex requests to authorized personnel.

Distinguishing between outcomes and the processes that generate them matters significantly. An agent might provide a correct answer despite retrieving the wrong document, or complete a task after calling unnecessary tools or searching beyond the customer's scope. It could even escalate a routine request it should have resolved independently. Such runs may appear successful in transcripts while concealing underlying weaknesses that emerge under different conditions.

Begin with a small set of observable requirements for each function. Required information must be supported by identified sources, and any modifications must receive confirmation. When information is unavailable, the agent should ask rather than speculate. High-risk rules warrant precise assertions; supporting language can accommodate reasonable variation.

Build test scenarios from actual user interactions

A maintainable initial test set should remain modest in scope. Ten genuine tasks drawn from real work provide more value than an extensive benchmark containing prompts users never submit.

Ten real tasks are more valuable than a large benchmark filled with prompts your users never send.

Support tickets, workflow logs, incident reports, and direct user conversations all serve as useful source material. The test set should include routine requests that constitute the bulk of typical workload, supplemented by scenarios involving ambiguous instructions or incomplete information. Include cases where documents are outdated or tools fail to respond. Some scenarios should require approval before the agent can proceed, and others should cover legitimate but uncommon requests.

Since agents operate across multiple turns in a conversation, test scenarios should reflect this reality. A test might request an account modification, supply the missing identifier in a subsequent message, and ask the agent to confirm the proposed change in a third exchange. The test should verify that the agent maintains the account identifier and proposed modification across turns without introducing unrelated information into the final action.

Each scenario requires fixtures that establish its context. The documents and tool responses used during execution should be frozen, and the account state should be pinned to a known configuration. The policy version and the agent's access permissions are equally important. A failure that cannot be reproduced becomes a matter of interpretation; fixed fixtures transform it into a concrete engineering issue.

Production failures should be retained as permanent regression test cases. Over time, the test suite documents the mistakes the team has already experienced and learned from.

Trace the complete execution path

Evaluating only final answers overlooks much of what separates an agent from a simple chatbot. Agents retrieve information, select which tools to invoke, supply arguments, interpret results, and determine whether to continue. Any stage in this sequence can deviate from the intended path even when the final response appears reasonable.

Final-answer scoring misses much of what distinguishes an agent from a chatbot. Any step in that loop can diverge from the intended path even when the response appears convincing.

Capture the original request and system instructions. Record the specific model version and application build. Document the prompt and retrieval configuration, including tool schemas. Then log every retrieved source with its version, every tool invocation with its arguments and results, permission checks, and the final response. Include latency, token consumption, and cost. The trace should answer practical questions without requiring reconstruction from disparate logs.

Combined with server-side enforcement and audit records, the trace should demonstrate that searches remained within the correct tenant and customer account. It should identify the approved policy source and the records cited in the response. For write operations, it should confirm that the user approved the change and that the server-side permission check succeeded. These are deterministic checks: they either occurred or they did not.

Clarity and usefulness involve greater subjectivity. A human reviewer or model-based evaluator can assess whether the response addressed the request, explained a limitation, or posed a reasonable follow-up. Attach these judgments to the trace. When a score declines, the team can identify which step changed.

This approach also makes evaluator failures more apparent. A model-based evaluator may produce different judgments following an upgrade or respond differently to a revised scoring rubric. Store the evaluator's version and instructions alongside its result. Periodically validate a sample of those scores against human reviews.

Separate deterministic rules from subjective scores

Agent quality cannot be reduced to a single unexplained metric. Monitor task completion and factual accuracy separately from retrieval performance. Distinguish policy compliance from user experience, latency, and cost.

Some signals permit acceptable variation: a response requiring 200 milliseconds longer may still be acceptable, and a somewhat longer answer might even improve clarity. Other signals allow no deviation. Unauthorized updates, cross-tenant retrieval, or missing approval should remain release-blocking conditions regardless of other metrics.

Compare a candidate version against a known baseline using identical scenarios and fixtures. Present the reviewer with changed responses and the sources behind them, allowing tool paths and individual scores to explain the differences. If the new version completes more tasks but doubles latency, that may represent an acceptable product decision. If it improves average scores while circumventing a permission gate, it does not.

Repeat scenarios when behavior exhibits variability. A task that succeeds inconsistently—such as once in ten attempts—has not yet achieved a reliable release threshold. Set thresholds according to risk level, and reserve absolute gates for rules the system must follow without exception.

Building evaluation infrastructure from scratch is unnecessary. Tools including Promptfoo, DeepEval, LangSmith, and Braintrust offer capabilities for constructing evaluation workflows. Depending on the tool, support may encompass scenario execution and trace capture. Some tools also employ models to score results.

Understanding the metrics vocabulary proves valuable. Conceptually, pass@k determines whether at least one of k attempts succeeds, while pass^k asks whether all k attempts do. Pass^k applies when consistent behavior matters, but it does not replace exact gates for rules an agent must obey.

Evaluation carries financial implications. Every live, end-to-end run invoking a model consumes tokens. Judge models cost more than string comparisons, and a large suite running on every commit accumulates expenses quickly. Reserve expensive judgments for scenarios carrying genuine risk.

Integrate evaluation into the release process

Execute the test suite whenever the team modifies a model or prompt. Changes to retrieval configuration, memory policies, or tool interfaces also trigger evaluation. Employ a fast test set for routine changes and a comprehensive set before major releases or model migrations. When a behavior change is intentional, require a reviewer to approve the new expectation rather than modify the test.

The records generated by this process require the same safeguards as the agent itself, since evaluation inputs may contain customer information. Traces can include retrieved text and internal instructions. They may also capture tool arguments containing credentials or personal data. Version the records and control access carefully. Consider redacting sensitive values before storage and applying appropriate encryption, access controls, and retention policies based on the data involved.

Maintaining this work closer to operational data can reduce overhead. Oracle AI Vector Search stores vector embeddings alongside business data, and SQL queries can combine similarity search with relational filters and lexical search. Teams using Oracle AI Database can maintain operational records and their vectors within a data platform already under their control.

The same platform can store evaluation traces and enforce access rules. Database-enforced access controls can apply row- and column-level policies within the database, providing an additional layer for enforcing data-access boundaries. The specific platform matters less than the principle: teams need durable evaluation cases, the inputs used for each run, and documentation of why each build passed.

A release gate should block releases containing known high-risk conditions while acknowledging that some judgments require context. Exact checks prevent permission and policy regressions. Thresholds catch measurable quality declines, and human review addresses ambiguous changes that scores cannot resolve.

Begin with ten cases and preserve every significant failure

Select ten genuine tasks this week. Document the expected result and the evidence the agent should reference. Note the actions it must avoid. Establish the fixtures, record the trace, and execute the cases before the next release.

Trust in an agent grows when the team can replay what happened and show that the next release still respects the boundaries users depend on.

When an incident occurs, add it to the suite. When a user discovers a failure nobody anticipated, retain it. The suite expands as the product evolves.

Confidence in an agent strengthens when the team can reproduce what occurred and demonstrate that subsequent releases maintain the boundaries users rely on. The evaluation system constitutes part of the product that ships.

Source: The New Stack