AWS Emulator Shows Why Coding Agents Need Real Feedback: Testing Against Mocks Fails
Amazon's new AWS Emulator lets AI coding agents test against a local copy of AWS services, but the real lesson is about verification loops. Without realistic feedback, agents learn the wrong lessons—and that problem extends far beyond cloud APIs.

This week, Amazon Web Services unveiled AWS Emulator, a locally-runnable version of AWS that enables coding agents to validate their work. The tool encompasses more than 200 AWS services and was constructed entirely through agent-driven development. The most compelling aspect of this release centers on an anecdote shared by Swami Sivasubramanian during the announcement.
Roughly two decades ago, Sivasubramanian participated in constructing the replication engine powering Alf, a Paxos-based lock service that underpins AWS infrastructure including S3 and DynamoDB. The initial implementation required approximately four months of work. When Sivasubramanian later tasked Kiro, AWS's agentic coding system, with recreating it under his supervision, he spent four nights observing before identifying the core issue: the agent lacked mechanisms to validate its own output. Once provided with a specification and testing environment, the reconstruction took roughly two hours.
The identical agent tackling the identical problem. Four nights transformed into two hours solely because the feedback mechanism changed.
An agent is only as good as its verification loop, and a verification loop is only as good as what it runs against.
Swami Sivasubramanian
Sivasubramanian emphasizes that validating distributed systems presents substantial challenges. This observation aligns with patterns observed across organizations deploying coding agents on live systems. The verification mechanism determines agent effectiveness, and the infrastructure it operates against determines the verification mechanism's value. AWS Emulator addresses this for AWS API interactions, yet it cannot address the services internal to your organization—which is precisely where agent-generated modifications typically fail.
AWS puts infrastructure inside the agent's loop
Coding agents operate through iterative cycles: generating code, executing it, analyzing results, and determining necessary modifications. Sivasubramanian's insight highlights that within distributed systems, this cycle operates within infrastructure. When the services receiving requests behave differently from production versions, the agent draws incorrect conclusions and perpetuates those mistakes.
AWS Emulator brings AWS services into this cycle on a developer's laptop. By pointing existing AWS tooling and code toward it using a single environment variable, tests execute against simulated services without requiring modifications.
Agents authored every component of the emulator, drawing from AWS's publicly available API specifications. Each capability is validated through fidelity tests that succeed against actual AWS first, then against the emulator, with all modifications passing a review gate where security-focused agents examine them. Sivasubramanian's takeaway is that when agents produce the code, engineers focus on establishing the specifications, tests, and review gates that determine correctness.
Initial commentary on Hacker News immediately raised concerns about trustworthiness. One user tallied approximately 30,000 commits across three weeks and questioned the resulting quality. This represents a legitimate concern directed at the wrong target. The authorship of code provides minimal insight. The infrastructure validating it provides substantial insight, and AWS's answer is actual AWS.
Slow or unrealistic feedback was a tax on people. For agents, it decides whether the loop works at all.
Swami Sivasubramanian
The shift involves who executes the loop. A human developer reviews modifications several times daily, while an agent evaluates them during each iteration. For humans, sluggish or inaccurate feedback represented a burden. For agents, it determines whether the feedback mechanism functions.
Every loop runs against mocks, an emulator, or the running system
Agents can validate their output through three mechanisms, each presenting distinct limitations.
Mocks represent the least reliable approach, though artificial intelligence has made them economical to produce. A Hacker News participant contended that generating AI-powered mocks now occurs faster than manual creation, noting that "AI has been a boon for test coverage." However, an agent generating its own mocks constructs them according to its own logic. Tests succeed because they were designed to succeed, resulting in expanded coverage without corresponding confidence gains.
Emulators represent meaningful progress. AWS Emulator undergoes validation against genuine AWS, meaning its operations derive from the actual service rather than agent assumptions.
An emulator remains a representation, however, and AWS acknowledges this transparently. The project repository maintains a directory documenting behaviors agents discovered that diverge from actual AWS, and unimplemented operations return errors instead of simulated success. An IDC analyst observed that the emulator currently lacks simulation of rate limiting, service disruptions, or regional availability variations—the scenarios where retry and failover mechanisms get tested.
An agent that writes its own mocks writes them to match its own assumptions. The tests pass because they were built to pass, so coverage goes up while confidence should not.
Sivasubramanian reflects similarly on his own Alf work. Simulation resolved most concurrency issues before deployment to actual servers. Testing on actual servers subsequently revealed different problems, because "the real world exposes things a simulation cannot."

You can't emulate your own system the way AWS emulated its own
Within the same Hacker News conversation, someone identifying as an AWS employee described the strategy most organizations employ with agents currently: isolate as much of the system as feasible, mock or simulate unavoidable dependencies, then enable the agent to proceed. This succeeds when the isolated portion is a cloud service with a documented specification. It fails when that portion is your organization's proprietary system.
AWS Emulator functions because AWS possessed two assets most organizations will never hold for their own systems: a thorough, machine-parseable specification of every operation and a stable reference point for validation in actual AWS.
Consider a payment system containing 60 microservices. Service contracts exist scattered across source code, several OpenAPI documents that diverged weeks prior, and engineer memory. No unified specification exists, and the reference implementation is the operational system itself, which transforms with each code merge. With agents submitting pull requests multiple times daily, it transforms faster than any representation could track.
This is also where agent-generated application code typically encounters problems. Failures seldom involve how a managed queue manages message visibility. They involve a field renamed in one service that a downstream consumer two teams away still accesses, an event structure a dependent service cannot process, a query performing acceptably on test data but slowly on production-scale information, or configuration, feature toggles and credentials existing solely in the cluster. No cloud provider emulator will ever encounter these.
The environment belongs on the engineer's list
Sivasubramanian identifies specifications, tests, and review gates as engineering responsibilities in this model. I would include the environment, since it determines what the agent can identify.
A realistic loop exhibits six characteristics. The modification executes in an environment resembling production, alongside the services, information, and settings it will eventually enter. Each modification receives its own separate environment, preventing 10 parallel agents from interfering with one another. The environment initializes in seconds, because an agent waiting 20 minutes abandons it and the loop reverts to unit testing. And the agent establishes the environment and retrieves results independently, without requiring manual intervention.
The loop additionally executes the integration and end-to-end testing suites the organization already depends on, not tests the agent created to validate itself. And it generates documentation a reviewer or review gate can evaluate before merging.
The challenging aspect involves achieving production-like behavior while maintaining rapid initialization. Duplicating an entire stack for each modification achieves behavior fidelity while sacrificing speed and expense. A group of 10 engineers executing multiple concurrent agent sessions might generate 100+ pull requests daily requiring verification. No organization can absorb the expense of 100+ complete system replicas.
Emulate what you don't own, and run what you own for real
Emulators and production-like environments represent complementary rather than competing approaches. They constitute layers within a unified mechanism.
For managed cloud offerings with documented interfaces, an emulator supplies agents with rapid, costless validation on a developer machine. When code writes to S3 and reads from a queue, AWS Emulator identifies API errors in seconds without consuming AWS resources.
Emulators and production-like environments are not competing ideas. They are layers in the same loop.
For your proprietary system, deploy exclusively the components the modification changes rather than replicating the entire architecture. Traffic and messages designated for that modification's environment reach the updated versions, while everything else originates from the persistent stable environment already operational. Data stores and messaging systems require lightweight per-environment divisions only where the modification needs data separation. This approach enables environments to start in seconds while maintaining production behavior. Uber and Lyft constructed internal infrastructure using this pattern for identical reasons.

Subsequently, require the review gate to validate both. A pull request that succeeded against an emulator and against the system it joins represents something a reviewer can have confidence in.
Every agent will have a loop, and realism is what separates them
Sivasubramanian's agent transitioned from four nights to two hours once it gained the capacity to validate its own output. For application organizations, the subsequent advancement hinges on what it validates against. An emulator handles the cloud services your code depends on, and only the operational system handles the services you maintain.
Possessing a verification mechanism is now standard. The fidelity of that mechanism determines how much confidence you can place in the resulting code. For organizations wanting agents to validate every modification in a production-like environment before merging, Signadot was constructed to establish that as the standard approach.