From Lambda to Kubernetes: Scaling Autonomous QA Agents Across Teams

Quality assurance teams managing complex products face a relentless stream of test failures. Each failure demands manual documentation: reproduction steps, screenshots, failure rates, and regression categorization. The process consumes 10 to 15 minutes per failure, and with 30 or more failures weekly across a team, an entire engineering day disappears into paperwork that should be automated.

One engineer built an autonomous agent to eliminate this work entirely. Rather than a dashboard or notification system, the agent observes test results, makes judgment calls about whether a failure is new, duplicate, or a regression, and takes action without human involvement. It files structured bugs, attaches screenshots, calculates reproduction rates, detects regressions across builds, and routes notifications to the right people. The team simply marks a test as FAIL and moves forward.

The system currently runs on AWS Lambda and works perfectly for a single team. But when other teams request the same capability, the architectural questions become complex. The challenge is not that Lambda fails, but that workload characteristics shift as the system scales, making different infrastructure choices more appropriate.

What I built (and why it works)

The architecture is straightforward: an EventBridge rule triggers a Lambda function every 15 minutes. The function polls a test management API, processes failures through a decision engine that checks for duplicates against DynamoDB and detects regressions by comparing against previous build results, then files bugs through the JIRA API. Total infrastructure cost runs under $5 per month.

Lambda proved to be the right choice for three reasons:

• Zero operational overhead: No servers to manage, no scaling configuration, no patching. Deploy once and forget about it.
• Cost proportional to usage: One team generates approximately 200 invocations monthly. At Lambda pricing, this is effectively free.
• Fast iteration: The decision engine underwent more than 20 revisions in the first month. Lambda's deployment model enabled shipping changes in seconds.

Within weeks, the system filed more complete and consistent bugs than the team had ever produced manually. No failures went undetected. Regressions were caught in minutes instead of days. QA status interruptions dropped to zero because anyone could query the system directly via Slack. The team saved 8 to 10 hours per week.

Other teams took notice. "Can we have that?" became a recurring question. The obvious answer seemed to be deploying another instance per team. But at scale, workload characteristics shift in ways that change the architectural calculus.

The scaling decision

Lambda handles single-tenant agentic workloads elegantly. The challenge at multi-tenant scale is not that Lambda cannot do it; rather, workload characteristics transform in fundamental ways:

From stateless events to long-running context: A single team's poll cycle completes in seconds. But cross-project correlation—asking whether a failure pattern appears across three teams—requires analyzing hundreds of results and their relationships in one pass. This represents a fundamentally different compute profile.

From isolated invocations to shared resource contention: When 50 functions trigger on the same EventBridge schedule, they all hit TestRail and JIRA APIs simultaneously. Centralized rate limiting, connection pooling, and backpressure become necessary—patterns that benefit from persistent, shared infrastructure.

From single-team to multi-tenant isolation: Each team requires guaranteed resources, network boundaries, and scoped credentials. The isolation model becomes as critical as the compute model itself.

Recent Lambda launches address many of these challenges individually. Managed Instances solve connection persistence. Durable Functions break the 15-minute ceiling for long-running workflows. Tenant Isolation provides native tenant boundaries. For many multi-tenant workloads, these capabilities narrow the gap significantly.

But for this specific workload—long-running agents requiring persistent connections, cross-tenant analysis, GPU inference, and namespace-level isolation simultaneously—the decision framework points toward a container-native execution model with serverless-like operations. That is where EKS Auto Mode fits.

The decision framework

This led to building a simple decision rubric—not Lambda versus containers, but which characteristics does the workload actually require?

Lambda with Managed Instances and Durable Functions handles stateless event processing, rapid iteration, and pay-per-invocation economics. Kubernetes with native namespaces, RBAC, and NetworkPolicies handles cross-tenant analysis requiring shared memory, centralized rate limiting with connection pooling, and namespace-level isolation.

The deciding factor for my workload: I need all of the right-column characteristics simultaneously. Together, they compose more naturally in a container-native environment.

Cross-team pattern detection requires a shared analysis context. ML-based bug classification requires burst GPU. Per-team isolation requires namespace-level boundaries. Centralized rate limiting requires a persistent shared service. Each of these is individually solvable in Lambda. Together, they compose more naturally in a container-native environment.

What unlocks at scale: cross-team intelligence

The most interesting architectural capability that emerges at multi-tenant scale is not per-team automation—it is cross-team pattern detection. This is what makes the scaling exercise worthwhile.

Cross-team failure correlation

When 50 teams run their test suites independently, failures appear isolated. But patterns emerge across teams that no single team can see:

• Three teams report navigation failures on the same firmware build, indicating a systemic regression rather than three independent bugs
• Test failures correlate with deployment timing across unrelated services, suggesting an infrastructure issue rather than product bugs
• Identical error signatures appear across teams with different test suites, pointing to a shared dependency failure

This requires an analysis engine that can hold context from all teams simultaneously—comparing fingerprints, correlating timestamps, and identifying shared root causes. That is a persistent, long-running workload with access to cross-tenant state. It runs naturally as a shared service in a platform namespace, consuming events from all tenant agents.

ML-based bug classification

Once enough historical data accumulates—thousands of bugs filed across teams—patterns become classifiable:

• Auto-assign bugs to the correct component owner based on failure signature
• Predict severity based on failure characteristics (crash versus degraded behavior versus cosmetic)
• Suggest related bugs for investigation based on similar root causes
• Detect anomalous failure rates before they become incident-scale problems

This inference workload is bursty—GPU capacity is needed when bugs arrive, not 24/7. A GPU NodePool that scales to zero when idle and provisions g5 instances only during inference keeps costs proportional to actual usage. The same pod that files a bug can synchronously call the classification service, get a component assignment, and include it in the ticket—all within the same request path.

Queryable knowledge base via Slack

The agent becomes more than an automation tool—it transforms into the organization's queryable QA state. Via Slack, anyone can ask:

• What is the failure rate for navigation this week across all teams?
• Show me regressions introduced in build 2.4.1
• Which teams are seeing the most P0 bugs right now?

This transforms the agent from a task automator into an intelligence layer. The answers require cross-team state aggregation, trend computation, and natural language interpretation—all served from persistent, indexed state that accumulates over time.

What EKS Auto Mode brings to this architecture

EKS Auto Mode launched at re:Invent 2024 with a specific promise: full Kubernetes without managing nodes. For a multi-tenant agent platform, this means:

• Nodes appear when pods need them. Deploy a tenant namespace with an auto-filer pod. The Karpenter-based scheduler provisions a node within seconds. When the pod is deleted, the node is consolidated or terminated automatically. EC2 instances never require manual attention.
• Security is not a configuration burden. Nodes run immutable Bottlerocket AMIs with read-only root filesystem, SELinux enforcing, and no SSH access. Nodes rotate every 21 days automatically. For long-running agent pods, this means a graceful drain roughly every three weeks, at most one missed poll cycle.
• Scaling is reactive to demand. During off-hours when agents are idle between polls, Auto Mode consolidates workloads onto fewer nodes. You pay for what you use.
• GPU workloads slot in without architectural changes. Define a GPU NodePool for the ML classification service. EKS Auto Mode provisions g5 instances only when inference pods are scheduled and terminates them when done.

The multi-tenant architecture

At scale, the system would have three layers:

1. Shared platform services: a rate limiter that prevents any single tenant from exhausting external API quotas, a connection pool manager that maintains persistent sessions to TestRail and JIRA, and a Prometheus instance collecting metrics across all tenants.
2. Per-tenant agent namespaces: each team gets an isolated namespace containing their auto-filer pod, reporter pod, and Slack bot pod. ResourceQuotas guarantee resources. NetworkPolicies enforce that Team A's pods cannot talk to Team B's pods. Each namespace's service account assumes a tenant-scoped IAM role via IRSA.
3. Shared ML inference namespace: GPU-enabled pods running classification models that all tenants can call. This namespace has its own NodePool that scales to zero when no inference is happening.

The agent code would remain essentially the same Python that runs on Lambda today—the core decision engine (observe, dedup, decide, act) does not change. What changes is the execution model: instead of being invoked per-event and destroyed, the agent runs as a long-lived async loop with persistent connections, graceful shutdown handling, and internal scheduling.

The AWS portfolio working together

The right framing is not "migrate off Lambda onto EKS." It is "use each service where it fits":

• Lambda remains the entry point. New teams start here. Single-team automation at $5 per month with zero ops. Event-driven triggers, rapid iteration, and pay-per-invocation economics.
• Lambda with Durable Functions handles teams that need longer analysis windows or multi-step orchestration without leaving the serverless model.
• EKS Auto Mode serves the platform layer when persistent multi-tenant services, cross-team intelligence, GPU inference, and namespace-level isolation are needed together.
• DynamoDB persists state across all layers—dedup records, bug history, trend data.
• EventBridge orchestrates events whether the consumer is a Lambda function or a Kubernetes pod.

A team that needs 30-minute analysis windows or multi-step workflows (poll → correlate → file → notify → wait for response → update) benefits from Durable Functions without leaving serverless. You stay here until you need cross-tenant shared state or GPU—that is the signal to evaluate Auto Mode.

Lambda validates the pattern. EKS Auto Mode operationalizes it at scale. They're stages in a journey, not competitors.

The intelligence moves up the stack as the organization's needs mature.

Key takeaways

• Start serverless: Lambda lets you validate the agentic pattern at zero operational cost. The $5 per month price point removes all friction for proving the concept works. Do not prematurely optimize for scale you do not have.
• Design for portability from day one: Structure your Lambda function as a pure decision engine with external state. Containerizing it means wrapping it in an async loop and adding a health endpoint. If you build Lambda functions as self-contained decision engines rather than tightly coupled to the invocation model, the transition is a one-day exercise whenever you need it.
• The decision trigger is not scale—it is workload characteristics: Do not move because you hit 50 teams. Move when you need capabilities that benefit from persistent, shared infrastructure: cross-team correlation, centralized rate limiting, GPU inference, streaming connections.
• Multi-tenancy is an isolation problem, not a compute problem: Both Lambda (Tenant Isolation) and Kubernetes (namespaces) solve this. Choose based on what other characteristics your workload needs alongside isolation.
• EKS Auto Mode closes the operational gap: Before Auto Mode, choosing EKS meant accepting node management overhead. Auto Mode removes that tradeoff. You get container-native execution with serverless-like operations.

The cost math

Single team on Lambda today: $3 to $5 per month. 8 to 10 hours per week saved. 67x ROI. Fifty teams on EKS Auto Mode would cost approximately $200 to $250 per month total—roughly $4 to $5 per team per month. Same per-team economics as Lambda, but with namespace isolation, persistent connections, GPU-ready infrastructure, centralized observability, and built-in organizational governance.

The cost-per-team barely changes. What changes is what you get for it.

What comes next?

The agentic pattern—observe, decide, act—is infrastructure-agnostic. The decision logic does not care about its deployment target. What changes is what becomes possible when agents stop operating in isolation.

The next frontier isn't single agents doing single tasks faster. It's agents coordinating with other agents.

An auto-filer agent detects a failure pattern and hands context to a root-cause agent, which traces the dependency chain across services, identifies the offending commit, and drafts a fix recommendation—all before a human sees the bug ticket. Agent-to-agent delegation, where one agent's output becomes another's input without human routing in between.

Beyond that: agents that build their own decision models. Today, the decision logic for dedup and regression detection is hand-coded. Tomorrow, the agent observes which of its filed bugs humans close as duplicates, learns from those corrections, and tightens its own classification boundaries. The system improves not because better code ships, but because it accumulates organizational judgment over time.

The system started as a weekend project to stop filing bugs manually. It now runs in production, eliminating an entire category of manual work from QA operations.

The infrastructure question stays the same—serverless for validation, containers for scale. But the agents running on that infrastructure are about to get significantly more capable. Teams with the platform layer ready will be the ones that can adopt those capabilities without rearchitecting.