AWS Lambda's eBPF and Rust Pipeline Captures Network Flows at Scale Without Packet Loss

When security incidents occur on any compute platform, investigators face the same urgent question: which workload connected to that endpoint, at what time, and how much data moved? The challenge intensifies on Lambda, where thousands of microVMs run for mere hundreds of milliseconds before terminating, leaving only the logs captured during those brief windows as evidence of what occurred.

Lambda's previous network flow logging system, inherited from an earlier single-tenant EC2 design, buckled under the density demands of the modern multi-tenant Firecracker microVM architecture. The system relied on kernel-side packet counting matched to tenants and sandboxes, paired with a userspace daemon that batched records and uploaded them. This approach functioned adequately with modest VM counts but collapsed for two critical reasons: rule explosion in iptables created linear performance degradation as host density increased, and the system lacked IPv6 support entirely. "A record that can't see half the address space isn't one you can trust, and the moment dual-stack IPv6 support was proposed for Lambda, the old approach was finished."

Architectural Requirements

The replacement system had to satisfy three non-negotiable constraints: correct attribution of traffic to specific microVMs and tenants, meaningful reduction in computational overhead, and first-class IPv6 support. An additional requirement emerged from operational reality: the new system needed to emit Amazon Ion records byte-for-byte identical to the legacy format, allowing the entire downstream ecosystem of flow-log and metering systems to continue functioning without modification.

The Three-Layer Design

The new system comprises three cooperating components working in concert. At the kernel level sits a set of small eBPF programs attached to traffic-control hooks on each microVM's virtual network devices. These programs intercept packets and emit compact events into ring buffers—they observe only, with no capability to copy, block, drop, or rewrite traffic. The middle layer consists of a Rust-based tagger process running unprivileged, one per network. It drains the ring buffer, aggregates per-packet events into per-flow records, and writes them to disk in the legacy Amazon Ion format. At the top sits a single privileged orchestrator process per host, responsible for loading eBPF programs, configuring traffic control, spawning and supervising the fleet of taggers, and exposing a lifecycle API over Unix domain sockets.

ONE WORKER HOST

   control plane
       |
       |  gRPC over a Unix domain socket
       v
   orchestrator ....... one privileged process per host
       |                (loads eBPF, configures TC,
       |                 spawns one tagger per network)
       v
   --- per network (one microVM) ------------------------------

   eBPF capture  -->  ring buffer  -->  tagger  -->  Ion records
   TC hooks on        one per           Rust,
   the network's      network           userspace
   devices

   ------------------------------------------------------------
       |
       v
   same billing + flow-log pipeline as before

Kernel Capture Without Interference

The eBPF programs attach to the clsact qdisc in traffic control on both ingress and egress sides of each network's devices—four attachment points per network. Traffic control provides an ideal vantage point: it sees every packet early, before downstream processing. Each program reads the packet and returns the "keep going" action, ensuring no packet modification or latency impact.

For each packet, the program parses headers—Ethernet, then IPv4 or IPv6, then TCP, UDP, or ICMP—and writes a fixed-size event into the network's BPF ring buffer. The event is deliberately small, approximately 24 bytes for IPv4:

/* one event per packet, ~24 bytes for IPv4 */
 struct flow_event {
     u8  ip_version;       /* 4 or 6 */
     u8  protocol;         /* TCP / UDP / ICMP */
     u8  direction;        /* ingress or egress */
     u8  device_id;        /* which of the network's devices */
     u16 local_port;       /* "local" is always the sandbox side */
     u16 remote_port;
     u32 flags_and_bytes;  /* TCP flags in bits [31:24], byte count in [23:0] */
     u32 local_addr;       /* 16 bytes for IPv6 */
     u32 remote_addr;
     u32 received_time_ms;
 };

The design counts bytes, not packets, with TCP flags and byte count sharing a single 32-bit word. "Doing less work per packet in the kernel is the entire point, so aggregation is somebody else's job." Local and remote fields are normalized by direction before leaving the kernel, ensuring userspace never reasons about direction when grouping flows.

The eBPF programs use a reserve-then-commit pattern with the ring buffer, avoiding data copying through syscalls and minimizing CPU consumption. However, passing the eBPF verifier—which must prove program safety before kernel loading—required careful design decisions. The team made the header parser a shared subroutine to avoid re-verification at each attachment point, bounded IPv6 extension-header walks to a fixed number of hops, ensured packet fragments report zero ports and flags rather than garbage, and handled byte counts correctly for coalesced packets from GSO/GRO. Beyond the verifier, each eBPF program runs through formal model checking with CBMC during every build, with harnesses asserting that the event struct's byte layout remains compatible across all attached programs.

Ring Buffer Sizing from First Principles

The ring buffer represents the single shared resource between kernel producer and userspace consumer, and its size involves real tradeoffs. Rather than guessing, the team derived the floor from each microVM's packet rate. For a ceiling of 100,000 packets per second per direction, draining roughly every 100 milliseconds:

ring bytes =~ 62,500 pps x 0.1 s x ~24 bytes x 2 directions
            =~ 300 KB

The ring buffer API requires a power of two, so the design defaults to 512 KiB—the smallest buffer that cannot overflow between drains at the guest's maximum packet rate. The running deployment currently provisions more generously, on the order of a couple of megabytes, while tuning the right per-workload value. The kernel checks how full the ring is and only forces a wakeup once it crosses about one percent full, allowing quiet flows to accumulate events essentially free while busy flows trigger immediate reads.

Aggregation in Rust

The tagger transforms raw per-packet events into per-flow records for the pipeline. Rust was selected for practical reasons: at this density, thousands of these processes run on a single host, each holding small amounts of state that must remain correct. Garbage-collected runtimes would introduce pause times and memory bloat under load, potentially creating gaps in the record. Rust provides predictable memory with no collector, plus compiler guarantees against entire categories of bugs that could cause misattribution. Each tagger runs in a few hundred kilobytes of RAM against a roughly one-megabyte budget, making thousands per host affordable.

Internally, the tagger runs a small set of cooperating tasks on a single-threaded async runtime. One task reads the ring, another manages flow state, and a third writes parcels. Only operations that genuinely block—receiving the ring descriptor and serializing Ion—run on a blocking pool. The tagger reads the ring through epoll, sleeping when idle and waking when events arrive.

As events arrive, the tagger drops them into a flow map keyed by device, the five-tuple, and a tenant attribution handle provided by the control plane. Matching events accumulate bytes, packet counts, and OR'd TCP flags. The kernel event carries no identity because every network has its own dedicated ring and devices—packets are separated long before the tagger sees them. The stream the tagger reads belonged only ever to that one tenant.

Once per minute, aligned to the top of the second to match the legacy system, the tagger serializes completed flows into Amazon Ion records using the exact schema the old daemon produced. Each file is written to a temporary name, flushed to disk, and renamed into place, ensuring readers see complete records or nothing. A separate flush loop with random jitter at startup drains completed flows even after a microVM goes quiet.

Privilege Isolation Through File Descriptors

The thousands of taggers that perform actual packet work hold no elevated privileges. They cannot load eBPF, touch traffic control, or even open the ring buffer map independently. All such power resides in one place: the per-host orchestrator, which runs with only the two capabilities it needs rather than as root.

An unprivileged tagger reads its ring buffer through a file descriptor opened by the orchestrator and passed over a Unix domain socket using the kernel's SCM_RIGHTS mechanism. "Passing a file descriptor over a socket is a decades-old Unix feature, and it lets us keep thousands of processes powerless while concentrating privilege in one small place." The tagger receives a ready-to-use handle and nothing else, never needing permission to create one. The privileged surface of the entire system is one small process per host, while thousands of processes touching customer traffic remain as powerless as possible.

Lifecycle Management

MicroVMs arrive and depart constantly, requiring the control plane to signal the orchestrator when to start and stop recording. This happens through gRPC APIs over a Unix domain socket, with methods to create flows, activate flows, recycle flows, tear flows down, and perform health checks.

Recording initiation splits into two calls deliberately. Create is expensive: it loads and attaches eBPF programs, configures traffic control, and spawns the tagger. Attaching to network devices requires a kernel lock that all such operations contend for, so the system batches these operations to prevent serialization. Activate is lighter—the machinery already exists, so it hands over customer metadata and transitions the flow to steady-state recording. Its latency budget is tight: under 2 milliseconds at p90, under 10 milliseconds at p99.9, matching the baseline of the replaced system.

Production Tradeoffs

The original design enforced a strict rule for network recycling: always destroy the tagger and spawn a fresh one. From a correctness perspective, the reasoning was sound—a brand-new process cannot carry stale metadata from a previous tenant, making cross-tenant flow contamination structurally impossible. However, production workloads revealed that forking and exec'ing new processes thousands of times as networks churned created significant CPU spikes at scale. The shipped system introduced a configurable knob allowing workloads that reuse networks to reuse the tagger after recycling, trading some structural guarantee for substantially less CPU churn. Workloads requiring strict cross-tenant-proof behavior can disable this option.

Measurable Improvements

The old design required a single host to maintain more than one hundred thousand firewall rules to record traffic for two thousand microVMs, with each additional microVM adding more rules and taxing every packet further. The eBPF version replaces this linear rule walk with constant-time map lookups whose cost does not increase as the host fills. This eliminates the linear tax and brings the density target—roughly double the microVMs per host—within reach.

The system delivers additional benefits aligned with its original constraints:

• IPv6 flows, invisible to the old tool, now get recorded like any other traffic, providing complete address space coverage instead of half
• Each tagger consumes a few hundred kilobytes of RAM against a roughly one-megabyte budget, making thousands per host practical
• Flow activation into steady-state recording remains under 2 milliseconds at p90 and under 10 milliseconds at p99.9
• The capture layer is observe-only and formally verified, while processes touching customer traffic hold no privileges, adding visibility while shrinking the trusted surface that could corrupt records
• Output records are byte-for-byte identical to the old format, allowing every downstream flow-log and metering consumer to continue functioning without modification

Lessons for Other Systems

These design principles extend beyond Lambda to Kubernetes pods, edge runtimes, and sandboxes for AI agents—anywhere multiple tenants share a host and require trustworthy traffic records.

• Observe from outside the hot path. When recording logic sits inline in packet forwarding, its cost taxes every packet, heaviest precisely when the record matters most. eBPF allows observation from the side with compact event emission while expensive work happens elsewhere
• Size buffers from concrete numbers. A buffer sized by actual rate limits times actual drain intervals produces a defensible number and enables promises of no dropped events under burst conditions
• Keep tenants apart at the point of capture. Giving each tenant its own ring and devices ensures streams never touch, allowing clean traffic labeling instead of post-hoc guessing
• Underestimate old primitives at your peril. File descriptor passing over sockets is decades-old Unix infrastructure that enables thousands of powerless processes while concentrating privilege in one small place
• Preserve compatibility when swapping engines. Byte-for-byte identical output enabled replacing the entire capture path with zero downstream migration and provided a record-for-record verification method

Source: The New Stack