AWS Continuum Achieves 89% on Security Benchmark, But Context Matters

On Thursday, AWS rolled out Continuum to general availability, a tool that identifies code vulnerabilities in repositories, validates them against AWS infrastructure, and generates pull requests containing remediation proposals. The launch followed AWS's publication of benchmark results on October 5, showing the system achieved an 89.0% end-to-end success rate on CyberGym-E2E, surpassing the previous public benchmark high of 65.9%.

The 89% benchmark claim

Continuum passed 819 of 920 tasks on CyberGym-E2E, according to AWS's testing. Matt Garman, CEO of AWS, stated on X that AI is "genuinely transformative for defenders" and that Continuum scored higher than any other AI agent on the test. The announcement came more than two weeks before the product's general availability release. Corey Quinn, chief cloud economist and co-founder of Duckbill, questioned on X whether the launch timing suggested the announcement had been accelerated from AWS's re:Invent conference, though he provided no supporting evidence for this speculation.

The recent PHP2Shell case demonstrates a broader challenge: while AI systems are becoming more effective at discovering vulnerabilities, the process of fixing them remains considerably more complex.

How CyberGym-E2E works

The benchmark allocates 90 minutes for an agent to locate and repair a vulnerability across 139 open-source projects. Agents receive no bug descriptions, proofs of concept, crash logs, or existing patches. The task requires identifying the flaw, reproducing it, and delivering a fix that halts the crash while preserving all existing tests. AWS notes that all tasks involve memory-safety bugs in C or C++, excluding numerous other programming languages and vulnerability categories.

The evaluation proceeds through four cumulative stages. Continuum identified and reproduced bugs in 92.5% of tasks, prevented crashes in 89.6%, and maintained passing tests in 89.0%. The final stage determines whether the patch also resolved the specific historical vulnerability the benchmark had selected. Since the agent never learns which historical vulnerability it should target, it only needs to discover and fix any genuine flaw in the codebase. Continuum's patches addressed the historical bug in 37.8% of tasks, compared to the previous public benchmark record of 26.2%.

The historical-bug score gap

AWS offers a reasonable explanation for the disparity between the 89% and 37.8% figures. Repositories frequently contain multiple vulnerabilities, and fixing a different one than the benchmark's target does not negate the value of the work performed. Continuum completed 89% of tasks by identifying and patching a genuine vulnerability without breaking existing tests, yet only 37.8% of those patches also fixed the specific historical bug the benchmark measured. Since the agent had no information about which bug it should target, the comparison is not entirely straightforward. The remaining question is how effectively Continuum performs when teams direct it toward a specific, identified vulnerability.

Who graded the benchmark

AWS conducted the testing internally and confirms that Continuum adhered to the benchmark's requirements, including restrictions on internet connectivity. The company verified that the agent was not simply retrieving publicly available fixes. Continuum achieved the 89% success rate within the 90-minute constraint. When given additional time, AWS reports, performance climbs to 93.7%. Although the benchmark and testing infrastructure are publicly available, the results published so far come exclusively from AWS's own testing.