Top AI coding agent stumbles on private codebases, Real-SWE benchmark reveals

The newly released Real-SWE benchmark from Y Combinator-backed Specific Labs takes a markedly different approach to evaluating coding agents than existing tests. Rather than presenting models with problems sourced from publicly available repositories, the benchmark places them inside private codebases belonging to actual businesses and asks them to solve engineering challenges comparable to those encountered in production environments.

Claude Fable 5.1 emerged as the top performer on Real-SWE with a score of 38.8%, despite failing more than six out of every 10 tasks. The benchmark's use of proprietary code and solutions—which remain unavailable to the public—makes it substantially less probable that models encountered this material during training. Specific Labs estimates that 99% of tokens in real-world enterprise systems remain hidden from frontier models, though the company acknowledges it cannot definitively rule out prior exposure to any given code snippet.

Performance rankings on private code

Behind Fable 5.1's 38.8%, the leaderboard showed GPT-6 Astra at 33.8% and Gemini 3.8 Flash at 31.2%. Performance dropped more steeply after that tier, with GLM 5.3 scoring 28.8%, Grok 4.6 and Muse Spark 1.3 both at 23.8%, Kimi K3 at 18.8%, and GPT-5.6 Sol at 16.2%.

Each model received eight attempts per task. The benchmark tested each model using its native coding tool—Fable 5.1 with Claude Code, Astra with Codex CLI, and Gemini 3.8 Flash with Gemini CLI—meaning the reported scores reflect the complete integrated system rather than the model in isolation. This setup choice matters: as demonstrated by GPT-6 Astra's performance on ARC-AGI, the scaffolding surrounding a model can meaningfully affect its results.

Six tasks defeated all models

Real-SWE draws on proprietary code licensed from operating businesses, including a consumer application serving more than 200,000 users and a fintech system that has processed over 100,000 bank statements. The benchmark's scope extends across entire codebases, with solutions typically touching a median of 11 files—nearly double the six-file median found in comparable benchmarks like FrontierCode and DeepSWE.

When examined at the individual task level, success rates plummeted further. Six of the ten tasks posted success rates below 15%. A billing schedule migration achieved a 14.1% fix rate, API token metering reached 12.5%, S3 storage tracking hit 10.9%, and a linearizable scan came in at 4.7%. A tax jurisdiction bug was patched just 3.1% of the time.

Not a single model solved the analytics stream reducer across 64 attempts.

Conversely, Astra and Gemini both succeeded on a multi-region sweep in all eight attempts, and Fable solved seven of eight—yet all three models failed every attempt at the linearizable scan. No agent demonstrated consistent reliability across the full benchmark.

Where agents failed most often

Fable 5.1's failures stemmed most frequently from missed requirements (36.7%) or integration errors (34.7%). Astra's failures split evenly between integration errors and unverified assumptions, each accounting for 34%. Integration errors appeared in nearly half of Gemini 3.8 Flash's failed runs, while GPT-5.6 Sol made unverified assumptions in 43.3% of its failures.

What the numbers tell us

Real-SWE does not definitively establish that public coding benchmarks suffer from data contamination inflation, and a sample of 10 tasks remains modest. However, the leading agent's failure rate exceeding 60% on private code it almost certainly had not encountered before suggests that successfully solving an isolated coding problem differs fundamentally from navigating an unfamiliar production codebase.