Testing Claude Fable 5.1 Against Its Predecessor: Real-World Performance Diverges From Benchmark Claims

Anthropic unveiled Claude Fable 5.1 this month with emphasis on a single benchmark achievement: its Terminal-Bench-Science score jumped to 52.6%, more than doubling Fable 5's 24.7%. However, when tested under conditions resembling typical user constraints, the performance gap narrows considerably.

The Terminal-Bench-Science benchmark presents models with a terminal and genuine scientific research problems to solve independently. Anthropic's published results were generated under laboratory conditions unavailable to most users: each model received up to eight hours per task, though the company has not disclosed the specific harness or budget employed. When the benchmark's public leaderboard evaluated Fable 5, it consumed $14,180 across 210 attempts using Claude Code at maximum effort, averaging $67 per attempt.

To understand how these models perform in realistic scenarios, five tasks from the benchmark's 70-task suite were selected—one from each scientific discipline—and run under modest constraints: a $12 spending limit and 60 turns per test. Both models operated in a plain terminal environment, and the complete testing cycle required approximately 12 hours.

The tests

Terminal-Bench-Science organizes its tasks across five scientific categories. The selected tests were:

• Symbolic regression (mathematics) – Identifying a hidden formula from a dataset containing 100 variables and a binary label, then developing a predictor that generalizes to unseen data
• Lorenz-96 assimilation (Earth sciences) – Reconstructing a chaotic atmospheric model using sparse, uncalibrated sensor readings with unknown temporal offsets; grading requires meeting all five criteria
• Reactor safety control (engineering) – Designing a controller for a chemical reactor that maximizes batch completion speed while maintaining temperature constraints across known and hidden fault scenarios
• Foraging cognitive model (life sciences) – Predicting lever selection by individual mice across trials, evaluated on previously unseen sessions
• Nanoindentation (physical sciences) – Extracting material characteristics from raw indentation curves containing drift, adhesion issues, defects, and unknown tip geometry

Symbolic regression

This represented the sole test where either model successfully cleared the benchmark's hidden test set. Fable 5.1 completed the task in 27 turns, uncovered the underlying structure, generated a working predictor, and terminated independently after 11.8 minutes, 27,088 output tokens, and $1.96 in costs.

Fable 5 exhausted all 60 available turns across 53.5 minutes, producing 39,461 output tokens at a cost of $4.20, without achieving success. A second attempt with Fable 5 consumed the full 60 turns again, required 60 minutes, generated 60,608 output tokens, cost $6.38, and also failed.

Lorenz-96 assimilation

This pair of runs proved the most expensive. Fable 5 reached the $12 spending ceiling at 45 turns following 97.7 minutes and 92,091 output tokens, finishing at $12.63. Fable 5.1 consumed all 60 turns over 126 minutes, generated 89,789 output tokens, and cost $10.70. Both models failed to solve the task; notably, Earth sciences is also the discipline where Fable 5 scores near zero on the public leaderboard.

Reactor safety control

Neither model produced a controller that satisfied the grader's test scenarios. Fable 5.1 generated the highest token count of any run, 157,710 tokens, before hitting the 60-turn limit after 40.9 minutes and $11.53 in spending. Fable 5 reached its cost limit at 49 turns after 63.8 minutes, having produced 121,978 output tokens and $12.04 in charges.

Foraging cognitive model

This test represented the longest execution in the series. Fable 5.1 was the only model to declare task completion. It constructed a model, validated it against its own scoring mechanism, and announced completion at 43 turns after 53.5 minutes, generating 65,518 output tokens and costing $5.65. The official grader rejected the submission.

Fable 5 never signaled completion. It reached the $12 limit at 60 turns after 139.3 minutes and 62,587 output tokens, totaling $12.13.

Nanoindentation

Both models failed this task. Each spent the majority of execution time reading raw curves and writing segmentation code. Neither generated a results file acceptable to the grader. Fable 5.1 exhausted its turn allocation at 29.6 minutes, 115,687 output tokens, and $10.91. Fable 5 depleted its budget at 48 turns after 34.1 minutes and 114,239 output tokens, ending at $12.59.

Results

The benchmark evaluates models across all 70 tasks with three trials each. Anthropic's published figures of 24.7% and 52.6% derive from this comprehensive assessment. The independent leaderboard places Fable 5 at 21.4%, closely matching Anthropic's result. Fable 5.1 does not yet appear on the independent leaderboard, making Anthropic's 52.6% the sole published score. This independent testing yielded 0% and 20%, falling below both published figures. However, five tasks constitute a limited sample.

The probability of obtaining these results by chance remains plausible even if the published scores are precisely accurate, so this evaluation neither validates nor refutes the doubling claim. The directional trend aligned, as the newer model demonstrated superior performance. The single task solved by Fable 5.1 fell within mathematics, the discipline where the leaderboard indicates Fable 5 achieves its strongest results.

What I think

The practical distinction between Fable 5 and Fable 5.1 for typical users appears minimal. This limited test sample cannot definitively prove or disprove Anthropic's benchmark claims, yet the evidence suggests the performance gap will not materialize for average users. One measurable difference emerged in operational costs: Fable 5.1 failed more quickly and economically, never exceeding the spending cap, whereas Fable 5 hit the limit four times.

If your tasks resemble the benchmark problems, the testing harness and budget matter equally as much as the model itself. With a specialized harness, extended time per task, and substantially larger budgets, results may approach Anthropic's published numbers.

Source: The New Stack