OpenAI's Astra Shows How the Software Wrapper Around AI Models Now Matters More Than the Model Itself

When ARC Prize tested GPT-6 Astra using its own standard harness, the model achieved a 62.7% score on ARC-AGI-3. The identical model, running through OpenAI's Provider Adapter, reached 98.6%. The model remained unchanged; only the software infrastructure surrounding it differed. That 36-point swing also came with a cost advantage: $17,332 using OpenAI's adapter versus $26,098 with ARC Prize's harness.

Investor Matt Turck, whose podcast has featured the architects behind ARC-AGI, captured the significance tersely: "This is wild," he wrote, then added in parentheses: "w/ its native harness." The observation underscores a fundamental shift in how AI systems perform: the engineering layer surrounding a model can be as consequential as the model's underlying capabilities.

Beyond raw performance, the efficiency gains were striking. Across 167 game-reasoning pairs that both harnesses successfully solved, OpenAI's Provider Adapter consumed 49% fewer tokens and executed roughly 3.66 times faster than ARC Prize's standard setup. The harness, not the reasoning dial, delivered the decisive advantage.

Benchmarks measure systems, not isolated models

ARC-AGI was intentionally designed to resist the brute-force scaling that undermines many benchmarks. This year's ARC-AGI-3 iteration raised difficulty further by placing models in interactive environments without instructions, explicit goals, or stated rules, then measuring how efficiently they learn to operate. When ARC Prize launched the benchmark, humans scored 100%; Frontier AI scored 0.51%.

The two harnesses differed in crucial ways. ARC Prize's standard harness allows models to carry forward notes they choose to retain. OpenAI's adapter preserves the opaque reasoning state between requests and compresses longer conversations, enabling the model to resume its own thinking rather than reconstructing context from scratch.

The performance spread across reasoning levels was dramatic. Astra inside OpenAI's harness with no reasoning effort scored 96.7% for $23,457. The same model at maximum reasoning within ARC Prize's standard harness scored 62.7% and cost $26,098. The harness outperformed the reasoning setting entirely.

OpenAI has not obscured how its system works—the adapter relies on documented Responses API capabilities that anyone can invoke. However, the assembled system that produced the 98.6% result is not available for purchase as a unified product. This distinction matters when evaluating claims about AI progress. OpenAI President Greg Brockman stated during a press briefing that "I think it's not unreasonable to feel that we are now in the AGI era," but this assertion describes a benchmark result from a particular system configuration, not proof that the underlying model itself constitutes AGI. The evidence supports conclusions about system performance, not necessarily about the model's fundamental capabilities.

The harness has become the product itself

On coding tasks, frontier models now cluster within a narrow performance band. Artificial Analysis scores its Coding Agent Index by running each model within a harness rather than standalone: Astra in Codex at 67, Opus 5 and Fable 5 in Claude Code at roughly the same level, Muse Spark 1.3 in Muse Code alongside them, and Fable 5.1 in Claude Code leading at 70. The unit being measured is already the combined pair of model and harness.

The major AI labs recognized this shift months ago. In April, documentation showed that Anthropic, OpenAI, Google, and Microsoft all treat the harness as a sellable product, disagreeing only on pricing models. Anthropic meters Managed Agents at $0.08 per session hour, layered on top of token rates. OpenAI released its Agents SDK without any runtime fee. Google and Microsoft bill sessions, memory, code execution, and observability as separate line items. None treat the harness as a free accessory bundled with the model.

This pattern extends beyond the major labs. Stripe acquired OpenRouter in August for a reported $8 billion, gaining control of a gateway that routes 10 trillion tokens daily across more than 400 models for 10 million developers. Patrick Collison framed the acquisition around tokens as the central currency for AI-building companies. Stripe purchased the routing layer positioned in front of the models themselves. Nvidia has built its own harness layer as well.

Nvidia's work with Claude Opus 5 illustrates the point. On its own, Opus 5 achieved 30.2% on ARC-AGI-3's public set. Wrapped in Nvidia's AVO, which provides persistent memory and programmatic supervision that intervenes when progress stalls, the system cleared all 183 levels across 25 environments. Nvidia stated the principle directly: "Model capability matters enormously, but the surrounding system determines how effectively that capability can be converted into sustained autonomous progress."

Harness engineering is now the core skill

Token usage varies dramatically depending on the harness. Testing an identical model across Aider, Claude Code, and OpenClaw revealed a 70-fold spread in token consumption. Cache hit rates ranged from approximately 70% down to 1.5% depending on the serving path. No model selection alone accounts for variation of that magnitude.

The engineering work itself involves familiar decisions: determining what an agent remembers and what it discards, what it can access and when it must pause for human approval. The practical details of building these systems are increasingly the focus of engineering effort.

Model selection was once the critical hiring skill. That assessment requires revision. Choosing the right model represents the simpler half of the problem, and it becomes easier each quarter as frontier models converge in capability. Within a year, teams operating AI agents will likely spend less time selecting models and more time engineering the systems that surround them.

Source: The New Stack