Developers

Coding Agent Framework Choice Can Multiply Inference Costs Fivefold, Study Shows

A comprehensive benchmark of 22 harness configurations across five open-weight models reveals that the same AI model can cost vastly different amounts to operate depending on which coding agent framework wraps it.

3 min read
Your AI coding agent could cost 5x more depending on the harness, benchmark finds

Researchers at Runta and Baseten have documented dramatic cost variations in AI-powered coding agents, finding that identical models can incur expenses up to five times higher when paired with different harnesses. The investigation examined 22 distinct harness configurations running across five open-weight models, executing 27,280 test runs that consumed approximately 83 billion tokens.

The findings underscore a counterintuitive challenge for developers seeking to optimize spending: a harness that delivers the lowest cost with one model may rank among the most expensive options with another. "Nearly 65% of estimated inference spending went to runs that failed verification or timed out," the researchers found, highlighting how unsuccessful executions dominate the expense picture.

Harness Rankings Shift Dramatically Across Models

The benchmark tested 22 harness configurations against five models delivered through Baseten's infrastructure: GLM-5.3, GLM-5.3 Flash, DeepSeek V4.1 Flash, Muse Spark Open, and Kimi-K3. Each configuration processed 62 terminal-based tasks four times per model, with researchers tracking pass rates, inference expenses and execution duration.

Performance rankings proved unstable across different models. Codex CLI achieved the lowest average cost per task when running GLM-5.3 but plummeted to 16th position out of 22 when paired with Kimi-K3. DeepSeek Harness Creator exhibited even more dramatic swings, ranking first with Muse before dropping to 21st with GLM-5.3 Flash.

Across the five models, harnesses shifted a median of 11 ranking positions when the model changed, while costs varied by as much as fivefold even when the model stayed the same.

The most extreme disparity emerged with DeepSeek, where Cline CLI consumed five times the tokens of OpenCode v1 while achieving only a two percentage point improvement in pass rate. To verify that random fluctuation could account for these differences, researchers reshuffled trials within each model 300 times. The actual results showed substantially larger gaps than the randomized trials, with a p-value of 0.003.

Failed Executions Dominate Spending

Unsuccessful runs consumed the majority of inference expenditure across all configurations. Failed executions accounted for 52.7% of estimated spending, with timeout errors contributing an additional 11.8%. Only 35.5% of costs went toward tasks that completed successfully, based on Baseten's published pricing.

Among the five best-value harnesses, 68% to 82% of unsuccessful runs ended with the agent submitting a final answer that failed verification rather than timing out.

The researchers identified 216 instances of reward hacking in execution logs, with 163 involving online lookups. All such attempts received zero scores. This pattern illustrates the persistent difficulty coding agents face when tackling complex tasks.

Agent Steps, Not Prompt Size, Drive Token Consumption

Runta investigated which factors most strongly influenced per-task expenses: prompt size, tool count, cache hit rates, and agent steps. Step count emerged as the sole factor with statistically significant correlation to cost per successful task, at 0.51.

Mini-SWE-agent exemplified how multi-step agent execution inflates token usage. Despite starting with the benchmark's smallest prompt at 882 tokens, it averaged 38 steps per successful task and ranked 17th out of 22 configurations for cost efficiency.

The five most cost-effective harnesses shared a design pattern: heavy reliance on shell commands, which represented 61% to 93% of their tool calls, and minimal use of subagents or context summarization. Runta did not test these design choices in isolation, so the analysis cannot definitively establish whether simpler architectures caused the lower expenses or merely correlated with them.

Source: The New Stack · Reporting supplemented by The Silicon Ledger staff.