DeepSeek's Coding Agent Stumbles on Benchmarks Due to Overzealous Cleanup

An open-source coding agent from DeepSeek is discovering that good housekeeping can come at a cost. The system loses ground on a major benchmark specifically because it performs a task-completion routine that software engineers typically consider sound practice: shutting down background processes when work concludes.

The issue emerged when Joël Niklaus, an optimization researcher at Hugging Face, integrated DeepSeek Harness into his leaderboard for coding agents. The board evaluates ten different harnesses running identical underlying models. While DeepSeek Harness performs competitively across most metrics, it falters dramatically on Terminal-Bench, where a single built-in behavior causes it to fail assignments that rival agents handle without difficulty.

A tidy habit backfires

The culprit is straightforward: DeepSeek Harness terminates background processes upon task completion. This creates problems for three Terminal-Bench tasks that require those processes to remain active. Testing revealed that DeepSeek Harness succeeded in only two of 36 attempts using DeepSeek-V4.1-Flash and GLM-5.3-Flash, whereas OpenCode, Pi, and Nous Research's Hermes Agent completed all 36 on the identical tasks.

DeepSeek Harness passed just two of 36 attempts … while OpenCode, Pi and Nous Research's Hermes Agent passed all 36 on those same tasks.

Joël Niklaus

This shutdown behavior reduced DeepSeek Harness's scores by between 1.5 and 1.7 percentage points. On DeepSeek-V4.1-Flash operating at high reasoning effort, this penalty alone exceeded the entire 1.3-point gap separating DeepSeek Harness from the leading harness.

DeepSeek's own technical documentation characterizes its local job registry as intentionally ephemeral. The documentation states: "Choose it when jobs should live in the harness process and die with it." Jobs requiring persistence across restarts or execution across multiple processes need a different backend configuration.

Same model, different results

Niklaus launched his Harness Leaderboard on November 5 following 28,849 trials distributed across nine original harnesses using three open models: DeepSeek-V4.1-Flash, GLM-5.3-Flash, and Flint. By holding the model, provider, and reasoning effort constant while varying only the harness, he uncovered striking performance disparities. Mini-swe-agent, a relatively straightforward bash-based agent, achieved a 51.4% task completion rate compared with 27.3% for OpenClaw, though several other harnesses performed at statistically equivalent levels to the top performer.

Economic efficiency showed equally dramatic variation. Pi delivered a 49.4% success rate at $0.29 per solved task, while OpenCode required $3.35 per solved task in the initial round of testing. Thomas Wolf, co-founder of Hugging Face, summarized the implications on X: "same open model, different harness => very different results."

Orchestration isn't a guarantee

Y Combinator President and CEO Garry Tan conducted parallel testing using Claude Opus 5.5 and GPT-6.1 Sol. His findings reinforced the leaderboard's central insight: harness selection matters enormously. Researchers observed that harnesses diverged more substantially on expense than on task success rates. When running Opus 5.5, OpenClaw incurred roughly double the per-task cost of other Opus harnesses, primarily because it repeatedly transmitted context alongside each query.

Tan's team also evaluated performance on workflow tasks. Across 46 sessions, Opus 5.5 read the workflow instructions but never executed the prescribed sequence, and these attempts consumed roughly twice the resources of successful runs.

Researchers cautioned that the evaluated tasks were relatively straightforward, leaving open the question of whether instruction-following would have produced better outcomes on harder problems. The findings underscore a broader principle: harness selection deserves as rigorous evaluation as model selection itself.

The results make a case for testing the harness just as carefully as the model.