← All posts

The coding harness argument is really an argument about defaults

There's a version of this post that would do very well and be slightly dishonest, so let me start with the number that makes it tempting.

On Artificial Analysis' Coding Agent Index, holding the model constant at Claude Opus 4.7 at medium effort, Claude Code scores 42 and opencode scores 51. Same model. Same 326 tasks. Same benchmark run.

Nine points is a lot on a composite index, and it isn't one benchmark carrying it. DeepSWE goes 27% to 40%. The repo Q&A set goes 23% to 36%. Only the terminal tasks are flat — 77 against 78. Two of three independent components move in the same direction, which is roughly as much as you can ask of a leaderboard that doesn't publish confidence intervals.

So: same brain, different body, nine points. The obvious conclusion is that we've all been holding it wrong, and that a Saturday morning and an npm install would buy the whole engineering org a fifth more capability for free.

That conclusion is wrong in an interesting way.

Read the rest of the row

The index score is the first column. The columns after it are cost, tokens and time, and they tell a different story.

Opus 4.7, same model          Index  DeepSWE  Terminal   Q&A   $/task  Tokens   Time
Claude Code, medium effort      42     27%      77%      23%   $1.80    4.6M    6.7m
opencode, medium effort         51     40%      78%      36%   $2.94    7.6M   12.5m
Claude Code, max effort         52     40%      78%      37%   $5.92   16.1M   15.8m

opencode's nine points came with 65% more tokens, 63% more spend and nearly double the wall-clock time. It didn't out-think Claude Code on those tasks. On the same model it can't. It out-spent it.

And then the third row ends the argument entirely. Take the harness everyone was about to uninstall, leave the model exactly where it was, and turn its effort dial from medium to max. Claude Code scores 52 — one point clear of opencode.

Both scaffolds reach the same ceiling on the same model. What separates them at their defaults is how much compute they're willing to spend getting there.

What a harness actually is

Strip away the branding and an agent harness is a loop with a budget. It decides how many times the model gets to read a file, run a test, notice the test failed, revise, and check itself before it commits to an answer. It decides how much of the repo goes into context and when to go looking for more. It decides when "I think this is right" is good enough to stop.

None of that is intelligence. It's a spending policy — expressed as a default, chosen by someone who wasn't looking at your codebase, and inherited wholesale by almost everyone who installs the thing.

That's why the harness comparison feels so decisive and explains so little. When two harnesses run the same model at different implied effort, you aren't comparing engineering quality. You're comparing two guesses about how much a task is worth.

Which is the same lever I wrote about a few weeks ago at the model layer, where a cost-optimised model at maximum reasoning effort climbed eighteen index points on a settings change. Same mechanism, one layer up the stack. Test-time compute is a control surface, and the industry keeps discovering it in a new place and treating it as a new product.

The part that is a real finding

There is still something in this data worth acting on, and it isn't "switch to opencode."

Look at what each configuration paid per point. opencode reached 51 on 7.6M tokens and $2.94. Claude Code needed 16.1M tokens and $5.92 to reach 52. Same model, same ceiling, roughly double the spend to get there.

That difference is real engineering. Context assembly, when to re-read versus re-reason, how aggressively to prune, how early to stop — those choices decide how much of your token spend converts into a correct answer instead of an expensive one. A harness that hits the ceiling for half the money is genuinely better built, even though the headline gap between the two default configurations mostly measured budget rather than skill.

So the useful comparison isn't index score. It's index score per dollar, on your tasks. Nobody's leaderboard can run that for you.

What this doesn't say

It doesn't say effort is free. Claude Code at max took 15.8 minutes a task against 6.7 at medium. For an overnight migration that's nothing; for an agent a developer is watching, it's the difference between a tool and a tab they abandon. Latency budget goes next to the cost budget, and both get set per task type.

It doesn't say the ceiling is fixed either. The same index has Claude Code on Opus 5 at extra-high effort at 68 for $8.17 a task. Model choice still moves the number more than anything else — this post is about the dial you never touched, not a claim that the dial replaces the model.

And it doesn't say these figures will hold. This is a live leaderboard: model versions rotate, harnesses ship changes weekly, and the specific numbers here have a shelf life measured in weeks. I'd be surprised if the rows above survive the quarter.

The habit that outlives the numbers

Every few months a chart appears showing that some tool beats the tool you use, and the honest reading is almost always the same: the two things were not asked to try equally hard.

So before you migrate anything, do the boring version. Take fifty to two hundred real tasks from your own repo with a pass/fail you'd defend in a review. Run your current harness across its effort settings against them. Find the point where your pass rate stops climbing — everything past it is money spent on thinking that didn't change an answer, and everything short of it is capability you already paid for and never collected.

Then, if you still want to switch harnesses, switch. You'll at least know what you're buying.

We're not all using the wrong coding harness. Most of us are using a reasonable harness on a default someone else picked, and calling the result a tool choice.


Figures from the Artificial Analysis Coding Agent Index, snapshot 26 August 2026. The index is a task-normalised average pass@1 with equal weight across DeepSWE (113 tasks), Terminal-Bench v2.1 (89) and SWE-Atlas-QnA (124), three attempts per task. Related: The cheapest AI upgrade we found was a settings change.

Get the monthly letter More posts