harness² plus loopmath: 95% of the gain at a quarter of the cost

The result

harness², from Google Cloud AI Research and Northwestern University, improves a coding agent's harness: its instruction file, skills and hooks. It works, and it is expensive. Every task it improves costs 11 sessions of a frontier model.

We kept harness²'s editors and replaced the step where it decides which edits to keep with loopmath: a small, designed set of runs and a statistical estimate of what each edit is worth. On 40 coding tasks, harness² with loopmath kept about 95% of harness²'s gain at about a quarter of its cost: 2.64 against 10.18 list dollars per task.

• Claude Haiku 5.5 does the tasks; Claude Opus 5.5 edits the harness.
• The base harness scores 0.746 on the benchmark's hidden tests. harness² as published lifts it to 0.798. harness² with loopmath reaches 0.796.
• harness² cut down to the same budget as harness² with loopmath reaches 0.773.

What harness² does

harness² (Xu and colleagues, 2026; code) treats the harness as the thing to optimise. For each task, editor sessions propose edits to the harness, each candidate is run on the task, and composer sessions merge the best ideas into a refined harness, without ever seeing the answer. A reflector then writes a lesson into a library for the task's domain, so later tasks start from what earlier ones learned.

The cost is in the editor model. At its published setting, each task runs 6 editor sessions, 3 composers, a selector and a reflector: 11 Opus sessions and 12 task runs. On our tasks, the editing sessions alone were 343.49 of harness²'s 407.06 list dollars.

Our question was simple: can a cheaper way of choosing which edits to keep hold most of that gain? Before the run we set the bar: the cheaper method must keep at least 80% of harness²'s gain over the base harness.

The method, in plain words

loopmath treats a harness edit the way it treats any change to an agent workflow: as an experiment with a cost and an effect to estimate.

1. Propose. Two Opus editors, one on how the agent works and one on domain knowledge, each propose up to 5 small edits that can be switched on and off on their own.
2. Probe. Instead of trying the edits one by one, loopmath plans 12 runs that switch them on and off in a balanced pattern. Each edit is on in half of the runs and off in the other half, so 12 runs can weigh up to 10 edits at once.
3. Score without the answers. Each run is scored by an answer-free tester: Claude Sonnet 5.5 writes acceptance tests from the task text and the repository. It never sees the hidden tests.
4. Estimate and keep. A fit over the scored runs estimates each edit's effect. The edits that help are kept and run once as the final harness. If no edit helps, loopmath ships a plain run and spends nothing more.

That is 3 Opus sessions per task, against harness²'s 11. The editors and the library are harness²'s own; the deciding is loopmath's.

The test

We ran three arms on the same 40 test tasks from WorkBuddy-Bench Code, using harness²'s own split, in the same order:

• Arm 1, harness² as published, at its default setting.
• Arm 2, harness² plus loopmath, as above.
• Arm 3, harness² at the same budget: harness² cut down to arm 2's number of Opus sessions, to check that arm 2 does more than simply run less harness².

The task model is Claude Haiku 5.5 running headless in Claude Code, in a fresh container for every run. The score is the benchmark's hidden tests, from 0 to 1, and only the one output an arm delivers for a task is scored. The base is harness²'s own one-sentence starting harness, run three times on every task and averaged, so every arm is measured against the same figure.

Every model call was logged with its tokens and priced at public list prices, so the arms can be compared dollar for dollar.

Results

arm	mean score	gain over base	share of harness2's gain	list $ per task
base, one-sentence harness	0.746			0.02
arm 1, harness2 as published	0.798	+0.053	100%	10.18
arm 2, harness2 + loopmath	0.796	+0.050	95%	2.64
arm 3, harness2 at the same budget	0.773	+0.028	52%	2.61

harness² with loopmath clears the bar we set. It keeps 95% of harness²'s gain, well above the 80% line, at 26% of harness²'s list dollars. Both harness²'s gain and loopmath's clear zero at the 95% level: both methods help on this benchmark. At the same money as harness² with loopmath, harness² cut down keeps 52% of the gain.

Where the money goes

Over the 40 tasks, harness² as published cost 407.06 list dollars and harness² with loopmath 105.74. Almost all of the difference is Opus editing sessions. loopmath runs more task runs than harness², 12 probes on every task, but a Haiku run costs a few cents, and its Sonnet tester costs less than its task runs. The expensive part, the frontier model, is where loopmath saves.

Spending only where it pays

loopmath does not force an edit onto every task. It ran a final harness only when its fit found at least one edit worth keeping. That happened on 20 of the 40 tasks, where it kept between 1 and 10 edits, 5.7 on average. On the other 20, no edit showed a gain on the tester, so loopmath shipped a plain run and stopped.

On the 20 tasks where loopmath shipped its edits, its mean gain over base is +0.070, against +0.083 for harness² on the same tasks. On the other 20, harness²'s eleven sessions bought +0.022, and loopmath's plain run did as well. So on the half of the tasks where an edit could make a difference, loopmath found most of what harness² found; on the other half, nobody gained much, and loopmath spent almost nothing finding that out.

Why it works

harness² spends its budget generating and merging candidates, then trusts a single run of each. loopmath spends a little on generation and puts the rest into measurement. A balanced design means every run informs every edit, so the same 12 runs that would test 12 candidates once each instead estimate 10 edits from 6 runs apiece. The edits that help are kept together; the ones that do not are dropped before they cost a final run.

This is what loopmath does for any agent workflow: it treats each run as an experiment, estimates what each choice is worth on success and on cost, and spends where the evidence says it pays. Harness edits are one more choice it can weigh.

Try it

loopmath is open source under the MIT licence. The harness command is being packaged now; the rest of loopmath, the predictions, the success-cost curve and the paired checks, is on GitHub and PyPI today. To try loopmath on your own harness, get in touch.