On PyPI · Python 3.11+ · MIT · runs on your machine
Agent runs are experiments,
treat them that way
loopmath is a CLI your orchestrator calls around each coding task. It compares agent workflows, prices each one before the task, and learns from every run after.
01 / what you can test
| What changes | How loopmath covers it |
|---|---|
| The workflow's shape | A graph of steps: one agent, implement then review, plan then implement, best of n, a swarm, or your own. It also tries one-step edits: a reviewer or planner added or removed, one more or one fewer parallel worker or repair round. |
| Models and efforts | Any allowed model and effort for each step, with reviewers from another family. |
| Harnesses | Claude Code, Codex or another harness for each step, as your settings allow. |
| Skills, plugins, MCP servers | A variant you record: a workflow file of its own, with the change in the step's options. loopmath prices it with the rest and learns from its runs. It does not find or install them for you. |
02 / the loop
loopmath predicts each workflow's chance of an accepted result and its cost, with a range, for this kind of task in this repository. It marks your goal, the lowest expected cost to an accepted result, and a workflow worth trying beside it.
Expected cost to an accepted result: the run's cost, plus a rescue by your usual workflow if it fails ($5.38 on average).
Fig. 1The workflows for one small bug fix. The diamond is the pick: $1.60 now, a 34% chance to beat your goal, and about $0.21 saved on each similar task after, so it pays back in about 8. The workflows start bunched with wide ranges and move apart as 40 runs arrive.
Your goal and the pick are two different graphs of agent steps. Here your goal has gpt-6.1-sol implement and claude-opus-5.5 review, and a rejected change goes back for repair. The pick runs two gpt-6-luna attempts in parallel, and claude-opus-5.5 selects one; at luna's prices, both attempts together cost less than one gpt-6.1-sol attempt.
Both runs start from the same commit, each in its own worktree. A blinded referee and the tests judge the two changes, and both runs are recorded.
Fig. 2One task, two workflow graphs on two cards: your goal, implement then review, and the pick, best of 2 then select. Each step lights up as it runs.
After the task, loopmath reads the real cost from the Claude Code and Codex logs and writes a receipt: the prediction beside what happened. Then it refits. One run moves the estimate a little, not all the way.
Fig. 3The trial predicted $1.60 ($1.05 to $2.45) and cost $1.38, with the tests passing. After the refit the estimate is $1.52, with a narrower range.
Before suggesting a trial, loopmath works out what one run of it would teach: how much lower the best expected cost is likely to be afterwards, averaged over every way the run could go and over every workflow. That drop is the trial's expected improvement, and the payback is the run's price divided by it. When nothing is worth trying, it says so.
Fig. 4The pick from Fig. 1, worked through. Before the task, loopmath's message gives the price, the chance to beat your goal, the expected saving and the payback.
Each task repeats the loop. In a simulation of forty similar tasks, the pick, best of 2 with gpt-6-luna, becomes the goal at task 5. At task 15 a new version of gpt-6-luna ships: it starts from the luna family's estimate, runs beside the goal, and takes over at task 16. At task 27 a model from a new provider ships, run as one agent for $1.10: it is tried beside the goal and takes over at task 31.
Every workflow runs at least once, and spend falls after each stretch of trials. Trials stop after task 32, and 39 of the 40 tasks are accepted. The last ten cost $1.35 per accepted change, against $5.38 for your usual workflow alone.
Fig. 5Simulated. Top: each run, by workflow, filled if its change was accepted; the shaded cell is the goal for that task. Middle: loopmath's estimate of each workflow's chance, with its 80% range; the new version starts from its family's estimate, with a wide range. Bottom: spend per accepted change over the last ten tasks; shaded, tasks with a trial. The true chances are the simulation's own; loopmath sees only the outcomes. Across 3,000 runs of the same simulation, the last ten tasks cost a median $2.48 per accepted change (10th to 90th percentile: $1.28 to $3.01). This run is one of the better ones.
03 / how it learns
Every estimate sits in two hierarchies. On the model side: provider, family, model, with effort estimated on its own and for each family. On the task side: task type, repository and subtype, with features such as size and language alongside. A run moves every estimate on its paths, by as much as the evidence supports, so a new model starts from its family's estimate and a new repository from its task type's, each with a wider range.
For each model family, loopmath also learns how costs on your runs differ from costs in our sweeps.
Illustrative. A run belongs to a model and to a task, so its evidence moves both trees: the model, its family and provider; the repository and its task type. A subtype, when you give one, sits below its repository.
04 / what it is
A command-line tool, not a harness: Claude Code, Codex, herdr or your own tool runs the agents, and calls loopmath before and after. It starts from a prior built from our own sweeps and experiments, then from your logs. Nothing leaves your machine, and every run is written as one open OCP document.
$ loopmath recommend --type bug_fix --repo acme/api …
bug_fix in acme/api, priced from 1,602 runs, 0 yours
| plan | implement | review | chance | cost | note |
|---|---|---|---|---|---|
| luna | luna | luna | 71% | $1.82 | not sure |
| sol | luna | luna | 94% | $0.70 | pick |
| sol | sol | sol | 99% | $1.91 | |
| - | opus | astra | 85% | $32.05 | default |
| the pick | estimate | 80% range |
|---|---|---|
| chance of an accepted result | 94% | 78% to 100% |
| cost of one run, typical | $0.09 | $0.01 to $0.60 |
| cost per accepted result | $0.70 | $0.04 to $1.79 |
chance: of an accepted result in one run (here, tests pass). cost: expected dollars per accepted result, fixing misses included.
$ loopmath recommend --type bug_fix --repo acme/api --feature size=s --feature lang=python Task: bug_fix in acme/api; runs per level: task 0, repo 0, type 101, all 1602, user 0 Rule: tests (the task's tests pass); fit fit_20261012192751 (0 min old) No usual workflow; reference: the default workflow (implement_review: claude-opus-5-5/high, gpt-6-astra/xhigh [cfg_ab15440a2d9c]): 85% success, typical run $7.15 (10,513,000 tokens; 80% of runs $0.94 to $68.42), expected rescue $0.90, expected cost per accepted result $32.05 (lower is better) The mean run costs $31.15 (68,089,000 tokens), higher because a few runs cost far more. Onboard to see your own costs. No habit runs of bug_fix tasks in acme/api (onboarded, or started with --source habit); name a usual workflow with --usual CFG. Rescue when a run fails: $5.99 per fixed miss, retrying with plan_implement_review: gpt-6-luna/high, gpt-6-luna/high, claude-sonnet-5/high (70% chance in one run), the cheapest workflow with at least a 70% chance, retried: each extra attempt has half the chance of the one before, up to 3 attempts in all; the retries fix 45% of misses; cost per accepted result = run cost + chance of failure x rescue Curve: 50/70%: plan_implement_review: gpt-6-luna/low, gpt-6-luna/low, gpt-6-luna/low: 71% success, $0.06 a run (2,709,000 tokens; median $0.01), expected rescue $1.76, $1.82 per accepted result (uncertain) 80/90%: plan_implement_review: gpt-6.1-sol/low, gpt-6-luna/low, gpt-6-luna/low: 92% success, $0.29 a run (3,388,000 tokens; median $0.07), expected rescue $0.47, $0.75 per accepted result (uncertain) 95%: plan_implement_review: gpt-6.1-sol/low, gpt-6-luna/low, gpt-6.1-sol/low: 95% success, $0.52 a run (4,484,000 tokens; median $0.14), expected rescue $0.27, $0.80 per accepted result 99%: plan_implement_review: gpt-6.1-sol/low, gpt-6.1-sol/low, gpt-6.1-sol/low: 99% success, $1.87 a run (10,958,000 tokens; median $0.39), expected rescue $0.04, $1.91 per accepted result Default pick: plan_implement_review: gpt-6.1-sol/low, gpt-6-luna/medium, gpt-6-luna/xhigh ($0.70 per accepted result) Goal (default): plan_implement_review: gpt-6.1-sol/low, gpt-6-luna/medium, gpt-6-luna/xhigh [cfg_e44128035bd0] Alternatives: plan_implement_review: gpt-6.1-sol/low, gpt-6-luna/medium, gpt-6-luna/max: 94% success, $0.32 a run, $0.70 per accepted result; from the recommended pick: reviewer effort: xhigh to max plan_implement_review: gpt-6.1-sol/low, gpt-6-luna/low, gpt-6-luna/xhigh: 93% success, $0.29 a run, $0.71 per accepted result; from the recommended pick: implementer effort: medium to low plan_implement_review: gpt-6.1-sol/low, gpt-6-luna/low, gpt-6-luna/max: 93% success, $0.30 a run, $0.71 per accepted result; from the recommended pick: implementer effort: medium to low; reviewer effort: xhigh to max plan_implement_review: gpt-6.1-sol/low, gpt-6-luna/high, gpt-6-luna/xhigh: 94% success, $0.33 a run, $0.71 per accepted result; from the recommended pick: implementer effort: medium to high Exploration (lookahead): best value: plan_implement_review: gpt-6.1-sol/low, gpt-6-luna/low, gpt-6-luna/low [cfg_6abfd96c79d1]: gain $0.13/run, 51% to beat the recommended pick, price $0.29, pays back after 3 similar runs biggest gain: plan_implement_review: gpt-6.1-sol/low, gpt-6-luna/low, gpt-6.1-sol/low [cfg_ab4c617cda31]: gain $0.17/run, 39% to beat the recommended pick, price $0.52, pays back after 4 similar runs You have no usual workflow for this task; the reference is the default workflow (implement_review: claude-opus-5-5/high, gpt-6-astra/xhigh), with an 85% chance of an accepted result; a typical run costs about $7.15 (10,513,000 tokens; 80% of runs $0.94 to $68.42). The mean run costs $31.15 (68,089,000 tokens), higher because a few runs cost far more. Onboard to see your own costs. Your goal is the lowest expected cost of an accepted result: plan_implement_review: gpt-6.1-sol/low, gpt-6-luna/medium, gpt-6-luna/xhigh, about $0.70 per accepted result. Run cost is what the agents cost for one run; cost per accepted result adds the expected cost of fixing a miss with plan_implement_review: gpt-6-luna/high, gpt-6-luna/high, claude-sonnet-5/high, where each extra attempt has half the chance of the one before, up to 3 attempts in all. The recommended workflow has a 94% chance of an accepted result in one run, and a 96% chance within 3 attempts. Trying plan_implement_review: gpt-6.1-sol/low, gpt-6-luna/low, gpt-6-luna/low alongside it costs $0.29 (3,388,000 tokens) now. There is a 51% chance it beats the recommended pick. Trying it once is expected to save about $0.13 on each future similar run, so it pays for itself after about 3 similar runs. The option with the biggest gain is plan_implement_review: gpt-6.1-sol/low, gpt-6-luna/low, gpt-6.1-sol/low: it costs $0.52 (4,484,000 tokens) now, has a 39% chance to beat the recommended pick, is expected to save about $0.17 per future similar run, and pays for itself after about 4 runs. rec: rec_01M4SHD3ZFGZ080MJSBCXK5X7F
$ loopmath recommend --type bug_fix --repo acme/api --feature size=s --feature lang=python --json | jq '.choices[0]'
{
"key": "goal",
"title": "Run the recommended workflow",
"label": "plan_implement_review: gpt-6.1-sol/low, gpt-6-luna/medium, gpt-6-luna/xhigh",
"config": "cfg_e44128035bd0",
"members": [
"cfg_e44128035bd0"
],
"cost_per_accepted_usd": {
"mean": 0.697498,
"lo": 0.040942,
"hi": 1.787275
},
"chance": {
"mean": 0.935863,
"lo": 0.782738,
"hi": 0.999705,
"of": "an accepted result"
},
"run_cost_usd": {
"mean": 0.313142,
"median": 0.085833
},
"expected_rescue_usd": 0.384356,
"p_accepted_within": {
"mean": 0.964482,
"lo": 0.879685,
"hi": 0.999836,
"attempts": 3
},
"bands": {
"chance": {
"50": [
0.948219,
0.998831
],
"80": [
0.782738,
0.999705
],
"90": [
0.657768,
0.999922
],
"95": [
0.548006,
0.999959
]
},
"run_cost_usd": {
"50": [
0.030753,
0.234474
],
"80": [
0.012709,
0.595707
],
"90": [
0.007594,
0.977232
],
"95": [
0.005649,
1.550256
]
},
"cost_per_accepted_usd": {
"50": [
0.091794,
0.7374
],
"80": [
0.040942,
1.787275
],
"90": [
0.024667,
2.519851
],
"95": [
0.015061,
3.687957
]
}
},
"wins": {},
"recommended": true,
"strategy": null,
"option": 1
}Real output of loopmath 0.2.8 from PyPI, run on October 12, 2026 in a fresh store that holds only the runs the package ships with.
05 / blog
What we learn from agent runs, with the method and the uncertainty.
all posts →harness2 improves a coding agent's harness, and it is expensive. Paired with loopmath, it kept about 95% of harness2's gain on 40 coding tasks at about a quarter of its list cost.
5 min read →06 / contact
About using loopmath, about OCP, or about a workflow you want to test.