Bring your hypothesis: test your agent workflow with loopmath

Work in progress. We are adding direct support for multi-step, multi-model flows: you declare the flow, its question and its judges once, and loopmath plans the experiments for it. Sections marked coming are where that lands. This page will change as it does; everything below works on 0.3.3 today.

How to start. Open Claude Code or Codex in an empty folder and say: Read https://loopmath.ai/tutorial/TUTORIAL.md and walk me through it, one section at a time.

For the agent: how to run this walk-through

You are walking a person through loopmath on their own work. Follow these rules in every section.

1. One section at a time. Open each one with two or three plain sentences on what it is for, then do its steps.
2. Before each command, say what it does. After it runs, show the output and explain it in plain words.
3. Ask before you install anything, before anything that spends money (a model call, a run of their flow), before any download, and before any file leaves this machine.
4. Ask the questions marked Ask and wait for each answer. Write what you learn into problem.md in the working folder.
5. Never make up a number. Every number you report comes from a command's output or from the person.
6. When a section says coming, say plainly that it is not built yet.
7. Do not run loopmath recommend for the person's flow. Today it plans coding tasks, and on a research flow it proposes coding workflows.
8. Put the demo in its own store (--home demo-store), so the person's own store stays clean.

Every command here was run on loopmath 0.3.3, the release on PyPI. loopmath --version should print loopmath 0.3.3.

0. What loopmath is, and what it is not

loopmath treats agent runs as experiments. It records, judges and compares runs of any multi-agent work you can score, code or not: a research flow that ends in a memo, a planning flow that ends in a decision, a coding flow that ends in a change. Planning a run before it starts is built for coding tasks today; for other flows that part is coming.

• It is a command-line tool. Your agents or your own orchestrator call it; it never starts or coordinates agents.
• It runs on your machine and connects to no other machine. It opens agent logs read only, and writes to its own store (~/.loopmath), to the skill folders it installs, and to paths you name.
• One caveat. If you have it read your Claude Code or Codex history (optional, in section 1), it labels each past task with a model you choose: your own Claude or Codex CLI, a local model, or none. That model sees short spans of text from each past session: up to 600 characters of its first prompt, the commit messages made while it ran, and the command that started it.
• It sends nothing to us. loopmath share writes a file you can read, and you decide whether to send it.

Agent. Say this in four lines, then ask whether to go on.

1. Onboarding (about 15 minutes)

You install loopmath and its skills, then check what it can see.

Agent.

1. Ask before installing: "May I install loopmath? It needs the network." Use uv, which brings its own Python and so does not depend on the one on this machine. If uv --version works, run uv tool install --python 3.12 loopmath. If uv is missing, ask to install it first (brew install uv, or the installer on docs.astral.sh/uv), then run the same command. If the person prefers pipx and it already works for them, pipx install loopmath is fine too. Then run loopmath --version.
2. If an install fails inside pip or Python itself rather than in loopmath, the machine's Python is broken, not loopmath. Signs: a ValueError from int(''), a "Symbol not found" error when Python loads pyexpat, or an SSL error. One case seen: Homebrew's Python on a macOS older than it was built for cannot load its XML module. Say so plainly, switch to uv tool install --python 3.12 loopmath, and do not try to repair Homebrew or the system Python.
3. Run loopmath skill install --dry-run, then loopmath skill install. The install above puts in the command only. This second step teaches Claude Code and Codex how to use loopmath: six skills in ~/.claude/skills, and for Codex six skills in ~/.codex/skills when that folder exists. If Codex has no skills folder yet, the six go in ~/.codex/loopmath with a short marked block in ~/.codex/AGENTS.md that lists them. Nothing outside those folders is touched, and no other AGENTS.md. The dry run lists every file first.
4. Run loopmath doctor. It shows what loopmath can see: logs, agents, store, fit, prices and skills. On a fresh install, "no fit yet" is expected.
5. Optional, only if the person also runs coding agents on this machine: loopmath onboard --dry-run reads the last 90 days of Claude Code and Codex logs and shows what it found and how many groups of sessions it would label. It calls no model. Ask which model should label them, or none; never pick it yourself. Then run loopmath onboard --labeler <choice> --yes, where the choice is claude:<model>, codex:<model>, command:<cmd> for a local model, or none. After that, loopmath analyze gives the cost per accepted run over the last 14 days. If their flow runs through an API or an agent SDK, its runs are not in these logs; section 4 brings them in.

What doctor printed on a fresh install with the skills in (paths shortened):

loopmath 0.3.3 doctor, store ~/.loopmath
warn  logs    claude-code none at ~/.claude/projects; codex none at ~/.codex/sessions
warn  agents  claude-code not found; codex not found
ok    store   ~/.loopmath writable (created)
warn  fit     no fit yet: run `loopmath fit` (or `loopmath onboard`)
ok    prices  no models seen in 30 days; table as of 2026-08-31
ok    skill   6 skills: claude-code user (skills), codex user (AGENTS.md block)

That home folder had no ~/.codex/skills, so Codex got the block form. Where Codex has its skills folder, the line reads codex user (skills).

2. Setting the problem (about 15 minutes)

Your agent interviews you about the flow you want to improve, and writes it down. The sharper the question, the fewer runs it takes to answer.

The worked example (synthetic). A research desk tests the hypothesis "Companies that cut guidance underperform their sector over the next 60 trading days." Four agents, all on Claude Opus, work in turn:

1. Scout finds the data and writes a data spec.
2. Analyst writes and runs the backtest.
3. Critic checks for look-ahead, survivorship and multiple testing, and can send the work back to the Analyst once.
4. Writer writes a memo whose verdict is supported, not supported, or inconclusive.

A run takes about two hours and costs about 10 USD. The desk's question about its own flow: can the Critic run on Sonnet instead of Opus, and the memos stay as good?

Agent. Ask these one at a time, and write the answers into problem.md under the same headings.

1. The flow. The steps in order. For each step: what it does, its model and effort, what it receives and what it passes on, and whether it can send work back.
2. One run. Where a run starts and ends, how long it takes, and roughly what it costs.
3. A good run. What makes the output good? What can a script check? What would you look at yourself?
4. The question. What do you want to know about the flow? Help them write it as one change and one measure, like the desk's question above. If they say something broad ("which models should we use?"), propose a first sharp question from it: one step, one change.
5. What it is worth. What a run costs, and what a better output is worth to them. Saving 1 USD a run is worth a bigger test on a flow that runs 500 times than on one that runs 10 times.
6. The records. Where does a run leave a trace today: the orchestrator's logs, an agent SDK's traces, a database, files? Section 4 needs, for each step, the model and the cost or tokens, and for each run, its score.
7. The budget. How many extra runs they can pay for while testing.

Then run loopmath task-types. Their flow's runs go in as research tasks: "Investigate, prototype or answer a question; the output is a finding."

Declaring a flow and its question to loopmath directly is coming. Today problem.md holds them, and each run carries the flow's shape.

In a retail flow: setting the problem

The same steps fit a retail planning flow, for example a weekly markdown plan. A data agent pulls the last weeks' sales and stock. A forecaster estimates demand at each discount. A planner proposes which items to mark down and by how much. A person approves or edits. A writer drafts the note to stores. "Which model should each step use?" is broad. "Can the forecaster run on a smaller model without more plans being edited by the person?" is sharp: one step, one change, one measure.

3. The problem as a statistical problem (about 10 minutes)

Run the same flow on the same question twice and you get two different memos. The models sample, the data pulls differ, and the judges disagree, so each run is one draw from a distribution. The outcome of a run is a random variable, and so is its cost. Three things follow.

Measure the outcome in more than one way.

• Checks are scripts: the memo has its sections, its numbers match the result files, the backtest reruns. They are cheap and exact, but a memo can pass every check and still be wrong.
• Agent judges score the output on a rubric. Use several, on different models. How far apart they land is information: in the demo, the three judges differ by a point or more on 14 of 24 memos.
• Your own grade is the scarcest, and the one that counts. Grade a random sample, not the memos that look odd, so you learn which judge agrees with you. In the demo, a person graded 4 memos, and on 3 of them the judges' mean matched the grade.
• Then say what "good enough" means, for example a judges' mean of 3.5 or more. That turns a score into accepted or not, and lets you compare what each accepted output costs.

Compare in pairs, and repeat.

• Run both versions on the same questions. A hard question then hurts both, and what is left is the difference between them.
• One pair proves little. Before testing a change, run the current flow 3 to 5 times on one question. If the judges' mean moves by half a point between runs where nothing changed, a change worth 0.2 points needs many pairs to show.

Decide when to stop before you start.

• loopmath's check (section 5) says adopt, reject, or run more after every pair. Its adopt line is set so that, in loopmath's simulations, a change that does not really help is adopted at most about 5% of the time, even though you look after every pair. It has a cap on pairs and takes a budget.
• If you try many variants and keep the best, the winner looks better than it is: part of its lead is luck. Section 7 has a tool for that.

Agent. With the person, add "How a run is judged" to problem.md: the checks, the judges (how many, which models, the rubric), the human sample (how many, drawn at random), the threshold for good enough, and the noise test (one question, 3 to 5 runs of the current flow). The noise test spends money on their flow, so ask before running it.

4. Bring your runs in (about 15 minutes)

loopmath reads runs in a plain format: one JSON line per run, with one entry per agent. Try it on the demo first, then on your own runs.

The demo

demo-runs.jsonl holds 24 synthetic runs of the example flow: 12 questions, each run once with the Critic on Opus (arm base) and once with the Critic on Sonnet (arm critic-sonnet). No agent ran. The scores and costs are made up, and every row says so. What loopmath prints about them is real output of loopmath 0.3.3.

Agent.

1. Ask, then download demo-runs.jsonl into the working folder.
2. Convert the rows and import them into a separate demo store: loopmath adapt harness-runs demo-runs.jsonl --out demo-ocp, then loopmath run import demo-ocp --finish --home demo-store. The import starts a fit in the background. Run loopmath status --home demo-store until it says background fit: last done, which takes a few seconds.
3. Run loopmath runs --home demo-store and explain one line: the models of the four agents, the cost, the rounds (2 means the Critic sent the work back), and the judges' score. "unknown" in the outcome column means no accept or reject is on record; the judges' score is beside it. The Critic shows as review, because loopmath folds role names onto its own, and a critic is a reviewer.
4. Add the person's grades. The demo has four, also synthetic. Run the four loopmath outcome lines below, then loopmath runs --run run_hr_125fcb83fe5dc083133c --home demo-store to show both signals on one run. A grade from the person is verified; a judge's score is reported.
5. Run loopmath posterior --level role --target 'judge>=3.5' --home demo-store and explain it with the notes below.

The four grades:

loopmath outcome --run run_hr_759a2e6a3c6e7123cf7a --signal human=3 --kind score --source user --tier verified --home demo-store
loopmath outcome --run run_hr_616629e2bcc548c0a742 --signal human=5 --kind score --source user --tier verified --home demo-store
loopmath outcome --run run_hr_125fcb83fe5dc083133c --signal human=5 --kind score --source user --tier verified --home demo-store
loopmath outcome --run run_hr_d47b338565c815330719 --signal human=4 --kind score --source user --tier verified --home demo-store

What loopmath runs --run run_hr_125fcb83fe5dc083133c --home demo-store printed (the grade's time will be yours):

run run_hr_125fcb83fe5dc083133c  finished  started 2026-10-27 12:00 PDT
task       Buyback announcements in high-volatility weeks (research, demo-desk)
workflow   stages-scout-analyst-review-writer: claude-opus-5-5/high, claude-opus-5-5/high, claude-sonnet-5-5/high, claude-opus-5-5/high [cfg_676ae98284a1]
source     habit
cost       $8.95 (n/a tokens) over 1 round
outcome    unknown, judge 4.33
receipt    no prediction on record
attempts   4, artifacts 0, signals 2
  2026-10-27 14:18 PDT  score judge = 4.33 (reported)
  2026-11-03 11:28 PST  score human = 5.0 (verified)  late

What loopmath posterior --level role --target 'judge>=3.5' --home demo-store printed (trimmed):

Target: judge >= 3.5 (from --target)
Your workflows (4), by chance to reach 3.5:
  stages-scout-analyst-review-writer: ...claude-sonnet-5-5/high...: judge 3.56 (2.41 to 4.97), 52% chance, $9.51 a run, 8 runs
  stages-scout-analyst-review-writer: ...claude-sonnet-5-5/high...: judge 3.44 (2.31 to 4.94), 48% chance, $9.49 a run, 4 runs
  stages-scout-analyst-review-writer: ...claude-opus-5-5/high...: judge 3.37 (2.26 to 4.78), 45% chance, $10.60 a run, 9 runs
  stages-scout-analyst-review-writer: ...claude-opus-5-5/high...: judge 3.26 (2.11 to 4.73), 41% chance, $10.57 a run, 3 runs
Your models, one agent alone at its best effort for judge:
  claude-opus-5-5/default: judge 3.43 (0.82 to 6.09), $6.10 a run; 0 solo runs here
  claude-sonnet-5-5/high: judge 3.67 (1.15 to 6.14), $4.65 a run; 0 solo runs here
...
  [score:judge] role x family reviewer|opus: -0.06 (-0.72 to +0.6), 12 runs
  [score:judge] role x family reviewer|sonnet: +0.06 (-0.59 to +0.71), 12 runs

How to read it:

• Ranges are 80% ranges. Each workflow line gives the expected judges' score, the chance a run reaches 3.5, and the cost of a run.
• Each version of the flow shows up twice: once for the runs where the Critic sent the work back and once for the rest. loopmath 0.3.3 counts those as two shapes. Compare the two versions by arm in section 5, not by these lines.
• The reviewer|sonnet line is the per-step estimate: the effect on the score of the Critic's model family. +0.06, with a range from -0.59 to +0.71, means no sign of a difference so far, and a wide range.
• Skip the "Your models" lines about one agent working alone. None of these runs had one agent alone, so those lines rest on loopmath's starting estimates, which come from coding runs. A range there can pass 5, because a linear score has no upper bound.

Your own runs

Your agent writes a small exporter: a script that reads your flow's records and writes one line per finished run. You review it before it runs. Here is one demo row, spread over lines for reading (in the file it is one line):

{"schema": "loopmath.harness-runs/0.1",
 "task_id": "q01",
 "task": {"type": "research", "repo": "demo-desk", "title": "Guidance cuts and 60-day sector-relative returns"},
 "experiment": "critic-model-demo",
 "arm": "base",
 "pair": "q01#1",
 "status": "done",
 "agent": "claude-agent-sdk",
 "model": "claude-opus-5-5",
 "effort": "high",
 "stages": [
   {"role": "scout", "model": "claude-opus-5-5", "effort": "high", "cost": {"usd": 1.196}},
   {"role": "analyst", "model": "claude-opus-5-5", "effort": "high", "cost": {"usd": 4.235}},
   {"role": "critic", "model": "claude-opus-5-5", "effort": "high", "cost": {"usd": 1.9681}},
   {"role": "writer", "model": "claude-opus-5-5", "effort": "high", "cost": {"usd": 1.0576}}],
 "outcome": {"name": "judge", "score": 2.67, "judge": "panel-of-3", "scale": "linear", "better": "higher",
             "judge_a": 3, "judge_b": 3, "judge_c": 2},
 "started_at": "2026-10-26T09:00:00-07:00",
 "ended_at": "2026-10-26T10:43:00-07:00",
 "tool": {"name": "demo-desk", "run_id": "q01-base", "task_id": "q01"},
 "ext": {"dev.loopmath.tutorial": {"synthetic": true, "note": "synthetic run for the tutorial; no agent ran"}}}

• task_id is the question. Both versions of a pair use the same one. task.type is research; task.repo names the project, in any words you like.
• arm says which version ran, and pair (<task_id>#<n>) ties the two runs of a pair together. For runs from before any test, use one arm such as base.
• experiment is a stable name for the test, never a folder path.
• agent, model and effort are the defaults for every stage; a stage can set its own.
• stages has one entry per agent session, in order: its role, model, effort and cost. Give {"usd": ...}, or tokens, which loopmath prices for models in its price table. A second pass after a send-back adds "round": 2 to that stage and to the Critic's second look.
• tool names your orchestrator and its own id for the run. The run's id in loopmath comes from tool.name, experiment and tool.run_id, so importing the same file again changes nothing.
• outcome is the score and who gave it. Extra members, such as each judge's score, are kept.
• ext holds anything else of yours, under a name of your own.

The format's full description is spec/harness-runs.md in the loopmath repository on GitHub.

Agent. Write the exporter with the person and show it before running it. Rows hold metadata only: models, costs, times and scores. Never copy prompts, documents or answers into a row. A run that stopped midway gets "status": "failed". Import with the same two commands as the demo, into the person's own store (no --home) or a store named for the project. Other judges' scores and the person's grades go in with loopmath outcome, as in step 4.

5. Test one change (about 10 minutes)

Now the question from section 2 gets its answer, from pairs.

Agent.

1. On the demo, run loopmath check-change from-runs --base-label arm:base --cand-label arm:critic-sonnet --rule 'judge>=3.5' --home demo-store, then explain it with the notes below.
2. Then run the same command with --rule score:judge, for the change in the raw score.
3. Ask the person what they would do, and why.
4. On their own runs, give each run its arm when exporting, run the pairs, and rerun the same command after each batch. Stop on its answer, or at the cap or the budget, not on a hunch.

What the first command printed (the last notes trimmed):

arm:critic-sonnet (12 runs) against arm:base (12 runs), rule judge>=3.5
verdict: more  P(better) 0.574 (adopt at 0.993 or above, reject at 0.050 or below), 12 scored pairs
gain: mean -0.000, median -0.000, 80% -0.147 to 0.142
cost per run: base mean $9.8104, median $9.7571, 80% $8.8311 to $10.8158; candidate mean $8.8699, median $8.7868, 80% $7.9438 to $9.8710
cost per accepted result: base mean $25.5316, median $23.3269, 80% $16.0802 to $36.9550; candidate mean $23.1080, median $21.0244, 80% $14.6855 to $33.4053
next pair worth $0.0000 a pair over 50 runs, buying up to 1 more pair (...); price $18.6803; not worth buying (net below 0 by two standard errors)
...
cap: 24 pairs, chosen for the 12 pairs recorded (...)

How to read it:

• --rule 'judge>=3.5' counts a memo as accepted when the judges' mean is 3.5 or more. Each arm had 5 of 12 memos accepted, so the "gain" in the chance of acceptance is 0. The Sonnet Critic costs about 0.94 USD less a run, so an accepted memo costs 23.11 USD against 25.53.
• verdict: more means not decided yet. P(better) 0.574 is the chance, given these pairs, that the Sonnet Critic really cuts the cost per accepted memo by more than the floor (5% by default). The check adopts at 0.993 and rejects at 0.05.
• "next pair worth $0.0000 ... price $18.6803": the value is what one more pair is expected to be worth over the next 50 runs of the flow, through the better decision it could lead to. From P(better) 0.574, one pair cannot carry the check to adopt or reject, so it is worth 0 against a price of about 18.68 USD (one run of each arm). With --horizon-runs 500 it is still 0. The check does not stop on this: it goes on to a decision, the cap (24 pairs here), or a budget you set with --budget-usd.
• With --rule score:judge, the same pairs give P(better) 0.833 and a gain of +0.195 points (80% range -0.068 to +0.457). The Sonnet Critic does not look worse; it is not yet shown to be better.
• So the evidence says the cheaper Critic is about as good, and the person decides: 12 more pairs (about 224 USD at these costs), keep Opus, or switch knowing it is not settled.

Several cheap tries at one step with the best one kept, for example three drafts from a small model and a pick, is just another arm. Export it under its own arm label and compare it the same way.

In a retail flow: one step at a time

To find the cheapest model that is good enough at each step, test one step at a time and leave the others as they are. Start with the structured steps, such as the data pull, where a script can check the output. Make the person's edits part of the measure: a plan the person had to rework counts against the arm.

6. What to try next (coming)

Choosing the change is still your call today. What we are building next: from your flow and its runs, loopmath will suggest which experiment to run next, such as a cheaper model at one step, dropping or merging a step, adding a check, or several cheap tries at one step with the best kept. It will also say how many repeats each experiment needs and when to stop. And it will learn which judge agrees with your own grades, so your time goes where the judges disagree. None of this is in loopmath 0.3.3.

Agent. Say plainly that this is not built. Ask which experiments on their flow they would most want ranked, and write the answer into problem.md. It is the most useful thing they can tell us (section 8).

7. Always on: did a change make it worse?

Once the flow runs for real, every change is a test: a model swap, a prompt edit, a tool that starts returning new fields. Keep a fixed set of 10 to 20 questions, rerun them in pairs after each change, and run the same check.

Agent.

1. With the person, pick the fixed questions: a mix of easy and hard, with ids that never change.
2. After a change, run each fixed question once on the old flow and once on the new one, export them with arms before and after, and run loopmath check-change from-runs --base-label arm:before --cand-label arm:after --rule 'judge>=3.5'.
3. For a loop that adopts changes by itself, loopmath gate (experimental) adds a policy: it judges only at planned points and stops buying pairs once more are not worth their price. On the demo, run loopmath gate --base arm:base --cand arm:critic-sonnet --rule 'judge>=3.5' --r-max 24 --home demo-store. It keeps the base, because 4 more pairs at about 18.68 USD each are not worth it over 50 future runs.
4. If the person tried many variants and kept the best, loopmath gate winner (experimental) says how much of the winner's lead is likely real. It needs every variant's scores (--scores FILE or --runs FILE), and --spread (one run's standard deviation on one question) when no variant ran twice on one question. On the demo, loopmath gate winner --runs demo-runs.jsonl --spread 0.4 says the Sonnet Critic leads by +0.195 points, keeps +0.164 of that after allowing for luck, and asks for 1 more pair on fresh runs before adopting: the search's own scores never adopt.

A watch that runs these checks by itself after every change is coming.

In a retail flow: after deployment

After the flow is deployed at a client, the fixed questions are fixed past weeks with known outcomes. Each change gets a before and after run of each week. The person's approve or edit on each plan is their grade, and the check says whether the change made plans worse.

8. Working with us

What we ask of design partners: run sections 1 to 5 on one flow of yours, then take 30 minutes with us on what it says. Send nothing unless you choose to. Before you do, loopmath share --preview prints exactly what would leave: task types, flow shapes with their step names, models, costs, and scores with their names. Titles, paths, prompts and experiment names are removed, and project and task names are hashed. Write to hello@loopmath.ai, or book a time.

Agent. Run loopmath share --preview --home demo-store to show what a share of the demo would hold. It prints a long JSON document: summarise what kinds of fields it holds, and show one run. Do not write or send a share file unless the person asks.

How this was checked

On 2026-11-03, the commands in this walk-through were run on loopmath 0.3.3, in an empty home folder, on the demo file. The outputs above are quoted from that run, trimmed where marked. Two steps were not run: the install itself (uv or pipx), and onboard --labeler ... --yes, which calls the labelling model. onboard --dry-run and analyze ran on an empty history only. Ask your agent to rerun any of them: the demo commands give the same numbers each time.