The scripts are in the repository, not in the installed package. Clone it,
then
cd recipes/papers/meta-harness before running the commands below. Browse this recipe on GitHub.harness(model) -> wai.Harness; every candidate is scored on the
same frozen tasks; the proposer reads every prior candidate’s source, score
and worst rows through the filesystem and writes the next file; and the
pick has to beat the baseline on held-out tasks and on a held-out model
before it counts. That is the loop of Lee, Nair, Zhang, Lee, Khattab and
Finn 2026 [1], with the harness as the object wai.Harness already
versions and runs.
What you will learn: how to freeze one task set so every candidate answers
the same asks (tasks=), what the proposer has to see (the paper’s point is
that nothing is compressed: source, scores, traces), and what a search
result has to clear before it is a result (an interval that excludes zero
on held-out tasks, the same on a held-out model, and
wai.harness.attribute saying the gain is the harness). You need nothing
for --dry-run. The live run needs the provider key for every model you
name (OPENAI_API_KEY, ANTHROPIC_API_KEY), or one OpenAI-compatible
router for all of them: vllm:<model>@<url> with the router’s key in
VLLM_API_KEY is how the run below reached both models through
OpenRouter. With --judge, the key for the judge. Seconds offline; about
seven minutes per candidate and model live at --concurrency 16.
The dry run demonstrates the loop with scripted candidates. Each candidate
carries an offline stand-in, a seeded agent whose planted-mistake rate the
candidate sets, so the numbers below show the mechanics (freeze, score,
propose, gate, attribute) and are not a replication. The live run is in
the Result section: Claude Haiku 4.5 as the search model, gpt-4.1-mini
held out, the three checked-in candidates, one round, and the gate passed
with +38 points on 30 held-out asks at 0.40x the baseline’s tokens. What
the paper reports: +7.7 points on online text
classification over a state-of-the-art context manager with 4x fewer
context tokens, +4.7 points on 200 IMO-level problems averaged across five
held-out models, and discovered harnesses that surpass the best
hand-engineered baselines on TerminalBench-2 [1].
Run it
Every run scores every file in
candidates/, in name order, and the first
file is the baseline. The first candidate draws the frozen set from the
seeds in common.py with the offline template writer (simulator=False,
deterministic, no key) and saves out/tasks.jsonl; every later candidate
and every model replays it with tasks=, so the asks match. With
--traces, the frozen set is the agent’s own traffic instead: one task per
distinct prompt in the file (a JSONL wai.load_traces reads, or an OTLP
JSON batch wai.rows_from_otel groups into conversations), at most
--budget of them. The holdout is then the latest days, whole days, until
it holds --holdout of the tasks, so no row the proposer reads comes from
the days that decide; train prompts that overlap a holdout prompt
(wai.decontaminate, the 8-gram rule) leave the proposer’s window, and
out/split.json records the days, the keys and the count dropped. The judge is a
program: it reads the reply for filler, checks that a reply claiming
success sits on a tool result that succeeded, and that nothing privileged
leaked. The outer loop is you, or the coding agent running
skills/harness-search: read
out/proposal.md, write the next file, run again.
A candidate
One file, one change. This iscandidates/01_no_filler.py: the baseline’s
worst rows open with a greeting, an apology or a hedge, so it tells the model
to answer in one sentence and caps the loop at four turns. build in
common.py turns it into the prompted loop on a real model, or the scripted
stand-in when the model name starts with scripted.
Disclosure fields, so the edit is a new harness version without anyone
naming it, and every row the candidate produces carries it.
What you get
python run.py --dry-run --propose --select, the three checked-in
candidates on the two scripted models:
02_check_result leads 12 of the 12
train tasks (per task, the best pass rate across candidates, ties shared),
and on the held-out tasks it beats the baseline by 27 points with an
interval of +15 to +42, which excludes zero, on the search model; on the
held-out model the same harness beats the same baseline by 50 points, +38
to +62. Then attribution over the three-by-two grid of holdout scores says
the harness explains 82% of the spread (interval 48 to 97) and the model
11%, so the gain is the harness. 01_no_filler is not selected: it is not
the best on train, and on its own it would not pass either, +8 points on
the same 12 held-out tasks with an interval of +0 to +19, which is why the
skill’s check writes a second file before it stops. On the scripted models
these numbers are what the planted rates make them; they show the gate
working, not the paper’s result.
Three files carry the state between rounds:
out/ledger.jsonl: one line per candidate, with its file, fingerprint, model, train and holdout pass@1 with intervals, task counts, and the path of its worst rows and of every row it produced.out/proposal.md: the proposer’s view. Every candidate’s source, its score with interval, its five worst rows on the train split (ask, reply, why the judge failed it), and one instruction: writecandidates/03_<name>.py, then run again.out/selected.json: the gate’s answer. The candidate picked ornull, the train tasks each candidate led, the paired delta and interval per model with the count of holdout tasks the baseline passed and the pick failed, the cost ratio, and the attribution verdict.out/split.json: how the tasks were split (by seed, or by day for traces), the keys on each side, and the train prompts dropped for overlapping a holdout prompt.
Result
python run.py --models "vllm:anthropic/claude-haiku-4.5@https://openrouter.ai/api/v1,vllm:openai/gpt-4.1-mini@https://openrouter.ai/api/v1" --budget 60 --concurrency 16 --propose --select,
2026-09-22. Claude Haiku 4.5 is the search model and gpt-4.1-mini the
held-out model, both through OpenRouter; 60 frozen asks drawn by the
offline writer from the six seeds, 30 train and 30 holdout split by seed, 4
rollouts each at the engine’s default sampling; the program judge; the
three checked-in candidates, one round, no candidate written by hand for
this run. The output, verbatim:
The gate:
02_check_result leads 30 of the 30 train tasks, and on the 30
held-out asks it beats the baseline by +0.38 [+0.27, +0.47] on Haiku
and by +0.09 [+0.04, +0.15] on gpt-4.1-mini, paired by task, both
intervals excluding zero, with no held-out task the baseline passed and the
pick failed. It costs 0.40x the baseline per rollout in tokens, so the
matched-cost margin holds with room. Attribution over the three-by-two
grid of holdout scores: the harness explains 61% of the spread (interval
47 to 77), the model 17% (10 to 25), and the same model leads under every
harness, so the gain is the harness and no ranking flipped.
Noise floor: the baseline harness was run four times on Haiku over the
same 30 held-out asks (out/tasks.jsonl replayed, nothing redrawn: this
run, an earlier search on the same set, and two re-runs of the baseline
alone). The four holdout means are 0.62, 0.65, 0.69 and 0.61,
eval_variance run_std 0.038, and the band a one-run-per-side delta has
to clear is 0.17 (wai.noise_band(run_std, df=3) = 3.18 x sqrt(2) x
run_std). +0.38 clears it; the +0.03 between the two winning candidates
does not, and the recipe does not call it a difference.
What moved, read off the rows. On Haiku the baseline failed 46 of 120
held-out rows, 40 of them for an apology (“I apologize for the
inconvenience” after a refund that went through), the rest for flattery
and boilerplate, and two for calling a refund done when the tool result
was stale or denied. The one-sentence rule in 01_no_filler removed
almost all of it (five rows left) and cut the mean reply from 761 to 180
characters; the read-the-tool-result rule in 02_check_result took the
last apologies with it and left one row. Tool calls per 120 rows went 193,
124, 134: the winners still look the order up, they stop narrating it. So
the search found the harness and named the mechanism: on a closed model, a
style the rubric forbids is a prompt line away, and the gate is what says
the line held on asks and on a model it was not written for, at a lower
cost. The ceiling this set leaves is 0.99; the next round is a harder set
(more seeds, more faults, a judge with more to catch), not a fourth
candidate.
Runs on the platform: https://while.ai/platform/runs?agent=meta-harness
(one iteration per candidate and model, each pinned to its harness
fingerprint; the harness x model grid carries the attribution sentence).
Checks
Learned
- The gate is the result. On the train split the third candidate led every task; on held-out asks the two winners sit three points apart inside a 17-point noise band, and the recipe reports that as a tie rather than the train-split order.
- On a closed model the lever is the line in the prompt, and it is worth measuring like a training arm: 62 to 99 of 100 with an interval, a noise floor, a held-out model and a cost ratio, for about two dollars of OpenRouter credit.
- A program judge fails for the reason it names. The baseline lost on apologies, not on tool use, and the trace tally says so before anyone reads a reply; the proposer’s next candidate should come from that tally.
- A judge that reads tool steps has to skip the turns that have none. The first version of
common.judgecounted a text-only turn as a failed tool result and flagged every “has been processed” as a false claim; the fix is onecontinue, and the dry-run numbers did not move because the scripted agent never emits a text-only turn.
The gate, in words
A candidate the proposer wrote from the train split’s worst rows has seen those rows. Its score there is the pick, not the proof. The pick is the candidate that leads the most train tasks, per task the best pass rate across candidates with ties shared, so a mean gained by regressing a subset does not win; that is the per-task frontier GEPA selects on [4]. The proof is the same harness on the tasks it never saw, on a model it was not tuned on, with an interval over tasks that excludes zero on both, and the count of held-out tasks the baseline passed every time and the pick failed every time is printed beside it. When--models names one model, the
held-out-model check is skipped and the selection says so.
The third check is cost. Wang et al. 2026 matched budgets and found harness
evolution lost to spending the same compute on more samples of the baseline,
and gained 0.6 points on held-out tasks when the harness was tuned on the
tasks it was scored on [5]. So the ledger carries cost per rollout (tokens
when every row has usage, model calls otherwise: the reply plus one per
tool call), and a pick may cost no
more than the baseline plus --cost-margin (0 by default: matched cost). A
candidate that wins by spending more is reported as a frontier point with
the margin that would accept it, not selected. wai.harness.attribute needs
at least two candidates and two models; with one model it is not printed.
Lambert 2025, chapter Evaluation, is the rule behind the split [2]; Miller
2024 is the interval over tasks [3].
Next
Writecandidates/03_<name>.py from what out/proposal.md says is still
wrong, and run python run.py --dry-run --propose --select again; the
skill in skills/harness-search is that
loop as a playbook for a coding agent, with the stop rule and the
Changed / Moved / Why / Learned / Reproduce report. With keys, name real
models: --models openai:gpt-4.1-mini,anthropic:claude-haiku-4-5, or the
OpenRouter form the Result section used, and a model judge with --judge anthropic:claude-haiku-4-5. On the checked-in set the ceiling is 0.99, so
the useful next run is a harder set: more seeds in common.py, a larger
--budget, or your own traffic through --traces. To keep the picked
harness as a version on the platform, harness.pin() is the record
tracked.run(harness=) takes (the harness page).
References
- Lee, Y., Nair, R., Zhang, Q., Lee, K., Khattab, O., Finn, C. Meta-Harness: End-to-End Optimization of Model Harnesses. arXiv:2603.28052, 2026.
- Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter Evaluation.
- Miller, E. Adding Error Bars to Evals. arXiv:2411.00640, 2024.
- Agrawal, L. A., et al. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. arXiv:2507.19457, 2025.
- Wang, Y., et al. Rethinking the Evaluation of Harness Evolution for Agents. arXiv:2607.12227, 2026.