Harness optimization from production traces

What you learn: how a coding agent runs a harness search on yesterday’s traffic, what it reads, what it writes, and the three checks a change has to pass. Needs: nothing for the dry run. Takes: seconds offline, minutes per candidate with a key.

Traces split by day, the agent writes one candidate from the worst rows, the gate checks the holdout, a second model and cost, the pick serves, and the next day is scored with the same judge

On a closed model the harness is the only thing you can change: the instructions, the tools, the turn cap, the retry, how context is built. Among comparable frontier models it explains more of the spread than the model does [1], and a harness searched with earlier candidates’ code and traces in view beat the hand-built ones on TerminalBench-2 [2]. The catch: tuned on the tasks it was scored on, harness evolution gained 0.6 points held out and lost to more samples of the baseline at the same budget [3]. The loop below is built around what the agent is not allowed to see. Runnable version: recipes/papers/meta-harness. Playbook for the coding agent: skills/harness-search. The object it searches over: the harness.

1. Traces in

Point the recipe at yesterday’s export: a JSONL in any shape load_traces reads, or an OTLP JSON batch, which rows_from_otel groups into conversations. One task per distinct ask. Nothing leaves your machine.

cd recipes/papers/meta-harness
python run.py --traces traces-2026-09-21.jsonl --propose --select --fresh

Rows carry the ask, the steps, the reply and a timestamp. Extra keys pass through.

from whileai.simulations import load_traces

rows = load_traces(
    [
        {
            "ts": "2026-09-19T09:00:00Z",
            "input": "Where is my refund for ORD-5412?",
            "output": "Certainly! It went through.",
        },
        {
            "ts": "2026-09-21T09:00:00Z",
            "input": "Refund ORD-8821, the jacket arrived torn.",
            "output": "Done.",
        },
    ]
)
print(len(rows), sorted(rows[0]))

2 ['final_text', 'input', 'output', 'prompt', 'steps', 'ts']

2. The split the agent cannot cross

The holdout is the latest days, whole days, until it holds half the tasks. The proposer reads earlier days only. Train asks that are near-copies of a holdout ask (decontaminate, the 8-gram rule) leave its window too. out/split.json records it, and the proposal opens by saying which days decide.

split: holdout is 2026-09-20 and later: 16 of 24 tasks; 1 train prompt(s) overlapped the holdout and left the proposer's window

3. What the agent reads

out/proposal.md: every candidate so far as code, its pass@1 on the train days with an interval over tasks, and its five worst rows. Nothing is summarized. The proposer’s edge in [2] was raw traces, not digests.

## 00_baseline.py

train pass@1 0.71 [0.52..0.85] on 8 tasks; holdout 0.67 [0.50..0.83]; fingerprint 71fed4102190

Worst rows:

- ask: I want my money back on ORD-5412 right now.
  reply: Certainly! Your refund went through.
  why: boilerplate in the reply; claimed success on a tool result that did not succeed

4. What the agent writes

One file, one change, chosen from the worst rows. A candidate defines harness(model) -> wai.Harness. The fingerprint hashes the instructions, the tool names and the Disclosure fields, so the edit is a new version without anyone naming it, and every row says which harness made it.

import whileai as wai
from whileai.harness import Disclosure

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "lookup_order",
            "description": "Look up an order by id.",
            "parameters": {
                "type": "object",
                "properties": {"order_id": {"type": "string"}},
                "required": ["order_id"],
            },
        },
    }
]
BASE = "Look the order up before you act."
scripted = wai.seeded_agent(TOOLS, rate=0.3, seed=0)  # stands in for the model offline

baseline = wai.Harness(
    agent=scripted, instructions=BASE, tools=TOOLS, label="baseline", model="scripted"
)
candidate = wai.Harness(
    agent=scripted,
    instructions=BASE + " Report what the tool returned; never claim success on an error.",
    tools=TOOLS,
    label="report_fault",
    model="scripted",
    disclosure=Disclosure(max_turns=4),
)
print(
    baseline.version,
    baseline.fingerprint,
    candidate.version,
    candidate.fingerprint,
    baseline.fingerprint != candidate.fingerprint,
)

baseline 71fed4102190 report_fault 93c49a7ba181 True

Then it runs the recipe again and reads out/selected.json.

5. The gate

Three checks, all printed.

The pick. The candidate that leads the most train tasks, per task the best pass rate across candidates with ties shared. Not the best mean: a mean gained by regressing a subset does not win [4].
Is it real. On the held-out days the pick beats the baseline with a paired interval that excludes zero, and again on a second model. The held-out tasks the baseline passed every time and the pick failed every time are counted beside it.
Did it just spend more. Cost per rollout is on the ledger: tokens when every row carries usage, model calls otherwise. A pick may cost no more than the baseline (--cost-margin 0). A candidate that wins by spending more is a frontier point, and the verdict names the margin that would accept it [3].

train tasks led: 00_baseline.py 5, 01_no_filler.py 7, 02_check_result.py 12 -> pick 02_check_result.py
holdout on scripted: 02_check_result.py vs 00_baseline.py +0.27 [+0.15, +0.42] over 12 paired tasks, 0 the baseline passed and the pick failed -> clears zero
holdout on scripted-b: 02_check_result.py vs 00_baseline.py +0.50 [+0.38, +0.62] over 12 paired tasks, 0 the baseline passed and the pick failed -> clears zero
cost per rollout: 02_check_result.py at 1.00x the baseline in calls -> within the margin
select: 02_check_result.py beats the baseline on the holdout and on a held-out model at 1.00x its cost

wai.harness.attribute over the candidate by model grid says whether the spread is the harness or the model. The agent loops on steps 3 to 5 until the gate passes or the rounds run out. A search that never clears is a result too.

6. Report, serve, next day

Every candidate is posted as a harness version, so the Runs page groups the dots by harness. Under the pick, five lines from the ledger: Changed, Moved, Why, Learned, Reproduce. You read one chart and decide. The holdout says what the pick can do on traffic it never saw. The next day says whether that survived. The skill ends by scoring the day after with the same judge and posting one LiveDay. A flagged rate near the holdout’s failure rate closes the loop. One well above it makes that day the new traces.

Moved: 67 to 73 points on 16 held-out tasks (holdout is 2026-09-20 and later), +6 [+2, +11]; 0 task(s) the baseline passed and it failed; 1.00x the cost per rollout.
next day: 21 of 100 flagged; the holdout said 27 of 100

When the harness is not enough

A search stalls when the model lacks something no prompt supplies. On a legal classification task, harness edits alone reached 50 points and harness plus weight updates reached 70 [5]. On an open model the same held-out test and judge carry over to training: evals, then SFT from traces or GRPO.

References

Zhang, Y. et al. Stop Comparing LLM Agents Without Disclosing the Harness. 2026. Harness variance 7.8x model variance; Disclosure follows its checklist.
Lee, Y. et al. Meta-Harness: End-to-End Optimization of Model Harnesses. 2026.
Wang, Y. et al. Rethinking the Evaluation of Harness Evolution for Agents. 2026. Matched budgets; +0.6 held out when tuned on the scored tasks.
Agrawal, L. A. et al. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. ICLR 2026. Selection on the per-task frontier.
Hebbar, P. et al. SIA: Self Improving AI with Harness and Weight Updates. 2026.
Lambert, N. Reinforcement Learning from Human Feedback, chapter Evaluation. 2025. The train split picks, the holdout decides.

Get started

Guides

Concepts

Recipes

Reference

Harness optimization from production traces

1. Traces in

2. The split the agent cannot cross

3. What the agent reads

4. What the agent writes

5. The gate

6. Report, serve, next day

When the harness is not enough

References

​1. Traces in

​2. The split the agent cannot cross

​3. What the agent reads

​4. What the agent writes

​5. The gate

​6. Report, serve, next day

​When the harness is not enough

​References

1. Traces in

2. The split the agent cannot cross

3. What the agent reads

4. What the agent writes

5. The gate

6. Report, serve, next day

When the harness is not enough

References