> ## Documentation Index
> Fetch the complete documentation index at: https://docs.while.ai/llms.txt
> Use this file to discover all available pages before exploring further.
> ## Agent Instructions
> Install with `uv add whileai`; import as `import whileai as wai`.
> Run the offline path first (`simulator=False`, `wai.seeded_agent`, a callable judge); no key is needed for it.
> Report every pass rate with its interval and n, as `scored.pass_at` prints it.
# Harness and weights: both levers against either alone
> Paper: Harness-Aware Self-Evolving: Co-Evolving Model Weights, Harness, and Task Solutions (HASE), Haochen Luo et al., arXiv:2607.03935, July 2026.
The scripts are in the repository, not in the installed package. Clone it,
then `cd recipes/papers/harness-and-weights` before running the commands below. [Browse this recipe on GitHub](https://github.com/whilehq/whileai-sdk/tree/main/recipes/papers/harness-and-weights).
**Paper:** Harness-Aware Self-Evolving: Co-Evolving Model Weights, Harness, and Task Solutions (HASE), Haochen Luo et al., arXiv:2607.03935, July 2026. [https://arxiv.org/abs/2607.03935](https://arxiv.org/abs/2607.03935). With SIA: Self Improving AI with Harness & Weight Updates, Prannay Hebbar et al., arXiv:2605.27276 \[2], and Prime Agent: A Self-Improving RLM Harness, Seth Karten et al., arXiv:2608.23552 \[3].
**Book:** the reward is a program that runs the code against hidden tests, the rollouts are grouped per prompt, and the harness (the tool, the turn cap, the text the model sees) is part of the setup a score is only comparable under \[4], \[5].
**Claim:** HASE's abstract: "a single model can generate task solutions or edit selected harness components in a multi-turn action space", and this "enables a single Qwen3-8B model to match the text-classification performance of a GPT-OSS-120B model that uses Claude Code as the harness proposer. In alpha factor mining, HASE outperforms the reported GPT-OSS-120B baseline. HASE also repairs imperfect evaluation components and converges to state-of-the-art performance in circle-packing algorithm discovery." SIA's abstract, on Chinese legal charge classification, GPU kernel optimisation and single-cell RNA denoising: "Combining both levers outperforms scaffold iteration alone on all three benchmarks." Prime Agent moves only the harness ("A persistent IPython REPL ... while Continual Harness preserves histories, memories, skills, prompts, and subagent specifications across trajectories") and "raises ARC-AGI-3 RHAE Best\@1 from 30% to 95.5%" with Opus 5; it trains no weights.
**The change:** the `both` arm trains the weights under the searched harness and is evaluated under it; the `harness` arm searches only the harness, the `weights` arm trains only the weights, and `neither` does neither. All four on the same held-out tasks.
What you will learn: how to put the two levers under one measurement. A
harness here is a `wai.Harness` with a skills text in its instructions
(Prime Agent's idea: the harness carries a SKILLS.md the proposer edits
between rounds \[3]), an optional `run_python` tool and a turn cap; the
search over harnesses is the Meta-Harness loop imported from
[`../meta-harness`](/recipes/papers/meta-harness) (ledger, proposal, gate \[6]). The
weights are GRPO with TRL and vLLM colocate, the rollouts under the current
harness's instructions, the reward `wai.verify.CodeExec` on hidden tests.
`wai.harness.attribute` over the 2x2 grid (base and trained weights, under
the baseline and the searched harness) says which lever moved the score.
You need nothing for `--dry-run`. `--smoke` and the full run need Modal
(`MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET`, or `modal token set`) and one
H100; `WHILEAI_API_KEY` is optional and posts the four arms to the platform.
## Recipe
1. Base: `Qwen/Qwen2.5-Coder-1.5B-Instruct` (`--base`). The first smoke ran the plain `Qwen/Qwen2.5-1.5B-Instruct` and it passed 0, 4 and 2 of 128 rollouts under the three harnesses at temperature 1.0 (half the baseline's replies timed out, the rest were NameError, TypeError and wrong numbers): a floor GRPO gets almost no signal from, so the coder variant of the same size is the default and that measurement is the reason. Data: `tasks.py` builds 143 quant coding tasks in 11 families over one seeded price table (eight tickers, 500 trading days, `random.Random(0)`, standard library only, so the dry run, CI and the grader's sandbox all build the same bars). Every task is one function over the table: max drawdown, rolling Sharpe, RSI, realized volatility, a momentum factor's Spearman rank correlation with next-day returns, a top-k long-short return, golden crosses, beta, ATR, VWAP, the longest up streak, with the ticker and the window varied by template. Each carries `privileged.tests`, asserts against the reference implementation's answer computed at build time; the answer is never in the prompt. The split is by family: 5 families train (64 tasks), 6 are held out (79), and `decontaminate(train, against=holdout)` drops 0. Split by task instead (`--split task`) and the same check drops 71 of 71, because two tasks from one template are near-copies under the 8-gram rule \[4]; that is why family is the default, and it is also the harder question, since the holdout asks for quantities training never showed.
2. Lever 1, the harness. `candidates/*.py`, in name order, each `harness(model) -> wai.Harness`: `00_baseline` is the bare instructions, no skills, no tool, one turn; `01_skills` adds a SKILLS.md text (how to read the table, sample standard deviation, what to return); `02_skills_tool` adds `run_python` (the model's code on the seeded table, the printed result back as the next message) and three turns. Every candidate plays every task `--k` times on the base weights; `ledger.jsonl`, `proposal.md` and the gate (`compare_runs` on the holdout, interval excluding zero) are the Meta-Harness recipe's own functions \[6]. The gate's pick is the searched harness; when no candidate clears it, the best on the train split is used and the run says so.
3. Lever 2, the weights. GRPO with TRL `GRPOTrainer` and LoRA, vLLM colocated for rollouts, one H100, the zero-rl-format-reward recipe's image and its memory fix (half a group per forward, twice the accumulation, `expandable_segments`). 40 optimizer steps, 8 generations, 6 prompts a step, 1024 completion tokens, lr 1e-4, KL 1e-4, `dapo` loss. The system prompt is the harness's instructions; the reward is `CodeExec` on the hidden tests. `weights` trains under `00_baseline`; `both` trains under the searched harness.
4. Eval: pass\@1 on the same held-out tasks, `--k` samples a task at temperature 1.0, each arm played under its harness (tool loop and all). The untrained base under the baseline harness is evaluated `--eval-runs` times for the noise floor (`eval_variance`). Four cells, paired by task (`compare_runs`): `harness` vs `neither`, `weights` vs `neither`, `both` vs `harness`, `both` vs `weights`. `wai.harness.attribute` over the full grid.
5. One training seed per arm: the verdict on any trained pair is unresolved until each trained arm has a second seed (`--train-seed`).
## Run
```bash theme={"theme":"vitesse-dark"}
cd recipes/papers/harness-and-weights
python recipe.py --dry-run # offline: tasks, the search on scripted candidates, the grid; under a minute
python recipe.py --smoke # Modal: 2 GRPO steps, 16 tasks a split; the live path end to end, under a dollar
python recipe.py # the full run: about 40 H100 minutes, under $5; writes results.json
```
On Windows, from the recipe directory: `PYTHONUTF8=1 PYTHONIOENCODING=utf-8 uv run --with modal --with datasets python recipe.py --smoke`.
| flag | default | what it does |
| -------------------- | -------------------------------------- | ---------------------------------------------------------------------------- |
| `--dry-run` | off | offline: scripted candidates, no GPU; the weights arms are marked not run |
| `--smoke` | off | Modal: `2` steps, `16` tasks a split, no results.json |
| `--k` | 4 | rollouts per task at eval; pass\@1 averages them, the interval is over tasks |
| `--steps` | 40 | optimizer steps per trained arm |
| `--generations` | 8 | rollouts per prompt, the GRPO group |
| `--prompts-per-step` | 6 | prompts per optimizer step: 48 rollouts a step |
| `--lr` | 1e-4 | LoRA GRPO learning rate, the filter-metric recipe's setting on this base |
| `--beta` | 1e-4 | KL coefficient, SimpleRL-Zoo's for models up to 14B |
| `--max-completion` | 1024 | completion tokens |
| `--lora-rank` | 32 | LoRA rank, alpha 2r, every linear layer |
| `--base` | Qwen/Qwen2.5-Coder-1.5B-Instruct | the base model, a Hub id (see step 1 for why not the plain Instruct model) |
| `--eval-runs` | 3 | base re-runs under the baseline harness, the noise floor |
| `--seed` | 0 | the table, the split, the scripted stand-ins |
| `--train-seed` | 17 | the adapter init and the sampler, both trained arms |
| `--holdout` | 0.5 | share of tasks held out |
| `--split` | family | `family` holds out whole templates; `task` mixes them (see step 1) |
| `--limit` | none (16 in `--dry-run` and `--smoke`) | tasks per split to play; the build is always the whole set |
| `--candidates` | candidates | the folder of candidate files |
| `--out` | out | tasks, ledger, traces, proposal, grid |
| `--fresh` | off | drop `out/` first |
| `--reuse` | off | take a paid stage from `.cache/` when it is there |
## A candidate
One file, one change. This is `candidates/02_skills_tool.py`: the skills
text from candidate 01 plus what the tool is for, the `run_python` tool, and
three turns (run, read, answer). `build` in `harnesses.py` turns it into the
harness a vLLM engine plays, or the scripted stand-in when the model name
starts with `scripted`.
```python theme={"theme":"vitesse-dark"}
"""Candidate 02: the skills file plus a REPL. Candidate 01's worst rows are
off-by-one on the window or the SMA alignment: the code runs, the number is
wrong. This candidate gives the model `run_python` (its code on the seeded
table, the printed result back) and three turns: run, read, answer. The
skills text grows by what the tool is for (Karten et al. 2026: a persistent
REPL plus skills the harness carries)."""
from __future__ import annotations
from harnesses import build
import whileai as wai
SKILLS = """
- Filter to the one ticker first: `rows = [b for b in bars if b["ticker"] == ticker]`. The
rows are already in date order, so `closes = [b["close"] for b in rows]`.
- A simple return is `closes[i] / closes[i - 1] - 1`; a log return is `math.log(closes[i] /
closes[i - 1])`. There is one fewer return than closes.
- "Last N" means the slice `[-N:]` of the returns or rows, not of the closes.
- Sample standard deviation divides by n - 1: `math.sqrt(sum((x - m) ** 2 for x in xs) /
(len(xs) - 1))`.
- Cross-sectional means across tickers on one day: build `{ticker: closes}` once, then loop
over day indices.
- Return a plain `float` (or `int` where the task says so). Do not read files.
- Test first: define the function, `print(fn(BARS, ...))` with the task's arguments, and read
the output. A traceback names the bug; a number that is not a plain float or int means the
return type is wrong. Then send the final block without the print.
- When two windows meet (a fast and a slow average), line them up on the same day index
before comparing; check the count on a small slice before trusting it on the table.
"""
TOOL = True
# Offline stand-in: only the occasional definition slip is left.
SCRIPTED_RATE = 0.75
def harness(model: str) -> wai.Harness:
return build(
model, skills=SKILLS, tool=TOOL, label="02_skills_tool", scripted_rate=SCRIPTED_RATE
)
```
The fingerprint hashes the instructions (skills text included), the tool
name and the `Disclosure` fields, so an edit to SKILLS is a new harness
version, and every row it produces carries it. A block with a `print` goes
to the tool (the table is `bars` there, and `BARS`); a block without one is
the answer. The proposer is you, or the
coding agent running [`skills/harness-search`](https://github.com/whilehq/whileai-sdk/tree/main/skills/harness-search):
read `out/proposal.md`, edit the skills text into `candidates/03_.py`,
run again.
## What you get
`python recipe.py --dry-run`, offline, the three checked-in candidates on
the scripted stand-in, 16 tasks a split, 4 rollouts a task:
```text theme={"theme":"vitesse-dark"}
tasks: 143 in 11 families, split by family: 64 train, 79 holdout; decontaminate dropped 0 train rows
playing 16 train and 16 holdout tasks (--limit)
00_baseline train 0.30 [0.19..0.41] holdout 0.41 [0.31..0.48] tool calls 0
01_skills train 0.52 [0.36..0.66] holdout 0.64 [0.50..0.77] tool calls 0
02_skills_tool train 0.69 [0.55..0.81] holdout 0.80 [0.69..0.89] tool calls 128
ledger: out/ledger.jsonl (3 candidates, 1 model)
holdout on base: 02_skills_tool.py vs 00_baseline.py +0.39 [+0.27, +0.50] over 16 paired tasks -> clears zero
select: 02_skills_tool.py beats the baseline on the holdout
searched harness: 02_skills_tool
neither pass@1 0.41 [0.31..0.48] over 16 held-out tasks harness=00_baseline model=base
harness pass@1 0.80 [0.69..0.89] over 16 held-out tasks harness=02_skills_tool model=base
weights not run (dry run: no GPU)
both not run (dry run: no GPU)
attribution skipped: attribution needs at least 2 harnesses and 2 models; got 2 harness(es) ['00_baseline', '02_skills_tool'] and 1 model(s) ['base']. One level has no spread to attribute; run the missing arm
harness vs neither: +0.39 [+0.27, +0.50] over 16 paired tasks -> clears zero
wrote out/dry_run.json; no results.json from a dry run
```
On the scripted stand-in these numbers are what the planted rates make
them: they show the loop (build, freeze, search, gate, the grid with its two
holes named, the paired pair that can be computed) and are not a
replication. The grader is real: every row is the candidate's code plus the
hidden tests in a fresh interpreter, and the tool candidate's 128 steps are
128 real `run_python` calls.
`python recipe.py --smoke`, Modal, one H100, 2026-09-21: 2 GRPO steps a
trained arm, 16 tasks a split, 4 rollouts a task, the coder base. This is
the live path proven end to end, not a result; at two steps the trained
cells are the base cells:
```text theme={"theme":"vitesse-dark"}
search: 00_baseline 1/128 rollouts passed
search: 01_skills 9/128 rollouts passed
search: 02_skills_tool 6/128 rollouts passed
search: 5.3 GPU minutes
00_baseline train 0.00 [0.00..0.00] holdout 0.02 [0.00..0.05] tool calls 0
01_skills train 0.02 [0.00..0.05] holdout 0.12 [0.03..0.25] tool calls 0
02_skills_tool train 0.03 [0.00..0.08] holdout 0.06 [0.00..0.14] tool calls 64
holdout on base: 02_skills_tool.py vs 00_baseline.py +0.05 [-0.02, +0.12] over 16 paired tasks -> could be chance
select: 02_skills_tool.py is best on train but does not clear the baseline on the holdout; write the next candidate
searched harness: 02_skills_tool (best on train; the gate did not pass on the holdout)
neither pass@1 0.02 [0.00..0.05] over 16 held-out tasks harness=00_baseline model=base
harness pass@1 0.06 [0.00..0.14] over 16 held-out tasks harness=02_skills_tool model=base
weights pass@1 0.02 [0.00..0.05] over 16 held-out tasks harness=00_baseline model=trained
both pass@1 0.06 [0.00..0.14] over 16 held-out tasks harness=02_skills_tool model=trained
attribution on pass_at_1: 2 harnesses x 2 models, 16 tasks each cell
harness vs neither: +0.05 [-0.02, +0.12] over 16 paired tasks -> could be chance
weights vs neither: +0.00 [+0.00, +0.00] over 16 paired tasks -> could be chance
both vs harness: +0.00 [-0.08, +0.09] over 16 paired tasks -> could be chance
both vs weights: +0.05 [+0.00, +0.09] over 16 paired tasks -> could be chance
wall clock: 10.4 GPU minutes, $0.69 on H100
wrote out/smoke.json; a smoke run claims no number and writes no results.json
```
The GPU minutes are the recipe's own clock over the three function bodies
(engine start, play, grade, train); Modal bills container time, which adds
the cold start of each of the three containers. The two training steps
logged a mean reward of 0.04 and 0.06 with two thirds of the groups at zero
standard deviation: on this task set the coder base passes about one
rollout in twenty at temperature 1.0, which is the floor the full run
climbs from.
The full run's numbers: see the Result section below.
## Result
`python recipe.py`, the full run, 2026-09-21, one H100 on Modal, 79 held-out
tasks in six families the training never showed, 4 rollouts a task, 40
GRPO steps a trained arm. The output, verbatim:
```text theme={"theme":"vitesse-dark"}
tasks: 143 in 11 families, split by family: 64 train, 79 holdout; decontaminate dropped 0 train rows
search: 00_baseline 15/572 rollouts passed
search: 01_skills 48/572 rollouts passed
search: 02_skills_tool 25/572 rollouts passed
search: 9.2 GPU minutes
00_baseline train 0.05 [0.02..0.09] holdout 0.00 [0.00..0.01] tool calls 0
01_skills train 0.08 [0.04..0.12] holdout 0.09 [0.05..0.13] tool calls 0
02_skills_tool train 0.04 [0.02..0.07] holdout 0.05 [0.02..0.08] tool calls 288
ledger: out/ledger.jsonl (3 candidates, 1 model)
holdout on base: 01_skills.py vs 00_baseline.py +0.08 [+0.05, +0.12] over 79 paired tasks -> clears zero
select: 01_skills.py beats the baseline on the holdout
searched harness: 01_skills
weights: pass@1 0.03 [0.01..0.04] | pass^4 (pass_pow_k) 0.00 [0.00..0.00] | pass@4 0.09 [0.04..0.15] | headroom 0.06 (79 groups, k=4)
weights hack scan: top feature contains:= sum
both: pass@1 0.09 [0.06..0.12] | pass^4 (pass_pow_k) 0.00 [0.00..0.00] | pass@4 0.30 [0.20..0.41] | headroom 0.22 (79 groups, k=4)
both hack scan: top feature contains:) / AND contains:calculate the
neither pass@1 0.00 [0.00..0.01] over 79 held-out tasks harness=00_baseline model=base
harness pass@1 0.09 [0.05..0.13] over 79 held-out tasks harness=01_skills model=base
weights pass@1 0.03 [0.01..0.04] over 79 held-out tasks harness=00_baseline model=trained
both pass@1 0.09 [0.06..0.12] over 79 held-out tasks harness=01_skills model=trained
attribution on pass_at_1: 2 harnesses x 2 models, 79 tasks each cell
harness base trained
00_baseline 0.3 2.5
01_skills 8.5 8.5
spread explained: harness 95% [71..99], model 2% [0..24], interaction 2%
harness moves the score by up to 7.1 points, the model by up to 1.1
model ranking flips across harnesses
the harness moved the score more than the model did; the leading model changes with the harness, so a model ranking from one harness does not carry
harness vs neither: +0.08 [+0.05, +0.12] over 79 paired tasks -> clears zero
weights vs neither: +0.02 [+0.01, +0.04] over 79 paired tasks -> clears zero
both vs harness: +0.00 [-0.04, +0.04] over 79 paired tasks -> could be chance
both vs weights: +0.06 [+0.03, +0.09] over 79 paired tasks -> clears zero
wall clock: 36.1 GPU minutes, $2.38 on H100
```
| Arm | pass\@1 | 95% CI | pass\@k | Steps | GPU min |
| ---------------------------------------------------------- | ------- | --------------- | ------- | ----- | ------- |
| Base, no training (`neither`: base weights, `00_baseline`) | 0.003 | \[0.000, 0.009] | 0.013 | 0 | 0 |
| `harness` (base weights, `01_skills`) | 0.085 | \[0.051, 0.127] | 0.253 | 0 | 0 |
| Baseline (`weights`: trained under `00_baseline`) | 0.025 | \[0.009, 0.044] | 0.089 | 40 | 15.9 |
| Recipe (`both`: trained under `01_skills`) | 0.085 | \[0.057, 0.117] | 0.304 | 40 | 11.0 |
Recipe vs baseline (`both` vs `weights`): **+0.060 \[+0.028, +0.095]** over
79 paired tasks. Verdict: **unresolved**. One training seed per arm; a second
seed on each trained arm (`--train-seed`) passed as `train_runs=` would
resolve it to moved or flat. The eval checks pass: the interval excludes
zero and the delta clears the eval's own re-run band (run\_std 0.0032 from 3
base re-runs, band 0.019 = t(df=2) 4.30 x sqrt(2) x run\_std).
The claim under test was that `both` beats `harness` and `weights`. It
beats `weights`. It does not beat `harness`: +0.00 \[-0.04, +0.04] on the
same 79 tasks. The trained weights under the skills harness land exactly
where the base weights under the skills harness land (0.085 both), and the
attribution says the same thing from the other side: the harness explains
95% of the spread across the four cells (interval 71 to 99), the model 2%
(0 to 24), and the model ranking flips (under the bare harness the trained
weights lead, under the skills harness they tie). On this base, this task
set and this budget, the skills text did what forty GRPO steps did, and
adding the steps to the text added nothing the interval can see. That is a
flat result on the papers' claim and it is reported as one.
What the search found is its own result. The gate picked `01_skills`, the
skills text with no tool: +0.08 \[+0.05, +0.12] over the bare baseline on
the holdout, on the base weights, with no training at all. `02_skills_tool`,
the same text plus `run_python` and three turns, did worse than the text
alone (0.05 against 0.09 on the holdout) while making 288 tool calls: the
1.5B model runs its code, reads a traceback or a number, and does not
repair it. Prime Agent's REPL \[3] pays off for a model that can act on what
it reads; this one cannot yet. The proposer's next candidate should say
what to do with the output, not only that the tool exists.
The trained weights under the bare harness moved +0.02 \[+0.01, +0.04] from a
base of 0.003, and their replies grew from 821 to 1,623 characters; under
the skills harness the trained replies are 1,130 characters against the
base's 800. The training reward on the train split climbed from 0.04 to
above 0.5 by step 30, so the policy learned the five training families;
what carried to the six held-out families is the two points above.
Runs on the platform: [https://while.ai/platform/runs?agent=harness-and-weights](https://while.ai/platform/runs?agent=harness-and-weights) (four versions of the agent `harness-and-weights`, behavior `quant_code`, each with its harness fingerprint and the run record).
## Checks
| Check | Source | Result |
| ------------------------------------------------------------------------------------------- | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Eval noise: the base evaluated 3 times under the baseline harness, `eval_variance` run\_std | \[4] | **run\_std 0.0032 from 3 re-runs**; a delta under 0.019 (`wai.noise_band(run_std, df=2)` = 4.30 x sqrt(2) x run\_std) is noise. `both` vs `weights` +0.060 clears it; `both` vs `harness` +0.00 does not |
| Holdout is clean: `decontaminate(train, against=holdout)` | \[7] | **0 of 64 train rows dropped** with the family split (71 of 71 with `--split task`, which is why family is the default) |
| Reward is a program, not a judge | \[7] | `CodeExec` on the hidden asserts, the table builder prepended; no model in the reward path, in training or in eval |
| Proxy vs target: `compare(proxy=)` | \[8] | the training reward is the target metric (the same tests on the train families), so there is no proxy gap to declare; `over_optimized` false |
| Length: mean completion length before -> after, per arm | \[8] | **821 chars base -> 1,623 `weights`, 1,130 `both`**; the `harness` cell is 800 |
| Hack scan on the last training batch: `hack_scan` | \[8] | `weights`: top feature `contains:= sum`; `both`: `contains:) / AND contains:calculate the`. Surface features of code that computes something, not a shortcut; nothing endorsed |
| Pinned: seed, torch, transformers, trl, peft, vllm | [the contract](/recipes/papers#the-contract) | seed 0, train seed 17; torch 2.13.0, transformers 5.17.0, trl 1.13.0, peft 0.21.0, vllm 0.29.0 |
Two notes on the grid. The `trained` level of the model axis is two
separately trained adapters, one under each harness (that is what
co-training means in HASE and SIA), so the attribution's model axis reads
"trained under the harness it is evaluated under", not one adapter moved
between harnesses. And the base model's own two cells come from the same
search rows that picked the harness, so the `harness` cell is the gate's
holdout evidence, not a fresh draw; the `weights` and `both` cells are fresh.
## Climb
| Round | What changed | pass\@1 | vs previous |
| ----- | ------------------------------------------------------------- | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| 0 | dry run, scripted stand-in, 16 tasks a split | `harness` 0.80, `neither` 0.41 (planted rates) | not a result |
| smoke | `Qwen/Qwen2.5-1.5B-Instruct`, 16 tasks a split, search only | 0, 4 and 2 of 128 rollouts under the three harnesses | floor; base switched to the coder variant |
| smoke | `Qwen/Qwen2.5-Coder-1.5B-Instruct`, 2 steps, 16 tasks a split | `neither` 0.02, `harness` 0.06, `weights` 0.02, `both` 0.06 | the path, not a result; 10.4 GPU min, \$0.69 |
| 1 | the full run: 40 steps, 79 held-out tasks | `neither` 0.003, `harness` 0.085, `weights` 0.025, `both` 0.085 | `both` vs `weights` +0.060 \[+0.028, +0.095]; `both` vs `harness` +0.00 \[-0.04, +0.04]; unresolved, 1 seed |
Round 2 is a second training seed on both trained arms, which is what
resolves the verdict. Round 3 is the proposer's next candidate: a skills text
that says what to do with the tool's output, since the REPL alone lost to
the text alone.
## Learned
* The two arms the papers do not run are the ones that make the claim testable: `harness` alone and `weights` alone, on the same holdout, paired by task. Without them "both beats scaffold iteration alone" has no interval. With them, on this setup, `both` beat `weights` by six points and `harness` by none.
* A skills text is cheaper than forty GRPO steps and, here, worth the same: +0.08 from a text edit on the base weights, +0.02 from training under the bare harness, and training under the text added nothing to the text. The attribution report says the harness explains 95% of the spread. Search the harness first; train the weights when the harness has stopped moving.
* A REPL is only a lever for a model that acts on what it reads. The 1.5B coder made 288 tool calls and scored below the same skills text without the tool. Prime Agent's gains \[3] came with Opus 5; the tool is not the mechanism, the reading is.
* A templated task set and an 8-gram contamination rule do not mix: split by task, and every train prompt is a near-copy of a holdout prompt. Split by template family, and the holdout is clean and harder. The recipe measures both and says which it uses.
* A skills file in the harness is a prompt edit with a fingerprint: `wai.Harness` makes the version without anyone naming it, so the proposer can edit SKILLS.md between rounds and every row still says which text it ran under.
Verified 2026-09-21, whileai 0.114, TRL 1.13.0 + PEFT 0.21.0 on torch 2.13.0, vLLM 0.29.0. 36.1 GPU minutes, \$2.38 on one H100 by the recipe's clock (function bodies; Modal bills the three cold starts too). Runs: [https://while.ai/platform/runs?agent=harness-and-weights](https://while.ai/platform/runs?agent=harness-and-weights)
## Artifacts on Hugging Face
| what | repo |
| ------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| `both` (root, trained under `01_skills`) and `weights/` (trained under `00_baseline`) | [`while-ai/paper-harness-and-weights-1.5b`](https://huggingface.co/while-ai/paper-harness-and-weights-1.5b) |
Part of the [Papers, replicated](https://huggingface.co/collections/while-ai/papers-replicated-6ab271de22542eb550d4251c) collection in the while-ai org.
## References
1. Luo, H., Huang, Y., Luo, S., Liu, F., Li, L., Hu, Z., Feng, J., Liu, Q. Harness-Aware Self-Evolving: Co-Evolving Model Weights, Harness, and Task Solutions. arXiv:2607.03935, 2026.
2. Hebbar, P., Manawat, Y., Verboomen, S., Ivanova, A., Palanimalai, S., Bhatia, K., Baskaran, V. SIA: Self Improving AI with Harness & Weight Updates. arXiv:2605.27276, 2026.
3. Karten, S., Zhang, A. L., Thomas, K., Müller, S., Bakouch, E., Auras, D., Senghaas, M., Obeid, F., Dunas, K., Hagemann, J., Jaghouar, S. Prime Agent: A Self-Improving RLM Harness. arXiv:2608.23552, 2026.
4. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter *Evaluation*.
5. Zhang, Wang, Ge, Xu, Hamm, Reddy. Stop Comparing LLM Agents Without Disclosing the Harness. arXiv:2605.23950, 2026.
6. Lee, Y., Nair, R., Zhang, Q., Lee, K., Khattab, O., Finn, C. Meta-Harness: End-to-End Optimization of Model Harnesses. arXiv:2603.28052, 2026.
7. Lambert, N. et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124, 2024.
8. Gao, L., Schulman, J., Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML 2023. arXiv:2210.10760.