The scripts are in the repository, not in the installed package. Clone it,
then
cd recipes/community/deepagents-review-four-arms before running the commands below. Browse this recipe on GitHub.wai.simulations.optimize keeps. Nothing else differs. Both models are scored on SWE-bench Verified
patch review, a benchmark nobody here built, and on 84 repositories the traces never touched.
What you will learn: that which traces you train on moves a fine-tune as much as the trainer
does, how to hold a comparison to one variable (same trainer, same server, same sampling), and
why a single base run is not a baseline. You need OPENROUTER_API_KEY and LANGSMITH_API_KEY
for traces, Docker or Modal for smithtune (it imports fcntl, so Linux or macOS; smithtune_modal.py
runs it on Modal), a Baseten key for its default council, and a Modal account for
training and serving. python run.py --dry-run needs none of it.
Result
The fair rerun (PREREGISTRATION.md, merged before training). The smithtune side follows
smithtune’s own skill: read 20 traces, co-write the rubric (rubric_codesigned.md), check it on
a 20-trace trial (calibration/), then run the council. Both sides are cut to the same size,
fine-tuned with smithtune’s renderer, split and schedule for exactly 2 epochs, three seeds each,
and scored on one server. Each arm pools its three seeds; comparisons are paired by review.
Experiment A: same 262 traces, two ways to pick 174.
Every While seed beats every smithtune seed on both benchmarks (9 of 9 pairs). The two picks share
142 of their 174 traces: the co-designed rubric kept 184 of 260 where the first rubric kept 258,
so reading traces with an agent moves smithtune most of the way to While’s pick. The 32 traces
that differ are the long correct reviews the council still keeps, and a model trained on them runs
out of steps without a verdict on 96 to 165 of 250 reviews, against 45 to 65 for While’s.
Experiment B: 400 more rollouts per side. smithtune’s side drew 400 unused tasks at random
(it cannot aim); While’s drew 400 aimed at the stock agent’s weak spots (
aim.py). Each side added
its rollouts to the original pool and picked with its own method; both cut to 317 traces.
While’s side is ahead by +17.3 [+13.9, +20.9] on SWE-bench Verified (47.5% against 30.1%) and
+13.7 [+10.3, +17.4] on the held-out repos, 8 of 9 seed pairs. Neither side beats its own
Experiment A version, so the extra 400 rollouts did not help either side; B separates the
pickers, not the value of aiming.
The registered schedule. As first registered, training stopped when validation loss stopped
improving (smithtune’s default). It kept epoch 1 on 10 of 12 runs, on losses about 0.001 apart
from epoch 2, and epoch-1 checkpoints mostly run out of steps. Experiment A under that schedule:
+9.5 [+6.4, +12.8] for While’s pick, with seed accuracies from 22% to 59% inside one arm. It is
published as registered (rows/srv-a-*-s*); amendment 1 moved both experiments to 2 epochs
before Experiment B was scored.
All numbers are in results.json under rerun (python make_results.py).
The first result (2026-09-28, one seed per arm, smithtune rubric written without reading
traces, council kept 258): SWE-bench Verified 46.0% against 56.8%, +10.8 [+4.8, +16.8], with 5.3
fewer calls; held-out repos a tie (63.8% against 61.4%). Base model, three runs: 11.6%, 12.0%, 9.6%.
The write-up with charts: Deep Agents SFT from traces.
How it works
data.pybuilds the review tasks fromnebius/SWE-agent-trajectoriesandnebius/SWE-bench-extra(pinned revisions): for each issue with a passing and a failing SWE-agent patch, one of each. Splits are by repository.wai.decontaminatechecks train and dev against the holdout (0 of 1,306 dropped, every rule ran). Four snapshots deleted from GitHub are listed inunavailable.jsonand left out everywhere.collect.pyruns the stock agent (agent.py:create_deep_agent, read-only filesystem over a tarball of the base commit) and writes LangSmith feedbackcorrectnessandmodel_callson every root run. 400 training reviews: 67% correct.- Without While:
smithtune dataset pullwith the correctness filter, thensmithtune dataset triagewithrubric.md. The council kept 258 of 260. - With While:
pick.pyrunswai.simulations.optimize(mode="sft")with reward 1 only for a correct review within 8 model calls, and writes feedbackwhile_pick = 1; smithtune pulls it with that filter. 173 kept. export_sft.pyruns inside the smithtune image and calls smithtune’sprepare_sft_rows(reasoning omitted, its default),split_rows(80/10/10) andrender_row_tokens(the Baseten renderer: one datum per assistant turn, history at weight 0).smithtune prepareitself stops without a Baseten Loops or Fireworks account; this skips only that preflight.train_modal.pytrains LoRA on one H200 with smithtune’s Baseten defaults: rank 8, batch 32, learning rate 1e-4, seed 42, up to 5 epochs, stop when validation loss does not improve, keep the best epoch. Both arms stopped after epoch 3 and kept epoch 2.serve_modal.pyserves the base and both adapters from one vLLM 0.26 server with the token-level Qwen3.5 tool parser (qwen35_tool_parser.py) andenable_thinking=falsefor every request.public_bench.pybuilds the SWE-bench Verified set from six public leaderboard submissions (s3://swe-bench-submissions,patch.diffandreport.json), none of whose twelve repositories is in the training traces.report.pyandmake_results.pydo the paired comparisons.
Run it
Run the fair rerun
Every step ofPREREGISTRATION.md, in order. smithtune_modal.py runs the smithtune CLI in the
pinned image on Modal; Docker works the same way with smithtune.Dockerfile.
<source> is the LangSmith workspace, project, window and correctness filter, written out in
PREREGISTRATION.md.
Caveats
- Three seeds per arm. Seed spread inside one arm is up to 18 points even at a fixed 2 epochs (smithtune skill path on SWE-bench Verified: 24.4% to 42.4%), so the paired intervals, which bootstrap over reviews, understate the full uncertainty. The seed-pair counts (9 of 9 in A) are the check that does not lean on them.
- The schedule changed after Experiment A was scored (amendment 1, early stopping to 2 epochs). The registered result is published beside it and points the same way, +9.5 [+6.4, +12.8].
- smithtune’s converter dropped one While-pick trace in each experiment for a structural problem, so While’s side trained on one fewer trace (173 against 174, 316 against 317).
- smithtune omits reasoning from training rows by default, so every tuned model is non-thinking and is scored with thinking off. With thinking on, through OpenRouter, the base scored 70.3%, 75.1% and 74.6% on the held-out repositories.
- Training ran on Modal, not Baseten Loops (not enabled for our workspace), with smithtune’s renderer, split and schedule. The council ran on Baseten, smithtune’s default.
- An agent read the 20 traces and drafted
rubric_codesigned.md; a person approved it from a summary. The LangChain team is invited to send a rubric of their own, and we will run it the same way.
Next
Run a rubric written by the smithtune team through the same pipeline. Then the thinking-on base on SWE-bench Verified. Traces keep flowing to LangSmith, so the samepick.py picks the next round.