Skip to main content
The scripts are in the repository, not in the installed package. Clone it, then cd recipes/community/which-half-can-a-small-model-take before running the commands below. Browse this recipe on GitHub.
I have one production agent and a cost line to defend. Its job splits in two: pick the tool (interesting) and, given what the tool returned, write the reply (boring). This asks whether a small open model can take over just the boring half, and measures it on a frozen held-out set before anything ships. The answer turned out to be about where the split is, not whether one exists — and the number that carries the finding is the untrained model’s, not the trained one’s.

​
Result

Qwen/Qwen2.5-1.5B-Instruct, LoRA SFT on 129 rows (24 distinct asks) from the incumbent’s own clean traces on the boring half only. One L40S, 159 seconds, $0.09. Paired on the same held-out asks, 8 rollouts each; noise floor from three passes of identical untrained weights on the same container. Against the incumbent, paired on the same asks:

​
The caveat that comes first

The “beats the incumbent” rows are close to tautological, and should not be quoted on their own. The incumbent here injects stylistic flaws — apology, hedging, sycophancy, boilerplate, leaking internal notes, claiming success after a fault — at a fixed 35% rate, uniformly, independent of how hard the ask is. Training on the subset a program marked clean therefore produces a model that does not do the thing that was filtered out. Any imitator would win. A real production agent’s failures correlate with difficulty; these do not, so this run says nothing about whether a small model can match a frontier agent on genuinely hard asks.

​
The number that is not tautological

The base, untrained 1.5B scores 0.670 on the boring half and 0.298 on the hard half. Nothing was trained; that 0.37 gap is a property of the task split alone, measured against a noise floor of 0.039 on the hard slice. That is the routing signal, and it is one an engineer can act on at runtime for free: the tool result is already in hand before the reply is written, so routing on “did the tool fault” costs no model call and no classifier. The base model’s failure has one name. Of its hard-slice failures, the dominant mode is ignore_fault — it reads a permission_denied or a malformed payload and cheerfully tells the customer the action succeeded. That is the specific thing a 1.5B cannot be trusted with out of the box.

​
The result I did not expect

Training on the boring half transferred to the hard half it never saw: 0.298 → 0.951, with ignore_fault falling to 15 rows out of 296. Twenty-four distinct examples of “say what the tool actually returned” were enough to fix fault-handling the training set never demonstrated. So the split I trained against stopped being the split that mattered — after SFT, both halves are safe, and the routing question dissolves. If that replicates, the advice is not “route the hard half to the big model” but “spend twenty examples and stop routing”.

​
Honest limits

  • The boring delta clears its noise band by 0.002 (+0.239 against a 0.237 band). It is a pass, but a marginal one. The band is wide because that slice has only 23 asks — see below.
  • The holdout is smaller than the library’s own power calculation wants. holdout_size(0.10, base=0.70, k=8) says 37 tasks per compared slice; the boring slice has 23, so it is powered for roughly +0.15. The cause is a hard ceiling on the keyless path (#698): the offline writer stops at ~120 distinct asks no matter what budget=, situations= or unique_situations= are set to.
  • One seed, one training run. The noise floor is across eval passes, not across training seeds, so it bounds evaluation noise only.
  • The grader was tuned against the incumbent’s phrasings. It may under-detect novel flaws from the small model, which biases in favour of the small model. See below for what it was checked against.
  • Training reward and eval metric are the same program, deliberately, so any gap is the split rather than reward design. It also means this run says nothing about reward hacking.

​
Is the grader worth trusting?

Everything here rests on a program that reads a trajectory and decides whether the reply followed policy. Because seeded_agent records what it planted on each row, that program can be checked against ground truth before it is trusted on a model whose rows carry no labels:
Zero false alarms over 1000 rollouts is the property that matters: when it says “flaw”, there is one. It under-detects rather than over-detects, so every pass rate here is an upper bound, equally for both arms. Getting there took one real correction, worth repeating because it is a trap. The obvious rule for “did the reply leak internal context” — token overlap between the reply and the row’s hidden note — flags the correct answer. The hidden note states the expected behaviour in the same words a good reply uses (“already done: say so and do not repeat the action” vs “That was already done for ORD-5307, so I did not repeat it”). That rule produced 56 false alarms on clean rows and dropped agreement to 0.888. Only verbatim scaffolding (hidden state:, expected outcome here is) is a leak. select()’s own built-in leak gate did not make this mistake — it dropped 0 of the same rows.

​
Cost line

Measured on this run, not estimated from a spec sheet. Generation throughput is 496 replies per 25.5 s on one L40S at $1.95/hour: 0.028per1000repliesisthenumbertoholdagainstyourownagent′sAPIbillforthesame1000replies;substituteyourprice,sincethebreak−evendependsentirelyonit.Thetrainingcostisnottheinterestingterm—at0.028 per 1000 replies is the number to hold against your own agent's API bill for the same 1000 replies; substitute your price, since the break-even depends entirely on it. The training cost is not the interesting term — at 0.009 you can retrain on every deploy and never notice. Add the cold start if you serve scale-to-zero: model load was ~25 s of the first pass’s 51 s.

​
Run it

No model API key is needed anywhere. The incumbent is wai.seeded_agent, the reward is a program, and the Modal container only generates text — grading happens locally, with the identical code that graded the incumbent, so the two arms cannot be scored by different rules. Run ids: Modal app ap-UjbpoQmEijbyasZmiS1XXH (L40S, 159 s), smoke ap-KiuXxz60VS9Z6BmvqkRuQc (CPU). Both stopped.

​
What did not work

  • Splitting on tier. Every row from simulate carries tier ∈ {ordinary, ambiguous, boundary, adversarial}, which looks exactly like the difficulty label you want. It is not a routing signal here: the incumbent’s pass rate is 0.72 on ordinary and 0.66 on the rest, intervals overlapping. Splitting on whether the tool faulted separated the base model by 0.37; splitting on tier separated it by nothing worth reporting. tier is also undocumented — it appears nowhere in docs/ or the README, so it is a field you find by printing a row.
  • select(mode="sft") left too little to train on. It keeps the best completion per prompt (rejection sampling, correctly), which took 129 clean rows down to 24 — one per distinct ask. With the ~120-ask ceiling that is not a training set, so this recipe trains on all 129 decontaminated clean rows and reports both numbers. out/train_sft_selected.jsonl holds what select would have kept.
  • Recording to the platform. wai.platform exposes track, Experiment, Run, RunRecord and ~100 other names; track wants an agent, and I have three arms of offline completions. No recipe in the tree posts scores from code, so there was nothing to copy, and the numbers live in results.json instead of a dashboard.
  • Raising the holdout past the ceiling. Covered above and in #698.

​
Files

Last modified on September 22, 2026