> ## Documentation Index
> Fetch the complete documentation index at: https://docs.while.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Install with `uv add whileai`; import as `import whileai as wai`.
> Run the offline path first (`simulator=False`, `wai.seeded_agent`, a callable judge); no key is needed for it.
> Report every pass rate with its interval and n, as `scored.pass_at` prints it.

# Which half of my agent's traffic can a 1.5B model take over?

> I have one production agent and a cost line to defend.

<Note>The scripts are in the repository, not in the installed package. Clone it,
then `cd recipes/community/which-half-can-a-small-model-take` before running the commands below. [Browse this recipe on GitHub](https://github.com/whilehq/whileai-sdk/tree/main/recipes/community/which-half-can-a-small-model-take).</Note>

I have one production agent and a cost line to defend. Its job splits in two:
pick the tool (interesting) and, given what the tool returned, write the reply
(boring). This asks whether a small open model can take over just the boring
half, and measures it on a frozen held-out set before anything ships.

The answer turned out to be about **where the split is**, not whether one
exists — and the number that carries the finding is the *untrained* model's,
not the trained one's.

## Result

`Qwen/Qwen2.5-1.5B-Instruct`, LoRA SFT on 129 rows (24 distinct asks) from the
incumbent's own clean traces on the boring half only. One L40S, 159 seconds,
**\$0.09**. Paired on the same held-out asks, 8 rollouts each; noise floor from
three passes of identical untrained weights on the same container.

| held-out slice                     | asks | incumbent | base 1.5B | trained 1.5B | trained − base            | noise band |
| ---------------------------------- | ---- | --------- | --------- | ------------ | ------------------------- | ---------- |
| **boring** (tool returned cleanly) | 23   | 0.688     | 0.670     | **0.964**    | +0.239 `[+0.174, +0.304]` | 0.237      |
| **hard** (tool faulted)            | 37   | 0.701     | 0.298     | **0.951**    | +0.659 `[+0.578, +0.736]` | 0.039      |

Against the incumbent, paired on the same asks:

| held-out slice | incumbent → trained | 95% interval       | verdict |
| -------------- | ------------------- | ------------------ | ------- |
| boring         | 0.688 → 0.964       | `[+0.190, +0.370]` | moved   |
| hard           | 0.701 → 0.951       | `[+0.169, +0.328]` | moved   |

### The caveat that comes first

**The "beats the incumbent" rows are close to tautological, and should not be
quoted on their own.** The incumbent here injects stylistic flaws — apology,
hedging, sycophancy, boilerplate, leaking internal notes, claiming success
after a fault — at a fixed 35% rate, uniformly, *independent of how hard the
ask is*. Training on the subset a program marked clean therefore produces a
model that does not do the thing that was filtered out. Any imitator would
win. A real production agent's failures correlate with difficulty; these do
not, so this run says nothing about whether a small model can match a frontier
agent on genuinely hard asks.

### The number that is not tautological

**The base, untrained 1.5B scores 0.670 on the boring half and 0.298 on the
hard half.** Nothing was trained; that 0.37 gap is a property of the task
split alone, measured against a noise floor of 0.039 on the hard slice. That
is the routing signal, and it is one an engineer can act on *at runtime for
free*: the tool result is already in hand before the reply is written, so
routing on "did the tool fault" costs no model call and no classifier.

The base model's failure has one name. Of its hard-slice failures, the
dominant mode is `ignore_fault` — it reads a `permission_denied` or a
malformed payload and cheerfully tells the customer the action succeeded.
That is the specific thing a 1.5B cannot be trusted with out of the box.

### The result I did not expect

Training on the boring half **transferred to the hard half it never saw**:
0.298 → 0.951, with `ignore_fault` falling to 15 rows out of 296. Twenty-four
distinct examples of "say what the tool actually returned" were enough to fix
fault-handling the training set never demonstrated. So the split I trained
against stopped being the split that mattered — after SFT, both halves are
safe, and the routing question dissolves. If that replicates, the advice is
not "route the hard half to the big model" but "spend twenty examples and stop
routing".

### Honest limits

* **The boring delta clears its noise band by 0.002** (+0.239 against a 0.237
  band). It is a pass, but a marginal one. The band is wide because that slice
  has only 23 asks — see below.
* **The holdout is smaller than the library's own power calculation wants.**
  `holdout_size(0.10, base=0.70, k=8)` says 37 tasks per compared slice; the
  boring slice has 23, so it is powered for roughly +0.15. The cause is a hard
  ceiling on the keyless path ([#698](https://github.com/whilehq/whileai-sdk/issues/698)):
  the offline writer stops at \~120 distinct asks no matter what `budget=`,
  `situations=` or `unique_situations=` are set to.
* **One seed, one training run.** The noise floor is across eval passes, not
  across training seeds, so it bounds evaluation noise only.
* **The grader was tuned against the incumbent's phrasings.** It may
  under-detect novel flaws from the small model, which biases *in favour of*
  the small model. See below for what it was checked against.
* **Training reward and eval metric are the same program**, deliberately, so
  any gap is the split rather than reward design. It also means this run says
  nothing about reward hacking.

## Is the grader worth trusting?

Everything here rests on a program that reads a trajectory and decides whether
the reply followed policy. Because `seeded_agent` records what it planted on
each row, that program can be checked against ground truth before it is
trusted on a model whose rows carry no labels:

```text theme={"theme":"vitesse-dark"}
grader vs ground truth: agreement 0.996 (caught 322, missed 4, false alarms 0, clean 674)
```

Zero false alarms over 1000 rollouts is the property that matters: when it
says "flaw", there is one. It under-detects rather than over-detects, so every
pass rate here is an upper bound, equally for both arms.

Getting there took one real correction, worth repeating because it is a trap.
The obvious rule for "did the reply leak internal context" — token overlap
between the reply and the row's hidden note — **flags the correct answer**. The
hidden note states the expected behaviour in the same words a good reply uses
("already done: say so and do not repeat the action" vs "That was already done
for ORD-5307, so I did not repeat it"). That rule produced 56 false alarms on
clean rows and dropped agreement to 0.888. Only verbatim scaffolding
(`hidden state:`, `expected outcome here is`) is a leak. `select()`'s own
built-in leak gate did not make this mistake — it dropped 0 of the same rows.

## Cost line

Measured on this run, not estimated from a spec sheet. Generation throughput
is 496 replies per 25.5 s on one L40S at \$1.95/hour:

| item                                               | measured | \$        |
| -------------------------------------------------- | -------- | --------- |
| LoRA SFT, 129 rows, 3 epochs, 51 steps             | 17 s     | 0.009     |
| one held-out eval pass (496 replies)               | 25.5 s   | 0.014     |
| whole run (3 base passes + train + 1 trained pass) | 159 s    | **0.09**  |
| **serving: 1000 replies**                          | 51 s     | **0.028** |

$0.028 per 1000 replies is the number to hold against your own agent's API
bill for the same 1000 replies; substitute your price, since the break-even
depends entirely on it. The training cost is not the interesting term — at
$0.009 you can retrain on every deploy and never notice.

Add the cold start if you serve scale-to-zero: model load was \~25 s of the
first pass's 51 s.

## Run it

```bash theme={"theme":"vitesse-dark"}
pip install whileai 'modal[api-proxy-support]'

python run.py                      # traffic -> graded -> split -> out/train_sft.jsonl + out/holdout.json
modal run train_modal.py           # 3 base passes, LoRA SFT, 1 trained pass  (~$0.09)
python analyze.py                  # grade the generations, print the verdicts, write out/results.json

sh smoke.sh                        # offline, no GPU, no key: the split only
```

No model API key is needed anywhere. The incumbent is `wai.seeded_agent`, the
reward is a program, and the Modal container only generates text — grading
happens locally, with the identical code that graded the incumbent, so the two
arms cannot be scored by different rules.

Run ids: Modal app `ap-UjbpoQmEijbyasZmiS1XXH` (L40S, 159 s), smoke
`ap-KiuXxz60VS9Z6BmvqkRuQc` (CPU). Both stopped.

## What did not work

* **Splitting on `tier`.** Every row from `simulate` carries
  `tier` ∈ \{`ordinary`, `ambiguous`, `boundary`, `adversarial`}, which looks
  exactly like the difficulty label you want. It is not a routing signal here:
  the incumbent's pass rate is 0.72 on `ordinary` and 0.66 on the rest,
  intervals overlapping. Splitting on whether the tool faulted separated the
  *base model* by 0.37; splitting on `tier` separated it by nothing worth
  reporting. `tier` is also undocumented — it appears nowhere in `docs/` or the
  README, so it is a field you find by printing a row.
* **`select(mode="sft")` left too little to train on.** It keeps the best
  completion per prompt (rejection sampling, correctly), which took 129 clean
  rows down to 24 — one per distinct ask. With the \~120-ask ceiling that is
  not a training set, so this recipe trains on all 129 decontaminated clean
  rows and reports both numbers. `out/train_sft_selected.jsonl` holds what
  `select` would have kept.
* **Recording to the platform.** `wai.platform` exposes `track`, `Experiment`,
  `Run`, `RunRecord` and \~100 other names; `track` wants an agent, and I have
  three arms of offline completions. No recipe in the tree posts scores from
  code, so there was nothing to copy, and the numbers live in `results.json`
  instead of a dashboard.
* **Raising the holdout past the ceiling.** Covered above and in
  [#698](https://github.com/whilehq/whileai-sdk/issues/698).

## Files

| file             | what                                                                          |
| ---------------- | ----------------------------------------------------------------------------- |
| `run.py`         | traffic, grading, grader validation, the split, decontamination, the SFT file |
| `grader.py`      | the program that reads a trajectory and names the policy flaws                |
| `train_modal.py` | 3 base passes, LoRA SFT, 1 trained pass, on your Modal                        |
| `analyze.py`     | grades the generations locally, prints the verdicts, writes `results.json`    |
| `results.json`   | every number above, with intervals                                            |