Training is done when the held-out score moved

You have rows worth training on and a locked test set. Three steps are left. Write the rows to a file a trainer reads. Train, on your own GPU, on Modal, or on Prime Intellect, with your own keys. Then prove it: run the old model and the new one on the same held-out tasks, and check that the difference is bigger than the noise. If it is not, nothing happened, and saying so is the result.

The mechanism

export writes one JSON object per line, in the shape trainers read, and refuses a broken row. The training is one command on your Modal account: the SFT recipe trains a small open model with a LoRA adapter on that file, on one A10G, in about ten minutes, for under twenty cents. It also runs the base model over the held-out tasks three times before it trains anything, because a score bounces from run to run and you need to know by how much before you can call a change real. That spread is the noise floor. The proof is a paired comparison. Every held-out task is run by both models, the difference is taken task by task, and the interval on the average difference has to exclude zero and clear the noise floor. Paired, because the same hard task is hard for both models, and pairing cancels that out. The library also refuses to compare two runs that were not on the same tasks.

Run it

Lessons 2 to 6 used the stand-in agent, because the ideas run offline. This lesson trains a real model, so it is the one lesson that needs an account: modal token new on your laptop. The judge is lesson 3’s, unchanged, and that is the point: it reads messages, so the same program scores the stand-in’s rows here and the trained model’s rows in step 3. Step 1 writes more rows than the earlier lessons, because sixty-four rows train nothing: four wordings of every ask, split by task the way lesson 5 said to, the passes kept the way lesson 6 said to.

import json
import re

import whileai as wai


@wai.tool
def get_order(order_id: str) -> dict:
    """Look up an order by id."""
    ...


ORDER_ID = re.compile(r"\bORD-\d+\b")
SUCCESS = {"ok", "created", "success", "done", "updated", "deleted"}
CLAIMED = re.compile(r"\b(done|went through|completed|succeeded|confirmed)\b", re.I)


def judge(row):
    """1 when the agent looked the order up, led with it, and claimed nothing the tool did not say."""
    messages = row["messages"]
    asked = set(ORDER_ID.findall(next(m["content"] for m in messages if m["role"] == "user")))
    calls = [c for m in messages if m["role"] == "assistant" for c in m.get("tool_calls") or []]
    if not asked:  # no id to look up: ask for one, do not guess
        return {"reward": int(not calls)}
    call = (calls[0].get("function") or calls[0]) if calls else {}
    args = call.get("arguments") or {}
    args = json.loads(args) if isinstance(args, str) else args
    if call.get("name") != "get_order" or args.get("order_id") not in asked:
        return {"reward": 0}
    reply = messages[-1]["content"] if messages[-1]["role"] == "assistant" else ""
    if args["order_id"] not in re.split(r"(?<=[.!?])\s", reply.strip())[0]:
        return {"reward": 0}  # the first sentence is the result, not a preamble
    result = next((m["content"] for m in messages if m["role"] == "tool"), "{}")
    status = json.loads(result).get("status")
    return {"reward": int(status in SUCCESS or not CLAIMED.search(reply))}


data = wai.simulate(
    wai.seeded_agent([get_order]),
    tools=[get_order],
    system_prompt="Help customers with orders.",
    simulator=False,
    mode="rl",
    repeats=4,
    repeat_policy="fixed",
    phrasings=4,  # four wordings of every ask
    budget=640,
    concurrency=1,  # the draw depends on the batch size; this is the size the SFT recipe's numbers came from
    seed=0,
)
scored = data.grade(judge=judge)

# 1. Split by task (lesson 5), keep the passes (lesson 6), write both sides.
tasks = sorted({r["scenario_id"] for r in scored.rows})
held = set(tasks[-40:])  # locked from here on
train = [r for r in scored.rows if r["scenario_id"] not in held]
holdout = [r for r in scored.rows if r["scenario_id"] in held]
train, report = wai.decontaminate(train, against=holdout)
written = wai.select(train, mode="sft").export(
    "train.jsonl", system_prompt="Help customers with orders.", tools=[get_order.schema]
)
with open("holdout.jsonl", "w") as f:
    for r in holdout:
        f.write(json.dumps({"scenario_id": r["scenario_id"], "messages": r["messages"]}) + "\n")
print(len(scored.rows), "rows over", len(tasks), "tasks;", len(held), "tasks held out")
print(report["n_contaminated"], "training rows dropped as near copies of a held-out ask")
print(written["n"], "rows written to train.jsonl")

rows over 104 tasks; 40 tasks held out
training rows dropped as near copies of a held-out ask
rows written to train.jsonl

Forty-six rows: the stand-in’s passes on the training tasks, each one a system prompt, the ask, the lookup, what came back, and a reply that led with it. holdout.jsonl carries the forty locked tasks with the answer the fake world gave each lookup, so the trained model is asked the same question in the same world. Step 2 is the training. One command, on your account, from the directory the two files are in. It loads Qwen/Qwen2.5-1.5B-Instruct, runs the base model over the held-out asks three times, trains a LoRA adapter for forty steps, runs the adapter once on the same asks and the same seed, and writes every sampled row back to your laptop as holdout_rows.jsonl. This is what it printed on the run the rest of this page reads from.

modal run recipes/04-train/sft/train_modal.py --data train.jsonl

46 rows from train.jsonl, 68 held-out asks from holdout.jsonl; 40 steps is 7.0 passes over the rows
46 training rows rendered; the longest is 411 tokens (max_length 1024)
68 held-out asks over 40 tasks
base pass 1 (seed 1): 272 rows in 84.1s
base pass 2 (seed 2): 272 rows in 83.0s
base pass 3 (seed 3): 272 rows in 79.5s
trained 40 steps in 35.2s: loss 3.072 -> 0.798
trained pass (seed 1): 84.3s
A10G: 373.2s total, training 35.2s; adapter at whileai-sft-runs:/lesson7-sft/adapter
wrote 1088 rows to holdout_rows.jsonl

Step 3 is the proof, on your laptop, from that file. The judge scores a real model’s rows the same way it scored the stand-in’s. The three base passes give the noise floor, and compare takes it directly.

rows = [json.loads(line) for line in open("holdout_rows.jsonl")]
for r in rows:
    r.update(judge(r))  # the same judge, on a real model's rows

base = [[r for r in rows if r["model_version"] == "base" and r["run"] == n] for n in (1, 2, 3)]
trained = [r for r in rows if r["model_version"] == "trained"]
noise = wai.simulations.eval_variance(*base)

print("before:", wai.pass_at(base[0]))
print("after: ", wai.pass_at(trained))
print("base passes:", " / ".join(f"{m:.3f}" for m in noise["means"].values()), " run_std", f"{noise['run_std']:.3f}")
print(wai.compare(base[0], trained, run_std=noise["run_std"], run_std_runs=3))

before: pass@1 0.25 [0.16..0.35] | pass^4 (pass_pow_k) 0.07 [0.01..0.15] | pass@4 0.48 [0.34..0.64] | headroom 0.24 (40 groups, k=4; groups are uneven (4 to 16 repeats); k is the smallest)
after:  pass@1 0.73 [0.63..0.81] | pass^4 (pass_pow_k) 0.41 [0.28..0.55] | pass@4 0.95 [0.87..1.00] | headroom 0.22 (40 groups, k=4; groups are uneven (4 to 16 repeats); k is the smallest)
base passes: 0.246 / 0.248 / 0.244  run_std 0.002
PASS
eval noise: run_std 0.002, a delta under 0.011 is noise (t(df=2)=4.30 x run_std x sqrt(1/1 + 1/1); run_std given from 3 re-runs)
answered: 100.0% before, 100.0% after
  pass_at_1                    0.246 -> 0.728  +0.481 [+0.342..+0.616]  up  (40 paired)  noise<0.011

Read the last line. Forty tasks, paired. The base model did the job on 25% of tries; the trained one on 73%. The difference is 48 points, and the interval on it runs from 34 to 62. It does not touch zero, and it clears the one point that re-running the eval moves the number on its own (three base passes landed within half a point of each other), so the verdict is PASS. Had it read +0.05 [-0.03..+0.13], the verdict would be no difference, and the honest sentence is “training did not move it”. Two more things the line says. pass^4 went from 7% to 41%: the share of asks the model gets right every time, which is what a customer sees, went up six times over and is still under half. And forty tasks is fewer than the 89 that lesson 4’s holdout_size asked for; the interval is as wide as that implies, and the gain cleared it because it is large, not because the set is.

One PASS is one PASS. Three passes of the base model measured how much the eval moves; the training ran once. Run the training again under a different --run-name before you tell anyone. A result you can repeat is the only kind this library is built to produce.

Then it starts again

The model you served is now the agent. Its traffic is the next set of rows. Lesson 2 to lesson 7, again, on what it still gets wrong. That is the loop, and the reason the agent gets better while it works.

Where to go now

Quickstart: the same program, with your own agent and your own judge.
Your model and your key: name the model as a string, keep the key in the provider’s own variable.
Evals: the measurement half on its own, for a team that is not ready to train.
Train on your own GPU: the SFT recipe this lesson ran, and GRPO and DPO on the same base, with the paired delta at the end.
Papers, reproduced: one recent post-training paper per recipe, each with its verdict.

Where it comes from

Miller, E. Adding Error Bars to Evals. arXiv:2411.00640, 2024. The paired difference, and why it is the right test for before and after.
Hu, E. J. et al. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685, 2021. The adapter the recipe trains.
Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapters Instruction Finetuning and Evaluation: SFT, and the noise floor from re-running an eval.
Shao, Z. et al. DeepSeekMath. arXiv:2402.03300, 2024. The GRPO recipe next to this one.

A teacher can score every word: training with no reward at all, when that beats a reward, and when it cannot.

From zero

Training is done when the held-out score moved

The mechanism

Run it

Then it starts again

Where to go now

Where it comes from

Next

​The mechanism

​Run it

​Then it starts again

​Where to go now

​Where it comes from

​Next

The mechanism

Run it

Then it starts again

Where to go now

Where it comes from

Next