Train on what the model gets right sometimes

Not every row is worth training on. For SFT the rule is simple: keep the rows that passed, because those are the examples to copy. For RL the rule is stranger, and it is the one idea in this course that surprises people. A task the model always gets right teaches it nothing, because there is nothing to fix. A task it always gets wrong teaches it nothing either, because there is no good try to push toward. RL learns from the difference between tries of the same task. So you keep the tasks in the middle.

The mechanism

Here is the RL update in plain words, for the method most teams use today, GRPO. Take one task and its four tries. Score each. Subtract the group’s average score from each try. The tries above average get pushed up, the ones below get pushed down. If all four scored the same, every difference is zero and the update does nothing. A group like that is a unanimous group, and the library drops it before it reaches the trainer. The rule of thumb that follows is the 20 to 80 percent band: keep tasks the model currently passes between one time in five and four times in five. Below that it cannot learn yet. Above it, it already knows. Two more gates run at the same time.

Duplicates. The stand-in agent repeats itself, and so do real agents at low temperature. Identical tries carry no difference to learn from.
What the reward is really tracking. A model learns whatever gets the score. If longer replies happen to score higher, it learns to be long. The scan checks every reward against features like length and hedging, within each task, and warns when one predicts the reward. That warning is a reason to look at the judge before you train, not a reason to skip it.

Run it

The judge is lesson 3’s, unchanged.

import json
import re

import whileai as wai


@wai.tool
def get_order(order_id: str) -> dict:
    """Look up an order by id."""
    ...


ORDER_ID = re.compile(r"\bORD-\d+\b")
SUCCESS = {"ok", "created", "success", "done", "updated", "deleted"}
CLAIMED = re.compile(r"\b(done|went through|completed|succeeded|confirmed)\b", re.I)


def judge(row):
    """1 when the agent looked the order up, led with it, and claimed nothing the tool did not say."""
    messages = row["messages"]
    asked = set(ORDER_ID.findall(next(m["content"] for m in messages if m["role"] == "user")))
    calls = [c for m in messages if m["role"] == "assistant" for c in m.get("tool_calls") or []]
    if not asked:  # no id to look up: ask for one, do not guess
        return {"reward": int(not calls)}
    call = (calls[0].get("function") or calls[0]) if calls else {}
    args = call.get("arguments") or {}
    args = json.loads(args) if isinstance(args, str) else args
    if call.get("name") != "get_order" or args.get("order_id") not in asked:
        return {"reward": 0}
    reply = messages[-1]["content"] if messages[-1]["role"] == "assistant" else ""
    if args["order_id"] not in re.split(r"(?<=[.!?])\s", reply.strip())[0]:
        return {"reward": 0}  # the first sentence is the result, not a preamble
    result = next((m["content"] for m in messages if m["role"] == "tool"), "{}")
    status = json.loads(result).get("status")
    return {"reward": int(status in SUCCESS or not CLAIMED.search(reply))}


data = wai.simulate(
    wai.seeded_agent([get_order]),
    tools=[get_order],
    system_prompt="Help customers with orders.",
    simulator=False,
    mode="rl",
    repeats=4,
    repeat_policy="fixed",
    budget=64,
    seed=0,
)
scored = data.grade(judge=judge)

print(scored.select(mode="sft"))  # the passes
print(scored.select(mode="rl"))  # the middle

sft selection: kept 12 of 64 rows
  eligible 12 (reward >= 1.0), junk 0, not passing 26
  privileged leaks dropped: 4
  distinct behaviors: 5, covered: 5
  completions per prompt: mean 4.0, median 4, max 4; 0 of 16 prompts have one. Rejection-sampling selection wants 10 to 30 so the pick is not biased (Lambert 2025, chapter Rejection Sampling; Llama 3 samples 10 to 30). Raise repeats= if you mean to choose among completions rather than filter.
rl selection: kept 16 of 64 rows
  band 20%..80% pass rate: 0 asks dropped (0 too easy, 0 too hard)
  unanimous groups dropped: 10; duplicates dropped: 27; truncated drop: 0
  privileged leaks dropped: 4
  groups kept: 6
  hack scan: no signal
  warning: 27 duplicate rollout(s) within 13 ask(s) dropped
  warning: reward punishes reply length (corr -0.74); check the judge before training
  warning: reward punishes boilerplate (corr -0.37); check the judge before training
  warning: no feature clears the noise floor (max |rho| 0.00, floor 0.00 from 100 within-ask shuffles): the reward is not separating rollouts of the same ask on anything measurable; check the judge before training
  warning: 3 rollouts per ask at the median; the floor is coarse below 4, re-scan at repeats>=8 before acting on a close call
  warning: Difficulty was measured from 4 rollouts per task, so a task's band assignment can be off by about ±0.3. Use repeats=16 for a firmer band (the count the 20-80 band is measured from, Lambert 2025, chapter Reasoning).

The gates are the point of the call, so read four of them.

privileged leaks dropped: 4. Four replies recite the answer key the grader was given. Lesson 3’s judge does not read that key, so it passed three of them; the export would refuse them anyway. Both modes drop them first.
unanimous groups dropped: 10. Ten asks where every try scored the same: the asks that name no order, where the stand-in guesses an id on every try and never passes, and the asks it got right all four times. Nothing to learn there. Six groups are in the middle, and those sixteen rows are the RL set.
reward punishes reply length (corr -0.74). The judge wants the result in the first sentence, and the planted mistakes put a sentence in front of it, so shorter replies really are the better ones here. On a real judge this line means: check whether it is grading the job or the word count. The line under it says no single feature predicts the reward within an ask, which is the healthy reading of the same scan.
Use repeats=16 for a firmer band. Four tries is a coarse estimate of a task’s pass rate. The band is measured from sixteen in the source it cites. Sixty-four rows is a lesson, not a training set.

Where it comes from

Shao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024. GRPO: the group average as the baseline.
Yu, Q. et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476, 2025. Dropping unanimous groups while sampling.
Yuan, Z. et al. Scaling Relationship on Learning Mathematical Reasoning with Large Language Models. arXiv:2308.01825, 2023. Keeping the passes for SFT, called rejection sampling.
Gao, L., Schulman, J., Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML, 2023. arXiv:2210.10760. Why a model learns what the score rewards rather than what you meant.
Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapters Rejection Sampling, Reasoning and Inference-Time Scaling (the 20 to 80 band) and Over-Optimization.

Training is done when the held-out score moved: export, train, and prove it.

From zero

Train on what the model gets right sometimes

The mechanism

Run it

Where it comes from

Next

​The mechanism

​Run it

​Where it comes from

​Next

The mechanism

Run it

Where it comes from

Next