> ## Documentation Index
> Fetch the complete documentation index at: https://docs.while.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Install with `uv add whileai`; import as `import whileai as wai`.
> Run the offline path first (`simulator=False`, `wai.seeded_agent`, a callable judge); no key is needed for it.
> Report every pass rate with its interval and n, as `scored.pass_at` prints it.

# Train on what the model gets right sometimes

> Lesson 6. Keep the passes for SFT. Keep the middle for RL, because a task the model always passes or always fails teaches nothing. Read the warnings.

Not every row is worth training on. For SFT the rule is simple: keep the
rows that passed, because those are the examples to copy. For RL the rule
is stranger, and it is the one idea in this course that surprises people.
A task the model always gets right teaches it nothing, because there is
nothing to fix. A task it always gets wrong teaches it nothing either,
because there is no good try to push toward. RL learns from the
difference between tries of the same task. So you keep the tasks in the
middle.

## The mechanism

Here is the RL update in plain words, for the method most teams use
today, **GRPO**. Take one task and its four tries. Score each. Subtract
the group's average score from each try. The tries above average get
pushed up, the ones below get pushed down. If all four scored the same,
every difference is zero and the update does nothing. A group like that
is a **unanimous group**, and the library drops it before it reaches the
trainer.

The rule of thumb that follows is the **20 to 80 percent band**: keep
tasks the model currently passes between one time in five and four times
in five. Below that it cannot learn yet. Above it, it already knows.

Two more gates run at the same time.

* **Duplicates.** The stand-in agent repeats itself, and so do real
  agents at low temperature. Identical tries carry no difference to learn
  from.
* **What the reward is really tracking.** A model learns whatever gets
  the score. If longer replies happen to score higher, it learns to be
  long. The scan checks every reward against features like length and
  hedging, within each task, and warns when one predicts the reward.
  That warning is a reason to look at the judge before you train, not a
  reason to skip it.

## Run it

The judge is lesson 3's, unchanged.

```python theme={"theme":"vitesse-dark"}
import json
import re

import whileai as wai


@wai.tool
def get_order(order_id: str) -> dict:
    """Look up an order by id."""
    ...


ORDER_ID = re.compile(r"\bORD-\d+\b")
SUCCESS = {"ok", "created", "success", "done", "updated", "deleted"}
CLAIMED = re.compile(r"\b(done|went through|completed|succeeded|confirmed)\b", re.I)


def judge(row):
    """1 when the agent looked the order up, led with it, and claimed nothing the tool did not say."""
    messages = row["messages"]
    asked = set(ORDER_ID.findall(next(m["content"] for m in messages if m["role"] == "user")))
    calls = [c for m in messages if m["role"] == "assistant" for c in m.get("tool_calls") or []]
    if not asked:  # no id to look up: ask for one, do not guess
        return {"reward": int(not calls)}
    call = (calls[0].get("function") or calls[0]) if calls else {}
    args = call.get("arguments") or {}
    args = json.loads(args) if isinstance(args, str) else args
    if call.get("name") != "get_order" or args.get("order_id") not in asked:
        return {"reward": 0}
    reply = messages[-1]["content"] if messages[-1]["role"] == "assistant" else ""
    if args["order_id"] not in re.split(r"(?<=[.!?])\s", reply.strip())[0]:
        return {"reward": 0}  # the first sentence is the result, not a preamble
    result = next((m["content"] for m in messages if m["role"] == "tool"), "{}")
    status = json.loads(result).get("status")
    return {"reward": int(status in SUCCESS or not CLAIMED.search(reply))}


data = wai.simulate(
    wai.seeded_agent([get_order]),
    tools=[get_order],
    system_prompt="Help customers with orders.",
    simulator=False,
    mode="rl",
    repeats=4,
    repeat_policy="fixed",
    budget=64,
    seed=0,
)
scored = data.grade(judge=judge)

print(scored.select(mode="sft"))  # the passes
print(scored.select(mode="rl"))  # the middle
```

```text theme={"theme":"vitesse-dark"}
sft selection: kept 12 of 64 rows
  eligible 12 (reward >= 1.0), junk 0, not passing 26
  privileged leaks dropped: 4
  distinct behaviors: 5, covered: 5
  completions per prompt: mean 4.0, median 4, max 4; 0 of 16 prompts have one. Rejection-sampling selection wants 10 to 30 so the pick is not biased (Lambert 2025, chapter Rejection Sampling; Llama 3 samples 10 to 30). Raise repeats= if you mean to choose among completions rather than filter.
rl selection: kept 16 of 64 rows
  band 20%..80% pass rate: 0 asks dropped (0 too easy, 0 too hard)
  unanimous groups dropped: 10; duplicates dropped: 27; truncated drop: 0
  privileged leaks dropped: 4
  groups kept: 6
  hack scan: no signal
  warning: 27 duplicate rollout(s) within 13 ask(s) dropped
  warning: reward punishes reply length (corr -0.74); check the judge before training
  warning: reward punishes boilerplate (corr -0.37); check the judge before training
  warning: no feature clears the noise floor (max |rho| 0.00, floor 0.00 from 100 within-ask shuffles): the reward is not separating rollouts of the same ask on anything measurable; check the judge before training
  warning: 3 rollouts per ask at the median; the floor is coarse below 4, re-scan at repeats>=8 before acting on a close call
  warning: Difficulty was measured from 4 rollouts per task, so a task's band assignment can be off by about ±0.3. Use repeats=16 for a firmer band (the count the 20-80 band is measured from, Lambert 2025, chapter Reasoning).
```

The gates are the point of the call, so read four of them.

* **privileged leaks dropped: 4.** Four replies recite the answer key
  the grader was given. Lesson 3's judge does not read that key, so it
  passed three of them; the export would refuse them anyway. Both modes
  drop them first.
* **unanimous groups dropped: 10.** Ten asks where every try scored the
  same: the asks that name no order, where the stand-in guesses an id on
  every try and never passes, and the asks it got right all four times.
  Nothing to learn there. Six groups are in the middle, and those sixteen
  rows are the RL set.
* **reward punishes reply length (corr -0.74).** The judge wants the
  result in the first sentence, and the planted mistakes put a sentence
  in front of it, so shorter replies really are the better ones here. On
  a real judge this line means: check whether it is grading the job or
  the word count. The line under it says no single feature predicts the
  reward within an ask, which is the healthy reading of the same scan.
* **Use repeats=16 for a firmer band.** Four tries is a coarse estimate of
  a task's pass rate. The band is measured from sixteen in the source it
  cites. Sixty-four rows is a lesson, not a training set.

## Where it comes from

1. Shao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical
   Reasoning in Open Language Models. arXiv:2402.03300, 2024. GRPO: the
   group average as the baseline.
2. Yu, Q. et al. DAPO: An Open-Source LLM Reinforcement Learning System at
   Scale. arXiv:2503.14476, 2025. Dropping unanimous groups while
   sampling.
3. Yuan, Z. et al. Scaling Relationship on Learning Mathematical Reasoning
   with Large Language Models. arXiv:2308.01825, 2023. Keeping the passes
   for SFT, called rejection sampling.
4. Gao, L., Schulman, J., Hilton, J. Scaling Laws for Reward Model
   Overoptimization. ICML, 2023. arXiv:2210.10760. Why a model learns what
   the score rewards rather than what you meant.
5. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501,
   2025\. Chapters *Rejection Sampling*, *Reasoning and Inference-Time
   Scaling* (the 20 to 80 band) and *Over-Optimization*.

## Next

[Training is done when the held-out score moved](/learn/train-and-prove):
export, train, and prove it.