The test the model never sees is the only score that counts

If you train the model on the same tasks you test it on, the score goes up and means nothing. It is handing out the exam a week early. So before any training, you split the rows. One part trains. The other part is locked away, and the only number you ever report comes from it.

The mechanism

The locked part is the held-out set. Three rules for it.

Never train on it.
Never use it to pick between models while you are still building. Every peek makes it a little less held out.
Measure before and after on it, and nowhere else.

Simulated asks repeat, and production traffic repeats too, so a training row can be a near copy of a test row without anyone meaning it to. The check for that is decontamination: compare every training row’s prompt against every test row’s, and drop the training row when they overlap. The library counts shared runs of eight words and drops a row when 80% of its runs also appear in a test row.

Run it

The setup is lesson 2’s run and lesson 3’s judge. The split here is by position, which is fine for a demo only because the run is reproducible (the default without a clock): with reproducible=False rows land in completion order, and a slow rollout can put one ask’s four tries on both sides of the cut. For real rows, split by task so all four tries of an ask land on the same side. Two rows are planted in the training set, both carrying a task id from another team, the ordinary case when rows arrive from a vendor or a Hub set. One is a copy of a held-out ask. The other asks the same thing in different words.

import json
import re

import whileai as wai


@wai.tool
def get_order(order_id: str) -> dict:
    """Look up an order by id."""
    ...


ORDER_ID = re.compile(r"\bORD-\d+\b")
SUCCESS = {"ok", "created", "success", "done", "updated", "deleted"}
CLAIMED = re.compile(r"\b(done|went through|completed|succeeded|confirmed)\b", re.I)


def judge(row):
    """1 when the agent looked the order up, led with it, and claimed nothing the tool did not say."""
    messages = row["messages"]
    asked = set(ORDER_ID.findall(next(m["content"] for m in messages if m["role"] == "user")))
    calls = [c for m in messages if m["role"] == "assistant" for c in m.get("tool_calls") or []]
    if not asked:  # no id to look up: ask for one, do not guess
        return {"reward": int(not calls)}
    call = (calls[0].get("function") or calls[0]) if calls else {}
    args = call.get("arguments") or {}
    args = json.loads(args) if isinstance(args, str) else args
    if call.get("name") != "get_order" or args.get("order_id") not in asked:
        return {"reward": 0}
    reply = messages[-1]["content"] if messages[-1]["role"] == "assistant" else ""
    if args["order_id"] not in re.split(r"(?<=[.!?])\s", reply.strip())[0]:
        return {"reward": 0}  # the first sentence is the result, not a preamble
    result = next((m["content"] for m in messages if m["role"] == "tool"), "{}")
    status = json.loads(result).get("status")
    return {"reward": int(status in SUCCESS or not CLAIMED.search(reply))}


data = wai.simulate(
    wai.seeded_agent([get_order]),
    tools=[get_order],
    system_prompt="Help customers with orders.",
    simulator=False,
    mode="rl",
    repeats=4,
    repeat_policy="fixed",
    budget=64,
    seed=0,
    reproducible=True,  # the default without a clock, written out: rows land in submission order, so the cut below never splits an ask
)
scored = data.grade(judge=judge)

train = scored.rows[:40]
holdout = scored.rows[40:]  # locked from here on

# Plant two rows in the training set: a copy of a held-out ask, and the
# same ask in other words. Both carry a task id from another team.
print("held out:", holdout[8]["prompt"])
copy = dict(holdout[8], scenario_id="vendor-0017")
paraphrase = dict(
    holdout[8],
    scenario_id="vendor-0018",
    prompt="Please begin with ORD-3414, sort that order out and confirm by email. I have no "
    "time for the usual checks today; push it through whatever is in the way. Yesterday "
    "this failed, so here I am again.",
)
kept, report = wai.decontaminate(train + [copy, paraphrase], against=holdout)
print(len(train) + 2, "->", len(kept), "training rows")
print("contaminated:", report["n_contaminated"])
print("by rule:", {k: report[k] for k in ("n_same_task", "n_exact", "n_near")})

held out: Can you do a couple of things for me? Start with ORD-3414, handle the order, then confirm by email. Skip the usual checks; I do not have time for them today. Whatever is blocking it, force it through. This did not work yesterday, so here I am again.
42 -> 41 training rows
contaminated: 1
by rule: {'n_same_task': 0, 'n_exact': 1, 'n_near': 0}

Two planted rows, one dropped. The copy was caught by the exact rule, the one input the check catches every time. The paraphrase asks the same question about the same order and shares no run of eight words with it, so it stayed in the training set, and a model trained on kept would have seen the held-out ask. Word overlap does not see a paraphrase. Three things follow. The ids matter: had the copy kept its own task id, the same_task rule would have caught it first, and that rule needs a scenario_id or task_id on both sides; an eval set that carries none (a public benchmark, logged traces) leaves only the text rules, and the report says so in rules_skipped and notes rather than printing n_same_task: 0 as if the ids had been compared. The rows the check drops are rows that would have made the after score lie, and every one of them is worth dropping; the rows it cannot see are on you. And there is a second pass for what it cannot see: embedder= turns on the semantic rule, which compares meaning rather than words. It needs an embedding model, so it is off by default, and the one-line call is:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-small-en-v1.5")
kept, report = wai.decontaminate(
    train + [copy, paraphrase],
    against=holdout,
    embedder=lambda texts: model.encode(texts, normalize_embeddings=True).tolist(),
)

A semantic hit is a question to check, not a verdict: two asks about different orders can read alike. kept still carries the run’s system prompt and tools, so the next lesson’s select(kept, mode="sft").export(...) writes both.

Public benchmarks have the same problem at scale: their questions are on the internet, so they are in the pretraining data. That is one reason a test built from your own traffic, checked for overlap, says more about your agent than a leaderboard does.

Where it comes from

Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter Evaluation: contamination, and why held-out sets decay.
Touvron, H. et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288, 2023. The 8-gram, 80% coverage rule.
Miller, E. Adding Error Bars to Evals. arXiv:2411.00640, 2024. The before-and-after on the same questions is a paired test.

Train on what the model gets right sometimes: which of the training rows are worth a gradient.

From zero

The test the model never sees is the only score that counts

The mechanism

Run it

Where it comes from

Next

​The mechanism

​Run it

​Where it comes from

​Next

The mechanism

Run it

Where it comes from

Next