> ## Documentation Index > Fetch the complete documentation index at: https://docs.while.ai/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > Install with `uv add whileai`; import as `import whileai as wai`. > Run the offline path first (`simulator=False`, `wai.seeded_agent`, a callable judge); no key is needed for it. > Report every pass rate with its interval and n, as `scored.pass_at` prints it. # A reward is a score you can defend > Lesson 3. A number per row, from a program that checks the answer or a model that reads it. How to check the checker before you trust it. You have 64 rows. Some are good and some are not, and the model will learn from whichever you keep. So every row needs a score. Usually it is 0 or 1: did the agent do the job. The score has to be one you can defend, because the model will learn exactly what the score rewards, including the parts you did not mean. ## The mechanism The score is called the **reward**. There are two ways to get one. * **A program.** The answer matches the reference, the tests pass, the tool was called before the reply. When a program can check the job, use the program. It is cheap, repeatable, and cannot be flattered. The field calls this a **verifiable reward**, and the program a **verifier**. * **A model.** When no program can check the job (was the tone right, did it explain the policy), a model reads the reply against a written rubric and answers 0 or 1. That is an **LLM judge**. A judge is a measurement instrument, so you check it before you use it. Label 50 rows by hand, run the judge on the same rows, and measure how often they agree. The library reports plain agreement and **kappa**, which is agreement after subtracting what two coin flips would agree on. ## Run it The setup is lesson 2's run. The judge below is the one this course uses from here to lesson 7. It is a program over the row's `messages`, so it scores the stand-in and, in lesson 7, a trained model, with the same three rules: look the order up with the id the customer gave (and ask when there is none, rather than guess), lead the reply with that order, and claim nothing the tool did not return. ```python theme={"theme":"vitesse-dark"} import json import re import whileai as wai from whileai.simulations import attach_labels @wai.tool def get_order(order_id: str) -> dict: """Look up an order by id.""" ... data = wai.simulate( wai.seeded_agent([get_order]), tools=[get_order], system_prompt="Help customers with orders.", simulator=False, mode="rl", repeats=4, repeat_policy="fixed", budget=64, seed=0, ) ORDER_ID = re.compile(r"\bORD-\d+\b") SUCCESS = {"ok", "created", "success", "done", "updated", "deleted"} CLAIMED = re.compile(r"\b(done|went through|completed|succeeded|confirmed)\b", re.I) # 1. A judge is any function from a row to a reward. This one is a program. def judge(row): """1 when the agent looked the order up, led with it, and claimed nothing the tool did not say.""" messages = row["messages"] asked = set(ORDER_ID.findall(next(m["content"] for m in messages if m["role"] == "user"))) calls = [c for m in messages if m["role"] == "assistant" for c in m.get("tool_calls") or []] if not asked: # no id to look up: ask for one, do not guess return {"reward": int(not calls)} call = (calls[0].get("function") or calls[0]) if calls else {} args = call.get("arguments") or {} args = json.loads(args) if isinstance(args, str) else args if call.get("name") != "get_order" or args.get("order_id") not in asked: return {"reward": 0} reply = messages[-1]["content"] if messages[-1]["role"] == "assistant" else "" if args["order_id"] not in re.split(r"(?<=[.!?])\s", reply.strip())[0]: return {"reward": 0} # the first sentence is the result, not a preamble result = next((m["content"] for m in messages if m["role"] == "tool"), "{}") status = json.loads(result).get("status") return {"reward": int(status in SUCCESS or not CLAIMED.search(reply))} scored = data.grade(judge=judge) print(scored.pass_at) # 2. A looser definition of the same job: the reply names the order, anywhere. @wai.verifier def names_the_order(candidate, reference, row): ids = [step["arguments"].get("order_id", "") for step in row["steps"]] return float(any(i and i in candidate for i in ids)) print(data.grade(judge=names_the_order).pass_at) # 3. Check the judge against labels. Here the labels come from the stand-in's # answer key (its `seeded` field); in real life a person writes them, 50 to 200. labels = [ {"scenario_id": r["scenario_id"], "rollout_index": r["rollout_index"], "label": int(not r["seeded"])} for r in scored.rows[:40] ] labeled, _ = attach_labels(scored.rows, labels, kind="human") trust = wai.judge_trust(labeled, judge) print("trusted:", trust["ok"]) print("agreement:", trust["agreement"]["agreement"], "kappa:", trust["agreement"]["kappa"]) ``` ```text theme={"theme":"vitesse-dark"} pass@1 0.58 [0.38..0.77] | pass^4 (pass_pow_k) 0.31 [0.12..0.56] | pass@4 0.75 [0.50..0.94] | headroom 0.17 (16 groups, k=4) pass@1 0.75 [0.50..0.94] | pass^4 (pass_pow_k) 0.75 [0.50..0.94] | pass@4 0.75 [0.50..0.94] | headroom 0.00 (16 groups, k=4) trusted: False agreement: 0.825 kappa: 0.6111 ``` Two different rewards, two different numbers, same rows. That is not a bug. A reward is a definition of the job, and the first line of any result is which definition it used. The third block is the check. The judge agrees with the labels 82% of the time, kappa 0.61, and the library still says `trusted: False`: with only 40 labels the lower bound on that agreement is 68%, under its 80% floor, and it flags that the judge passes long replies more often than short ones with the same label. Read `trust["disagreements"]` before you argue with it. Five are asks that name no order, where the stand-in guessed an id and its answer key does not count that as a mistake; two are replies that quote the hidden answer key, which this judge does not read (lesson 6 drops those rows before training). The number told you where to look, and you looked. That is what checking the checker means. The numbers after each pass rate are the interval. Lesson 4 is about them. For now: `0.58 [0.38..0.77]` means the true rate very likely sits between 38% and 77%. ## Where it comes from 1. Lambert, N. et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124, 2024. Reinforcement learning with verifiable rewards: the reward is a program. 2. Zheng, L. et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS, 2023. arXiv:2306.05685. How well a model judge agrees with people, and where it is biased. 3. Cohen, J. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20(1), 1960. Kappa. 4. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025\. Chapter *Reward Modeling*. ## Next [A pass rate without an interval is a guess](/learn/why-one-number-is-not-a-result): what `0.58 [0.38..0.77]` means and why the brackets matter more than the number.