Verifiers: verifiable rewards

The scripts are in the repository, not in the installed package. Clone it, then cd recipes/01-simulate/verifiers before running the commands below. Browse this recipe on GitHub.

A verifier is a reward that is a program, not an opinion, the reward of reinforcement learning with verifiable rewards [1]. It reads a rollout and returns pass, fail, or a partial score, with no model call. Every verifier honors the judge contract, so it plugs into data.grade(judge=...), evaluate, optimize and a gated push exactly where an LLM judge would. From the repo root:

python recipes/01-simulate/verifiers/run.py

No key, no model, seconds. What you will learn: where a verifier reads the candidate and the gold, how to compose checks, and how a verifier’s rows feed optimize and a gated push unchanged. The script shows math (MathEqual), an answer-and-format gate (All([...])), code execution against hidden tests (CodeExec), a JSON-schema check (JSONSchema), and then hands the math rows to wai.optimize(mode="rl") and wai.training_rows to show where the answer key stops travelling.

The pieces

from whileai.simulations.verify import (
    ExactMatch,
    Includes,
    Regex,
    MultipleChoice,  # text
    Numeric,
    MathEqual,  # math
    JSONValid,
    JSONSchema,
    JSONField,  # structured output
    CodeExec,  # run code against tests
    All,
    Any,
    Weighted,
    verifier,  # compose / wrap
)

Where the answer comes from. The candidate is the rollout’s final_text (or the last assistant turn). The gold is read from the row’s privileged.reference — which the training export never projects, so the answer key cannot leak into a training file — with flat fields (answer, target, solution, …) as a fallback. Point any verifier at another column with field=.
Compose. All needs every check to pass (right answer and right format), Any needs one, Weighted is a graded rubric in [0, 1].
Your own. @verifier def f(candidate, reference, row): ... returns a bool or a score, or a (score, reason) pair.
Tool calls. wai.verify.tool_calls(row) (or one message) returns the calls as ToolCall(name, arguments, id) with arguments always a dict, from a rollout row’s flat {"name", "arguments"} and from an exported row’s OpenAI wire shape alike, so a reward that checks “did it open with get_order” scores the same rows the same wherever it runs.

In the loop

The gold travels with the task, not the spec: simulate() writes prompts, rollouts and world state, never an answer key, so a verifiable task set is rows you bring that already carry privileged.reference (or privileged.tests for CodeExec). Score them with the verifier, run the RL gates (reward band, unanimous groups, duplicates), push (the push needs WHILEAI_API_KEY, or wai login):

import whileai.simulations as wai
from whileai.simulations.score.judging import run_judge
from whileai.simulations.verify import MathEqual

from run import MATH_ROWS as rows  # k rollouts per prompt, each with privileged.reference

scored = run_judge(rows, MathEqual())  # the verifier IS the reward
rows, _ = wai.optimize(scored, mode="rl")  # GRPO data, gradient checked
wai.push_rows(rows, "math-rl-v1", gate=True, mode="rl")

On a SimulationData the same step is data.grade(judge=MathEqual()). The optimized rows still carry privileged (they are SDK rows, and a verifier has to be able to re-score them); wai.training_rows(rows) is the export that never projects it, which is what run.py prints at the end. MathEqual decides with Math-Verify (pip install "whileai[math]"), the verifier behind Open R1 and lighteval, and stops with that line when it is missing. On 3,840 held-out MATH-500 completions the string-and-number rule it replaced failed 199 correct answers (\frac 59 vs \frac{5}{9}, \text{(C)} vs C, set order) and passed 64 wrong ones whose last digit matched the gold (6\sqrt{3} against 1+2\sqrt{3}): 6.8% of verdicts.

Code execution safety

CodeExec runs the candidate in a fresh subprocess with isolated mode, a private temp directory, a wall-clock timeout, and CPU/memory caps on POSIX. That stops runaway loops and accidents. It is not a security boundary against hostile code — for untrusted policies, run the verifier inside a container or the hosted sandbox.

References

Lambert, N. et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124, 2024.

Get started

Guides

Concepts

Recipes

Reference

Verifiers: verifiable rewards

The pieces

In the loop

Code execution safety

References

​The pieces

​In the loop

​Code execution safety

​References

The pieces

In the loop

Code execution safety

References