> ## Documentation Index > Fetch the complete documentation index at: https://docs.while.ai/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > Install with `uv add whileai`; import as `import whileai as wai`. > Run the offline path first (`simulator=False`, `wai.seeded_agent`, a callable judge); no key is needed for it. > Report every pass rate with its interval and n, as `scored.pass_at` prints it. # whileai > The platform client: sign in, tracked agents, runs, verdicts. 30 public names. `import whileai`, then `whileai.name`. | Name | What it does | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | [`Anthropic`](#anthropic) | Anthropic's Messages API. | | [`Endpoint`](#endpoint) | Any OpenAI-compatible server you run: vLLM, SGLang, TGI, LM Studio, a trained adapter behind a URL. | | [`Fireworks`](#fireworks) | An open model Fireworks serves, named the way Fireworks names it (`accounts/fireworks/models/`). | | [`Harness`](#harness) | The program around the model: instructions, tools, the loop that runs them, and the facts a reader needs to compare two of them. | | [`Hosted`](#hosted) | The model While hosts, on your account key (`wai login` or `wai.configure(api_key=...)`). | | [`Judge`](#judge) | An LLM judge: a model, a rubric, and the context the agent was under. | | [`Ollama`](#ollama) | A model served by Ollama on this machine. | | [`OpenAI`](#openai) | OpenAI's chat API. | | [`ScoredData`](#scoreddata) | Scored trajectories: the one representation grade and eval share. | | [`SimulationData`](#simulationdata) | *no docstring* | | [`Verifier`](#verifier) | Base class. | | [`compare`](#compare) | Compare an `after` run to a `before` run on pass\@1 and every shared marker, and say whether the change is real. | | [`configure`](#configure) | Set the process defaults. | | [`context`](#context) | Override the settings inside a `with` block, on this thread only. | | [`decontaminate`](#decontaminate) | Drop training rows whose prompt overlaps an evaluation set. | | [`export`](#export) | Write `training_rows` as JSONL, gated so a broken row never reaches the trainer. | | [`hack_scan`](#hack_scan) | Rank the features that separate reward within each ask, against a permutation noise floor, and say what a grouped update would learn. | | [`judge_trust`](#judge_trust) | Measure whether the judge can be trusted, against human labels and under attack. | | [`methods`](#methods) | Training methods as objects, and the config a trainer reads from them. | | [`pass_at`](#pass_at) | Compute pass\@1, pass^k and pass\@k from graded rows, grouped by task. | | [`platform`](#platform) | Report what you trained to the While platform, so a person can decide. | | [`preflight`](#preflight) | Spec-quality report for an agent. | | [`rows`](#rows) | Rows from your own prompts and completions, in the shape every measurement reads. | | [`seeded_agent`](#seeded_agent) | Build a demo agent whose mistakes are on purpose and recorded on the row. | | [`select`](#select) | Keep the rows worth training on, for SFT or RL: `optimize` as an object. | | [`settings`](#settings) | *no docstring* | | [`simulate`](#simulate) | Generate situations for an agent, roll them out, and return the rows. | | [`tool`](#tool) | Turn a typed function into a `Tool`: signature to schema, docstring to text. | | [`verifier`](#verifier) | Decorator: turn `fn(candidate, reference, row) -> score` into a Verifier. | | [`verify`](#verify) | Verifiers: programmatic, verifiable rewards (Lambert et al. 2024, arXiv:2411.15124; Lambert 2025, chapters Reasoning and Tool Use). | ## config Settings once, override per call: `configure()` and `context()`. ### configure ```python theme={"theme":"vitesse-dark"} configure( agent: Any = None, judge: Any = None, simulator: Any = None, api_key: str | None = None, ) -> Settings ``` Defined in [`whileai/config.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/config.py). Set the process defaults. Each argument is a backend object (`wai.OpenAI(...)`), a spec string (`"openai:gpt-4.1-mini"`) or `None` to leave that role as it is. Returns the settings, whose repr says what each role resolves to. * `agent`: the policy under test. * `judge`: the grader; never the same model as the agent by default. * `simulator`: the writer of user messages; the agent's model unless set. * `api_key`: the While account key. A spec string is checked here: `agent="openai/gpt-4.1-mini"` (the DSPy spelling) or `agent="gpt-4.1-mini"` raises and names the string to type, instead of failing later inside `simulate`. ### context ```python theme={"theme":"vitesse-dark"} context( agent: Any = None, judge: Any = None, simulator: Any = None, api_key: str | None = None, ) -> Iterator[Settings] ``` Override the settings inside a `with` block, on this thread only. with wai.context(judge=wai.OpenAI("gpt-4.1")): strict = data.grade(wai.Judge(rubric=RUBRIC)) ### settings ```python theme={"theme":"vitesse-dark"} settings = Settings(agent=default (While hosted), judge=default (While hosted), simulator=default (While hosted), api_key=unset) ``` ## harness The harness: the program around the model, as one object you can run, fingerprint and compare. ### Harness ```python theme={"theme":"vitesse-dark"} class Harness( model: Any = None, instructions: str | None = None, tools: Sequence[Any] | None = None, agent: Callable[[str], Trajectory] | None = None, label: str | None = None, disclosure: Disclosure | None = None, ) ``` Defined in [`whileai/harness.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/harness.py). The program around the model: instructions, tools, the loop that runs them, and the facts a reader needs to compare two of them. Reach for it when the thing you are changing is not the weights. Pass it wherever an agent goes (`simulate(harness)`), and every row carries `harness = {label, hash, model, kind}`, so `attribute` can later say which lever moved the score. `pin()` is the platform record for `tracked.run(harness=)`. * `model`: a backend (`wai.OpenAI("gpt-4.1-mini")`), a `provider:model` string, or `None` for the configured or hosted model. On a command harness it is the name the CLI is told. * `instructions`: the system prompt (a command harness appends or prepends it the way its CLI allows, and the disclosure says which). * `tools`: `@wai.tool` functions, plain functions, or OpenAI function schemas; a command harness lists the tools the CLI ships. * `agent`: a callable `message -> trajectory` to fingerprint, or the runner `Harness.command` builds. `None` is the prompted loop the SDK plays itself. * `label`: the version name the platform shows; `h-` when left out. Name variants `prompt@model` and the Runs page groups by both. * `disclosure`: a `Disclosure`; presets fill it from the CLI flags and the context files present in `cwd`. Lambert 2025, chapter Evaluation: a score is only comparable with the setup held constant, so the setup is what the fingerprint hashes. #### Harness.into\_simulate ```python theme={"theme":"vitesse-dark"} into_simulate( self, tools: Any, system_prompt: str | None, max_turns: int | None, ) -> tuple[Any, Any, str | None, int | None] ``` What `simulate(harness)` hands the engine: the agent to play, and the tools, system prompt and turn cap the call did not name taken from the harness. A prompted harness becomes its model spec so the engine plays the world and the scheduled faults itself; a command or callable harness is played as it is. #### Harness.pin ```python theme={"theme":"vitesse-dark"} pin(self) -> Any ``` The platform record (`whileai.platform.Harness`) for `tracked.run(harness=)`: same label, same fingerprint. #### Harness.stamp ```python theme={"theme":"vitesse-dark"} stamp(self) -> dict[str, Any] ``` What every row from this harness carries under `harness`. #### Harness.tool\_schemas ```python theme={"theme":"vitesse-dark"} tool_schemas(self) -> list[dict[str, Any]] | None ``` The tools as OpenAI function schemas, or `None` when there are none (the engine keeps its "no tools given" branch). ## judge `Judge`: an LLM grader as an object. Configure once, apply to rows. ### Judge ```python theme={"theme":"vitesse-dark"} class Judge( rubric: Any = None, model: Any = None, api_key: str | None = None, policy: str = '', tools: Sequence[dict] | None = None, use_privileged: bool = False, name: str | None = None, ) ``` Defined in [`whileai/judge.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/judge.py). An LLM judge: a model, a rubric, and the context the agent was under. * `rubric`: what doing the job means, as text. A `Rubric` object (`wai.simulations.Rubric`) is scored item by item instead. * `model`: a backend object or spec string. Default: the judge from `wai.configure`, else `WHILEAI_JUDGE`, else the hosted Phi-4. * `api_key`: for that model; the backend object's key, else the provider's environment variable. * `policy` and `tools`: the agent's system prompt and tool schemas, so the judge sees the rules the agent was under. A `SimulationData` supplies its own; set these when grading a bare row list. * `use_privileged`: show the judge each row's `privileged` block (principle, reference, hidden state) the agent never saw. * `name`: recorded on every graded row; defaults to the model name. Reference: LLM-as-a-judge, Lambert 2025, chapter Reward Modeling; Zheng et al. 2023, arXiv:2306.05685 (position and length bias, why `judge_trust` should follow). ## methods Training methods as objects, and the config a trainer reads from them. ### methods Module `whileai.methods`. Training methods as objects, and the config a trainer reads from them. import whileai as wai teacher = wai.Endpoint(url="[https://my-vllm.example/v1](https://my-vllm.example/v1)", model="Qwen/Qwen3-32B") cfg = wai.prime\_rl\_config("refunds-v1", wai.OPD(teacher), model="Qwen/Qwen3-4B", out="opd.toml") print(cfg) # what was written, which knobs the trainer reads, which it ignores # then, on a box with two GPUs and your keys: uv run rl @ opd.toml `method=` takes a string (`"grpo"`, `"sft"`, ...) or one of the objects here. An object is the string with its knobs attached, the way `torch.optim.Adam(lr=)` is "adam" with its knobs: every default is named and cited in `defaults.py`, a bad value is refused on construction with the fix in the message, and the object prints. Nothing here trains. The trainer is prime-rl, TRL or Tinker, on your GPUs with your keys; the objects say what to run and `prime_rl_config` says it in the trainer's own words. * `OPD`, on-policy distillation. The student samples, a frozen teacher scores every sampled token, and the loss is the per-token reverse KL (Agarwal et al. 2023, arXiv:2306.13649; Thinking Machines 2025\). Qwen3 reports the same AIME score as RL at a tenth of the GPU hours (arXiv:2505.09388). * `OPSD`, on-policy self-distillation. The teacher is the same model given privileged context the student never sees: a passing demonstration (Shenfeld et al. 2026, arXiv:2601.19897), the reference answer (Zhao et al. 2026, arXiv:2601.18734), or a successful rollout plus the environment's feedback (Hübotter et al. 2026, arXiv:2601.20802). It learns where GRPO has no gradient, and it hurts thinking models (Kaur et al. 2026, arXiv:2607.05184), so the writer says so. * `Async`, any method with a bound on how many optimizer steps a rollout may lag the policy, and the per-token correction for the gap (Noukhovitch et al. 2024, arXiv:2410.18252; Khatri et al. 2025, arXiv:2510.13786). * `FlashReinforce`, `SAO` and `BPCO`, the single-rollout methods: one trajectory per prompt, no group to take a baseline over, so the baseline is the batch mean (FlashReinforce, Hu et al. 2026) or a critic (SAO, Hou et al. 2026, arXiv:2607.07508; BPCO, Qi et al. 2026, arXiv:2608.23566). They are how a production trace, which comes one per prompt and cannot be re-run, becomes a training signal. Each object's `update(batch)` is the update rule itself, in plain Python, so a trainer (or a test) can apply it to any batch of trajectories and read what it kept and why. On prime-rl all three are refused, with the reason and the fix in the message: its reward advantages are group relative (`grpo`, `max_rl`: reward minus the group mean, zero over a group of one) or an EMA baseline (`rae`), it hosts no value model, and its losses mask per token (`ipo`, `icepop`) with no sequence trust region and no clip. The nearest thing it runs is `"rae"`. * `GroupwiseGrading`, a grader that tells passing rollouts apart, in the reward (GRS) or in the advantage (GAR) (MiMo-V2.6, Xiaomi 2026, technical report section 4.3). It lives in `whileai.groupwise` and is re-exported here; it shapes a TRL reward function rather than writing a prime-rl config. * `prime_rl_config`, the TOML prime-rl reads, from a method object and a taskset. ## models Where a model call goes, as an object whose repr says so. ### Anthropic ```python theme={"theme":"vitesse-dark"} class Anthropic(model: str, api_key: str | None = None) -> None ``` Defined in [`whileai/models.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/models.py). Anthropic's Messages API. Key: `api_key=` or `ANTHROPIC_API_KEY`. ### Endpoint ```python theme={"theme":"vitesse-dark"} class Endpoint(model: str, api_key: str | None = None, url: str = '') -> None ``` Defined in [`whileai/models.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/models.py). Any OpenAI-compatible server you run: vLLM, SGLang, TGI, LM Studio, a trained adapter behind a URL. Key: `api_key=`, else `VLLM_API_KEY` or `OPENAI_API_KEY`; a loopback or plain-http URL needs none. ### Fireworks ```python theme={"theme":"vitesse-dark"} class Fireworks(model: str, api_key: str | None = None) -> None ``` Defined in [`whileai/models.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/models.py). An open model Fireworks serves, named the way Fireworks names it (`accounts/fireworks/models/`). Key: `api_key=` or `FIREWORKS_API_KEY`. ### Hosted ```python theme={"theme":"vitesse-dark"} class Hosted(model: str = '', api_key: str | None = None) -> None ``` Defined in [`whileai/models.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/models.py). The model While hosts, on your account key (`wai login` or `wai.configure(api_key=...)`). The default for every role when nothing else is configured. The agent is a Qwen3-4B; the judge is a Phi-4, a different family on purpose. ### Ollama ```python theme={"theme":"vitesse-dark"} class Ollama(model: str, api_key: str | None = None) -> None ``` Defined in [`whileai/models.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/models.py). A model served by Ollama on this machine. No key. ### OpenAI ```python theme={"theme":"vitesse-dark"} class OpenAI(model: str, api_key: str | None = None) -> None ``` Defined in [`whileai/models.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/models.py). OpenAI's chat API. Key: `api_key=` or `OPENAI_API_KEY`. ## platform Report what you trained to the While platform, so a person can decide. ### platform Module `whileai.platform`. Report what you trained to the While platform, so a person can decide. The platform draws one screen per tracked agent: the held-out score by version with the frontier model as the line to beat, the training curve, what moved on the behaviors you did not train, the judge checks, live traffic on the served version, and cost. This module is how a coding agent fills that screen. The person reads it and presses Promote. Your agent framework stays yours. `track` takes the agent object you already have (OpenAI Agents SDK, Pydantic AI, LangGraph, Claude Agent SDK, or anything with a name, a model and tools) and reads the model, the instructions and the tool list off it to fingerprint the harness. Or describe it by hand:: from whileai.platform import Behavior, Frontier, Harness, Judge, track tracked = track( "refund-bot", model="Qwen/Qwen3-4B", harness=Harness(instructions=SYSTEM\_PROMPT, tools=\["lookup\_order", "issue\_refund"]), frontier=Frontier(name="Sonnet 5", score=81, cost\_per\_1k=18.0), ) tracked.behavior( Behavior( name="refunds", test\_version="v2", n=240, judge=Judge(agreement=0.86, human\_n=60, length\_bias=0.08), noise\_floor=2.4, reward\_is\_judge=False, ) ) tracked.noise\_floor("refunds", base\_a, base\_b, base\_c) # or measure it: base re-run rows run = tracked.run("v4", method="GRPO", targets=\["refunds"], trained\_on=\["refunds-grpo"]) run.log(10, reward=0.41, kl=0.01) # or trainer.add\_callback(wai.TrainerCallback(run)) run.score("refunds", 83, ci=2.7, n=240) # every behavior, not only the targets run.finish(hours=2.1, gpu="1xH100", cost\_usd=31, record=RunRecord( # drawn as one table data=Data(train="refunds-grpo", n\_train=1024, holdout="refunds-test-v2", n\_holdout=240), optimizer=Optimizer(loss\_type="dapo", lr=5e-5, beta=1e-4, num\_generations=8, seed=17), eval=EvalSetup(metric="pass\@1", k=4, run\_std=0.02, run\_std\_runs=3), provenance=Provenance(pins=\{"trl": "1.13.0"}, paper="2503.18892"), )) print(tracked.verdict()) # refunds: v4 beats v3 by 5 (interval excludes zero); 1 regression print(tracked.brief()) # what happened, what it means, what to do next: the top of the Runs page print(\*tracked.evals(), sep="\n") # the Evals table: eight checks per behavior Say what the runs are for, and show your working. The experiment block sits at the top of the Runs page, a figure grid follows the run table, and a note sits under its run. Figures are illustration; the verdict above comes from the scored evals:: tracked.experiment( question="Does GRPO on refunds-grpo lift refunds without moving length?", measure="pass\@1 on refunds-test-v2, n=240, 95% interval", decide="promote when the interval clears the 2.4 noise floor", ) tracked.figure("reward-by-step", fig, caption="Training reward, v4") # plotly Figure or dict run.note("Reward flattened at step 300; the last 100 steps bought nothing.") Every object is a pydantic model, validated before it leaves the process, and each one says which chapter of Lambert 2025 (arXiv:2504.12501) it comes from. Chapters are cited by title because the numbering has moved between editions. Logging never raises into a training loop: points are buffered, sent in batches, and a failed send is retried on the next flush. ## selection `Selection`: the rows worth training on, as an object that prints its report. ### select ```python theme={"theme":"vitesse-dark"} select( source: Any, mode: str | None = None, target: int = 1000, band: tuple[float, float] | None = None, endorsed: Sequence[str] = (), truncated: str = 'drop', output: str | None = None, ) -> Selection ``` Defined in [`whileai/selection.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/selection.py). Keep the rows worth training on, for SFT or RL: `optimize` as an object. * `source`: a `SimulationData`, a `ScoredData`, a row list or a JSONL path. * `mode`: `"rl"` or `"sft"`; defaults to the run's own mode, else `"rl"`. * `target`: about how many rows to keep. * `band`: the RL difficulty band as a pass-rate range, `(0.2, 0.8)` by default (Lambert 2025, chapter Reasoning; Yu et al. 2025 (DAPO), arXiv:2503.14476). * `endorsed`: feature names the reward should track, so the hack scan can call a shortcut a hack. * `truncated`: `"drop"`, `"keep"` or `"penalize"` for rollouts cut at the token cap (DAPO's overlong handling). * `output`: write the kept rows there as JSONL. In both modes a row whose reply quotes its own privileged context (the reference answer, the principle, the hidden world state) is dropped before any other gate and counted in the printed report, so `export` never refuses a row this kept. ## simulations.data SimulationData, conversation rebuild, row export, and the grade entry points that operate on a finished run. ### SimulationData ```python theme={"theme":"vitesse-dark"} class SimulationData( trajectories: list[dict] = , arm_yield: dict = , stopped_because: str = 'budget', declared_tools: set = , stages: list[str] = , scaffold_chars: int = 0, degraded: list[str] = , warnings: list[str] = , semantic: bool = False, profile: AgentProfile | None = None, embedder_name: str = '', elapsed_seconds: float = 0.0, rows_per_second: float = 0.0, arm_weights: dict = , scenario_generation_seconds: float = 0.0, embedding_selection_seconds: float = 0.0, rollout_seconds: float = 0.0, row_seconds: list = , unique_prompts: int = 0, scene_brief: str = '', scene_brief_seconds: float = 0.0, first_row_seconds: float = 0.0, semantic_duplicate_rate: float | None = None, unique_behavior_signatures: int = 0, coverage_curve: list[dict] = , coverage: dict = , search: dict = , budget: int = 0, path: str = '', mode: str = 'explore', repeat_policy: str = 'none', n_situations: int | None = None, requests_per_situation: int = 1, rollouts_per_request: int = 1, unique_situations: bool = False, allocator: dict = , writer_model: str = '', user_model: str | None = None, system_prompts: dict[str, str] = , ) -> None ``` Defined in [`whileai/simulations/data.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/data.py). #### SimulationData.compare\_judges ```python theme={"theme":"vitesse-dark"} compare_judges( self, judges: Mapping[str, Any] | Sequence[Any], gold: str = 'gold_reward', allow_model_gold: bool = False, concurrency: int = 8, floors: tuple[float, float] = (0.8, 0.6), ) ``` Grade these rows with several judges and rank them against the gold labels. `judges` maps a name to a spec string (`"typesafe:jev-latest"`), a backend object, a `wai.Judge` or any judge callable. Each grades its own copy of the rows under this run's system prompt and tools, then is scored the way `judge_trust` scores one judge: agreement with a Wilson interval, kappa, leak rate, unsure and unjudged counts, seconds per row. Returns a `JudgeComparison` that prints as a table ranked by kappa; `whileai.judge_comparison.compare_judges` has the full account and takes a bare row list. #### SimulationData.grade ```python theme={"theme":"vitesse-dark"} grade( self, grader=None, judge=None, llm: bool = False, llm_spec: str | None = None, spec: str | None = None, api_key: str | None = None, path: str | None = None, concurrency: int = 32, llm_concurrency: int = 16, version: str | None = None, use_privileged: bool = False, scale: tuple[float, float] | None = None, rubric: str | None = None, trust: str = 'warn', payload_chars: int = 8000, max_tokens: int = 120, ) ``` Grade this run's rows in place with the hosted judge or your own callable. Reach for it right after `simulate` to score the rows without leaving the object. Simulation never calls it on its own. Three paths, chosen by what you pass: * No callable (or `llm=True`): `grade_llm`, the hosted LLM judge (Phi-4 unless `WHILEAI_JUDGE` is set, a different family from the hosted Qwen policy), read from `VLLM_API_KEY`. It writes `reward` (0 or 1) and `reason` onto the rows in place and returns the judge report, a dict with `graded`, `n0`, `n1`, `backend`, `judge_version` and `warnings`. * `grader=`, a plain callable returning a number or `{"reward": ..., "reason": ...}` per row: scores every row in place over `concurrency` threads and returns the run itself, so `data.grade(my_grader).pass_at` reads through. * `judge=`, the contract path: any callable honoring the judge contract, which returns `{"reward": 0 or 1, "reason": str, "markers": {name: value}}` per row (a bare number works too). The contract and its failure modes are written out in full in `whileai.simulations.score.judging` (note the `score.`; there is no `whileai.simulations.judging`). It returns a `ScoredData` of copies: the trajectories here stay unmodified, judge errors are marked per row instead of coerced to 0, and its output feeds `export_dataset` and `simulate(traces=...)` directly. Arguments that matter: * `version`: names the judge's version (model, rubric hash) and is recorded on every scored row; the hosted grader stamps its own. * `rubric`: what doing the job means, as text, for the hosted judge; without it the judge grades the conduct floor only, and says so. * `use_privileged`: `True` shows the hosted judge each row's `privileged` block (principle, reference, hidden state) the agent never saw. * `trust`: the judge check against the rows' human labels (`attach_labels(kind="human")`), run on every path, with the summary stamped on each graded row's `judge_meta["trust"]`. `"warn"` (the default) logs one line when the check failed or no labels exist, `"require"` raises instead, `"off"` skips it. * `path`: write the graded run's JSONL there afterwards. * `spec`: which model judges on the hosted path, as a backend spec (`"typesafe:jev-latest"`, `"openai:gpt-4.1-mini"`); the same keyword `grade_llm`, `pairwise_judge` and `rubric_judge` take. `llm_spec` is its older name and still works. ```python theme={"theme":"vitesse-dark"} data = wai.simulate(agent, tools=TOOLS, simulator=False, budget=16) data.grade(lambda row: 1.0 if row.get("final_text") else 0.0) print(data.pass_at) ``` #### SimulationData.grade\_llm ```python theme={"theme":"vitesse-dark"} grade_llm( self, spec: str | None = None, base_url: str | None = None, model: str | None = None, concurrency: int = 16, api_key: str | None = None, path: str | None = None, limit: int | None = None, prompt: str | None = None, use_privileged: bool = False, rubric: str | None = None, trust: str = 'warn', payload_chars: int = 8000, max_tokens: int = 120, ) ``` Binary 0/1 situation grade. Default brain is the hosted judge (Phi-4 unless `WHILEAI_JUDGE` is set), never the policy model. `use_privileged` shows the judge each row's `privileged` block (principle, reference, hidden state) the agent never saw. `trust` is the judge check against human labels: see `grade`. `payload_chars` caps the evidence the judge reads per row and `max_tokens` its reply (defaults `JUDGE_PAYLOAD_CHARS` and `JUDGE_MAX_TOKENS` in `defaults.py`); both land in `judge_meta`. #### SimulationData.leak\_report ```python theme={"theme":"vitesse-dark"} leak_report(self, min_len: int = 12) -> dict[str, Any] ``` Did any reply quote its own `privileged` block? Reads the trajectories, which still carry the block; `rows()` is scrubbed and would check nothing. Same report as `leak_report`. #### SimulationData.llm\_grade ```python theme={"theme":"vitesse-dark"} llm_grade( self, spec: str | None = None, concurrency: int = 16, api_key: str | None = None, path: str | None = None, ) ``` Advisory LLM pass. Leaves deterministic reward untouched. #### SimulationData.push ```python theme={"theme":"vitesse-dark"} push( self, name: str, api_key: str | None = None, parent: str | None = None, agent: str | None = None, publish: bool = False, description: str | None = None, gate: bool = True, purpose: str = 'train', holdout: float | None = None, endorsed: Sequence[str] = (), strict_hacks: bool = False, ) -> dict ``` Upload this run to your While account as a dataset. `purpose` is the section it lands in on the Datasets page (`"train"` by default; `"holdout"` or `"eval"`). `holdout=0.2` keeps a fifth of the tasks (by `scenario_id`) out of the training set and pushes them as a second, linked dataset with purpose `"holdout"`; the entry carries it as `["holdout"]`. The simulation mode is recorded on both. `api_key` defaults to the `WHILEAI_API_KEY` env var, then the key saved by `wai login`. Pass `parent` (a `ds_...` id) when this run iterates on an existing dataset, so lineage shows on the platform. `publish=True` with an `agent` name also puts it on the public catalog at huggingface.co/while-ai as a card. Returns the registry entry with `datasetId`. `gate=True` runs `publish_gate` first: every graded row gets a `calibration` stamp (per-task pass rate, k, producing policy), and an RL-shaped run that is ungraded or has no mixed group is refused with `PublishGateError`. The gate report is returned as `entry["gate"]`. `gate=False` uploads rows as they are. `endorsed` names what the reward should track (feature-name substrings, e.g. `"tool:lookup_order"`) for the gate's `hack_scan`; `strict_hacks=True` refuses a set whose reward is best explained by something else. #### SimulationData.rank ```python theme={"theme":"vitesse-dark"} rank(self, path: str | None = None) -> dict ``` Second-pass quality scores. Leaves conduct `reward` untouched. Writes `quality`, `quality_reason`, `quality_scores` on each trajectory and rewrites the saved JSONL, or `path` if you pass one. #### SimulationData.report ```python theme={"theme":"vitesse-dark"} report(self) -> dict ``` Run-level coverage summary (same as `data.coverage`). #### SimulationData.select ```python theme={"theme":"vitesse-dark"} select( self, mode: str | None = None, target: int = 1000, band: tuple[float, float] | None = None, endorsed: Sequence[str] = (), truncated: str = 'drop', ) ``` The rows worth training on, as a `Selection` that prints its report. With no `mode`: diverse pass-labeled demonstrations via `select_for_sft`, one of each distinct way of being right before any repeats, junk and duplicate prompts dropped. With `mode="rl"` or `"sft"`: `optimize`, the full gate sequence (privileged leaks, difficulty band, unanimous groups, duplicates, truncation, hack scan), with `band`, `endorsed` and `truncated` as there. Requires graded rows — grade in-loop (`grade=True`, `grader=`) or afterwards with `grade()`. The report lands in `search["selection"]` and on the result's `.report`. ```python theme={"theme":"vitesse-dark"} rows = data.select(mode="rl") print(rows) # what each gate dropped and why rows.export("train.jsonl") ``` #### SimulationData.training\_set ```python theme={"theme":"vitesse-dark"} training_set( self, output: str | None = None, target: int = 1000, validate: bool = True, ) -> dict ``` Select the recommended rows and export them trainer-ready. `select()` picks diverse pass-labeled rows, `export_training` writes them as chat JSONL with this run's system prompt and tools and the tool-call round-trip gate. Returns the export report with the selection report attached; pass `output` to write the file. Raw simulation rows are not the training artifact — this is. ## simulations.export Training-ready rows: system prompt, tool schemas, standard wire format. ### export ```python theme={"theme":"vitesse-dark"} export( source, output: str | None = None, system_prompt: str | None = None, tools: Sequence[dict] | None = None, strip_think: bool = True, validate: bool = True, mask_mode: str = 'assistant', unroll: bool = False, max_tool_output_chars: int | None = None, format: str = 'openai', push_to: str | None = None, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/export.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/export.py). Write `training_rows` as JSONL, gated so a broken row never reaches the trainer. Reach for it when graded rows are ready for SFT. It builds the rows with `training_rows` (system prompt in, tool schemas on, tool calls in the wire format, `` blocks stripped), checks them, and writes one JSON object per line. It returns the report: `path`, `n` and `n_written`, `rewards`, `with_system`, `with_tools`, `format`, `mask_mode`, `tool_call_roundtrip`, `privileged_leaks`, `warnings`, and what was cut or unrolled. `export_dataset` and `export_training` are one function object under two names (`export_dataset is export_training`): same arguments, same file, same report. Write `export_dataset` in new code (it exports a dataset, not a training run); the older spelling is kept so nothing written today breaks. * `source`: a `SimulationData` (system prompt and tools come from its profile), a row list, or a JSONL path; for the last two pass `system_prompt` and `tools`. * `output`: the file to write. With a path source and no `output` it writes `.train.jsonl` next to it. It never overwrites the source. * `validate` (`True`): refuse to write a dataset whose tool calls do not round-trip to structured arguments, or whose assistant turns quote the row's own `privileged` block (the export scrubs the key, not the reply that recited it); `False` exports anyway and leaves the report to read. The leak check reads the source before the scrub, so pass the `SimulationData` or its `trajectories`; rows that already came through `rows()`, `save()` or a file carry nothing to check, and `report["privileged_leaks"]` says so. * `format`: `"openai"` (the default) writes the OpenAI chat-completions wire row: the full `messages` list, `function.arguments` as a JSON string, and the ask carried alongside as `prompt`. `"trl"` writes what `trl` can actually load: conversational SFT rows (arguments as dicts, the ask under `prompt_text`). TRL decides "is this conversational?" from the column set, so a `prompt` string beside `messages` makes it skip the chat template without an error and train on the bare ask, which is why the TRL rows do not carry one. The report says which format and which argument encoding the round-trip gate checked. * `mask_mode` with `format="trl"`: the TRL rows carry no `loss_mask`, because trl 0.19.1's `SFTTrainer` never reads one. Its collator (`DataCollatorForLanguageModeling`) labels every token and unlabels only from two token-level columns the trainer builds itself: `completion_mask` from `prompt`/`completion` rows, and `assistant_masks` from `assistant_only_loss=True`, which needs a `{% generation %}` block in the chat template (Qwen2.5-Instruct has none). So `mask_mode="final"` and `unroll=True` write prompt/completion rows (prompt up to the last assistant turn, that turn as the completion), and TRL trains on exactly what the mask asked. `mask_mode="assistant"` (the default) writes `messages` rows, and TRL trains on every token of them unless `assistant_only_loss=True` is set. The report's `mask_mode` states which of the two TRL will do, and `trained_messages` / `masked_messages` count what TRL trains, not what the SDK intended; a `format="trl"` export on 94 rows was measured at 9x the tokens its `loss_mask` marked (#507). * `push_to`: a Hub repo (`"me/my-set"`) to upload the written file to, with your own token: `HF_TOKEN` from the environment or the login `hf auth login` cached, through `huggingface_hub` (`pip install 'whileai[hf]'`). Private by default; no platform call. `wai.hub.push` is the same upload for a file, a directory or rows you already hold, with `token=` and `private=`. The report gains `hub` (`repo_id`, `url`, `commit`). * `strip_think`, `mask_mode`, `unroll`, `max_tool_output_chars`: passed through to `training_rows`, which explains each. ```python theme={"theme":"vitesse-dark"} report = wai.export_dataset(data, "train.jsonl", format="trl") print(report["n"], report["mask_mode"], report["tool_call_roundtrip"]) ``` ## simulations.generate.offline\_agent The free path, with something to catch. ### seeded\_agent ```python theme={"theme":"vitesse-dark"} seeded_agent( tools: Sequence[dict], rate: float = 0.35, seed: int = 0, behaviors: Sequence[str] | None = None, ) -> Callable[[str], dict] ``` Defined in [`whileai/simulations/generate/offline_agent.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/generate/offline_agent.py). Build a demo agent whose mistakes are on purpose and recorded on the row. Reach for it to try the whole loop offline, with no key and no model: it gives a run something to catch, and each row says what was planted, so a grader or a marker can be checked against the truth. It returns a callable `message -> trajectory` for `simulate(agent=...)`. Honest by default: it picks the tool the ask names, calls it through `world()` (faults fire), and reports what came back. On `rate` of rollouts, drawn deterministically from `seed`, the prompt and the rollout index, it does one thing from `behaviors`: `hedging`, `sycophancy`, `apology` and `boilerplate` add the phrase `style_report` looks for; `ignore_fault` claims success although the tool faulted; `leak` quotes the row's privileged context. Each row it answers carries `seeded`: what it did on purpose, `[]` when it behaved. * `tools`: the tool schemas the agent may call. * `rate` (`SEEDED_RATE`, 0.35): the share of rollouts with one planted mistake, high enough that a 20-row demo run catches every behavior kind at least once; a convention, not a real failure rate. * `seed` (0): fixes which rollouts misbehave and how. * `behaviors` (`SEEDED_BEHAVIORS`): the subset of mistakes to draw from. ```python theme={"theme":"vitesse-dark"} agent = wai.seeded_agent(TOOLS, rate=0.4, seed=3) data = wai.simulate(agent, tools=TOOLS, simulator=False, budget=20) print(sum(1 for row in data.trajectories if row["seeded"]), "planted") ``` ## simulations.schema The typed row: four objects, one flat wire shape, one version stamp. ### rows ```python theme={"theme":"vitesse-dark"} rows( prompts: Sequence[str | Sequence[dict]], completions: Sequence[str | Sequence[str]], reward: Any = None, references: Sequence[Any] | None = None, task_ids: Sequence[str] | None = None, markers: Sequence[Any] | None = None, ) -> RowList ``` Defined in [`whileai/simulations/schema.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/schema.py). Rows from your own prompts and completions, in the shape every measurement reads. The front door for a public benchmark: GSM8K questions and a model's answers become the same rows `simulate()` emits, so `pass_at`, `compare`, `eval_variance`, `holdout_size`, `decontaminate` and `select` take them unchanged. Every row carries the five keys the measurement calls read, and nothing else is required: * `task_id`: what the rollouts of one prompt group under; every interval is over tasks, never rows (Miller 2024, arXiv:2411.00640). Defaults to a stable hash of the prompt text, so the same prompt gets the same id on every call. The engine's `scenario_id` carries the same value. * `prompt`: the prompt text, or the last user turn of a message list. * `final_text`: one completion. * `reward`: a number in \[0, 1]; 0 and 1 are the binary outcome `pass_at` counts and `select` bands on, anything between is a partial score those two skip. * `markers`: `{name: number}`, one behavior measurement per completion. Input as well as output: `compare(proxy="marker:name")` and `eval_variance` read them wherever they came from. `prompts` is a sequence of strings or message lists. `completions` is one string per prompt, or one sequence per prompt (k completions of the same prompt: what `pass_at`'s k-way numbers and `select(mode="rl")` need). `reward` is a `Verifier` (`wai.verify.MathEqual()`), a callable `(prompt, completion)` or `(prompt, completion, reference)` returning a number in \[0, 1], a judge-contract callable `(row) -> verdict`, or the precomputed numbers themselves, nested like `completions` or flat; without it the rows carry no reward and only `decontaminate` has a use for them. `references` is the gold per prompt, kept under `privileged.reference` where a verifier reads it and no training export projects it. `task_ids` names the tasks; `markers` is one dict per completion, nested like `completions`. Returns a `RowList`: a list of typed rows (`schema_version` 1, `to_row` shape) that also feeds `select(...).export()`. A verifier or callable is run through `run_judge`, so the rows say what scored them (`judge_name`, `lineage`) exactly as `data.grade()` writes. Reference: Lambert 2025 (rlhfbook), chapters Evaluation and Reasoning; Miller 2024, arXiv:2411.00640, for the task-level intervals. ```python theme={"theme":"vitesse-dark"} import whileai as wai questions = ["What is 2 + 3?", "What is 7 * 6?", "What is 10 - 4?", "What is 9 / 3?"] gold = ["5", "42", "6", "3"] before = [["5", "4", "5", "5"], ["41", "41", "42", "40"], ["6"] * 4, ["3", "2", "3", "3"]] after = [["5"] * 4, ["42", "42", "42", "41"], ["6"] * 4, ["3"] * 4] base = wai.rows(questions, before, wai.verify.MathEqual(), references=gold) tuned = wai.rows(questions, after, wai.verify.MathEqual(), references=gold) print(wai.pass_at(base)) # pass@1 with its interval print(wai.select(base, mode="rl", band=(0.2, 0.8))) # drops the unanimous groups print(wai.compare(base, tuned)) # is the change real ``` ## simulations.score.delta Did training move the behavior, and did anything else slip? ### compare ```python theme={"theme":"vitesse-dark"} compare( before: Sequence[dict], after: Sequence[dict], target: str | None = None, must_not_regress: Sequence[str] = (), markers: Sequence[str] | None = None, by: str | Callable[[dict], Any] | None = None, run_std: float | Mapping[str, float | None] | None = None, run_std_runs: int | None = None, train_runs: TrainRuns | None = None, proxy: str | None = None, n_boot: int = 2000, seed: int = 0, balance_rollouts: bool = False, alpha: float = 0.05, power: float = 0.8, ceiling_pass_rate: float = 0.9, answered_gap_points: float = 0.1, answered_alpha: float = 0.01, ) -> DeltaReport ``` Defined in [`whileai/simulations/score/delta.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/delta.py). Compare an `after` run to a `before` run on pass\@1 and every shared marker, and say whether the change is real. Reach for it after a change (a prompt edit, a trained adapter, a model swap): both sides are graded rows, ideally on the same pinned tasks (`simulate(tasks=before)`) with the same rollouts per task. It returns a `DeltaReport`, a dict that prints itself. The keys a caller reads first: `headline_verdict` (`PASS` only for a gain the report supports, `NO DIFFERENCE` for an interval over zero, `NOT COMPARABLE (causes)` when the arms cannot be compared, `FAIL` for a regression, a failed guard, or over-optimization), `ok` (the gate: no regression, no failed guard, comparable arms; it does not say the change helped), `metrics` (one entry per metric with its delta, interval and verdict), `warnings` (each naming its fix), `not_comparable`, `n_paired_tasks` and `n_unpaired_tasks`. `print(report)` writes it with `headline_verdict` on the first line (`format_delta_report(report)` is the same string). Arguments that matter: * `target`: the metric the run was meant to move (`"pass_at_1"` or `"marker:name"`); its verdict is the headline. * `proxy`: the metric the run was actually trained on (the training reward as a marker, such as `"marker:first_action"`). When the proxy moved up and the target did not, or the proxy's interval sits entirely above the target's, the report is `over_optimized` and fails: the policy learned something the target does not credit (Gao et al. 2022, arXiv:2210.10760). * `must_not_regress`: metrics whose significant drop fails the report. Marker metrics go by marker name; pass\@1 is `"pass_at_1"`. * `by`: split the target by a group on each row (a top-level row key, a marker name, or a callable `row -> group`). The report gains `groups`, the target compared within each, so a headline that moved cannot hide a kind of prompt that moved the other way. A group whose target dropped significantly is listed in `groups_down` and warned about; it does not flip `ok`, which stays the `must_not_regress` contract (name the group's metric there if it should). * `run_std` and `run_std_runs`: the evaluation's own re-run standard deviation, per metric or as one number, and how many re-runs it was computed from. See the noise floor below. * `train_runs`: the rows of every independent training seed of each arm, when the two sides are separately trained models: a list of row sets for the after arm (the before arm untrained), or `{"before": [...], "after": [...]}` with `None` for an arm that was not trained. See training seeds below. * `alpha` (0.05): the false-positive rate every verdict runs at. Each interval is at `1 - alpha` (`ci95` at the default), the re-run band uses the same quantile, and `family_error` is `1 - (1 - alpha) ** n_metrics`. `power` (0.8) feeds the sizing line (`detectable_effect`, `holdout_size`). `tasks_needed` is sized from the task sd measured on the paired rows in hand (`holdout_size(before=, after=)`), and `tasks_needed_source` says so (`"rows"`); the binomial model, which cannot see the covariance pairing buys, asked for about twice the tasks (#733). * `balance_rollouts` (off): trim every paired task to the rows both sides have, chosen by `seed`, so pass^k and pass\@k share one k; `balanced` says how many rows each side gave up. * `ceiling_pass_rate` (`CEILING_PASS_RATE`, 0.9), `answered_gap_points` (`ANSWERED_GAP_POINTS`, 0.1) and `answered_alpha` (`ANSWERED_P_MAX`, 0.01): the thresholds of the `ceiling` and `answered` flags below. Pairing. Tasks pair by the key `pass_at` groups on; tasks on one side only do not pair, their count is `n_unpaired_tasks`, and when any were dropped a warning says so. `situations` is the cause in `not_comparable` when fewer than half the tasks are on both sides (`paired_share` under 0.5 with tasks on one side only): the arms drew different situation sets, so the delta over the few that pair is between two evals, and the fix is to pin the after side to the before run's tasks (`tasks=`) or compare per tier with `dataset_report`. Unequal rollouts. When a run lost rollouts (`data.report()["rollouts_lost"]`), one arm can sit at k=4 and the other at k=2; the report warns, next to the sizing line, naming both. Unequal k is a precision issue, not a bias: a task's pass rate is its mean over however many rows it has, so rows lost at random leave the paired delta unbiased and only widen its interval (simulated, k=4 against k=2 on half the tasks: mean delta on the true value, interval about 10% wider). Rows lost for a reason are the problem: a timeout that takes the hard runs, an empty reply on the long ones, and the surviving rows on that arm score higher than the arm does. No trimming fixes that; `balance_rollouts` costs precision (another 10% on the interval in the same simulation) and removes no bias (failures dropped on one arm: delta 0.32 untrimmed, 0.32 trimmed, true 0.05). Only re-running the short arm on its short tasks does, and `data.report()["rollouts_lost_by"]` says why the rows went missing. Noise floor. One evaluation is a draw, not a distribution (Lambert 2025, chapter Evaluation, "why many comparisons are unreliable", and its evaluation-variance appendix). With one run on either side and no `run_std`, a target that moved reads `moved_unreplicated` and a warning says how to fix it. Pass `eval_variance(...)["run_std_by_metric"]` as `run_std` so pass\@1 and each marker are judged against their own floor: a marker on a subset of tasks is several times noisier than pass\@1, and pass\@1's floor reads a re-run draw of it as a regression. A scalar applies one floor to every metric. A metric the mapping lacks, or carries as `None`, is never given another metric's floor: it gets `noise_note: "no_replicate_floor"`, a warning, and its verdict rests on the task interval alone. A metric whose delta is inside `noise_band(floor, n_a, n_b, df)` is `within_noise`: not improved, not slipped, not a regression, and a target there reads `within_eval_noise` rather than moved, because re-running the eval moves it that much on its own. The band is `floor * sqrt(1/n_a + 1/n_b)` (the delta is a mean of `n_a` runs against a mean of `n_b`) times 1.96 for a given floor, which is taken as the eval's spread. A floor that came from re-runs is an estimate, not the spread: pass `run_std_runs` (`eval_variance(...)["n_runs"]`) and the band uses the two-sided t quantile at `df = run_std_runs - 1` instead (three re-runs: 4.30 x floor x sqrt(2) with one run per side, not 1.96; under pure noise the 1.96 band lets about one delta in five through at df=2). A given `run_std` without `run_std_runs` keeps 1.96 and a warning names the fix. When both row sets carry two or more `lineage.eval_run` values (`simulate(tasks=..., runs=3)`) the report computes each metric's floor itself, pooled over the two sides, and uses the t quantile at `df = sum(runs - 1)` (three runs per side: 2.78 x floor x sqrt(2/3)); `run_std` is then the headline metric's floor, `run_std_by_metric` has them all, `noise_band` is the headline band, `noise_rule` spells it out, and `eval_runs` says how many runs each side had. An arm handed in through `train_runs` as N row sets averages N eval draws, so `eval_runs` counts those too (three seeds a side: `sqrt(1/3 + 1/3)`); the band used to read only the lineage and came out 1.73x too wide for three seeds (#750). Training seeds. The noise floor measures the eval; a delta between two separately trained models also carries training variance, which the floor cannot see (#356: one recipe read -0.065 \[-0.117, -0.013] on one run and +0.050 on the next, at one seed per arm). Pass `train_runs` and the headline metric gains a between-seed term: each trained arm's per-seed means give a between-seed standard deviation `train_std[arm]`, the delta's variance adds `std**2 / n_seeds` per arm, and `train_ci95` is the interval centred on the across-seed delta `train_delta` and widened in quadrature by the two-sided t quantile at `train_df = sum(n_seeds - 1)` (Lambert 2025, chapter Evaluation; Miller 2024, arXiv:2411.00640, on the variance components a claim rests on). `moved` then needs that interval to exclude zero as well; when it covers zero the verdict is `no_change_detected` and a warning says the seed spread ate the delta. Fewer than `MIN_TRAIN_SEEDS` (2) seeds on a trained arm resolves nothing: the verdict is `unresolved`, the interval and floor lines still print, and the line says "one training seed per arm; add a seed to resolve". Without `train_runs` the report says nothing about training seeds (a prompt edit or a model swap has none); the paper-recipe contract (`recipes/papers/check.py`) reads a one-seed delta as unresolved. Comparability. `config` says what each side was produced with (`pass_at(...).config` per side: task count, k, temperature, max\_tokens, policy and judge versions, prompt hash). A warning names each setting the two sides disagree on, and says so when both sides are the same policy version (Lambert 2025, chapter Evaluation: a comparison is only as good as the settings it was run under). `config[side]["answered_share"]` is the share of rows per side with a spoken reply once `` markup is gone, and every rate is conditional on it. The two shares are compared with a pooled two-proportion z test; when it clears `answered_alpha` the warning states p and the gap, and when the gap also exceeds the re-run band (or `answered_gap_points` with no band) the report fails with `answered` in `not_comparable` and names the mechanism: a reasoning base against a reasoning-suppressed adapter under one shared `max_tokens` runs out of budget inside `` and never answers, so the adapter wins every row the base did not reply to. `not_comparable` lists every such cause under one prefix, `NOT COMPARABLE:`; none are raised here. A replay (`simulate(tasks=...)` or `runs=N`) keeps the writer of the run it replays on `writer_model`, so two runs of one call compare as one writer. Situations nobody's model wrote (a `seeds=` ask, the offline template writer, or a replay of either) count as one writer for this check: nothing there could have moved with the weights. `ceiling` is set when the before side already passes `ceiling_pass_rate` of its tasks, or when fewer than `CEILING_MIN_TASKS_WITH_ROOM` paired tasks (and under half) are not already passed every time: there is little room left for an improvement to show, whatever the training did. ```python theme={"theme":"vitesse-dark"} report = wai.delta_report(base.rows(), tuned.rows(), target="pass_at_1") print(wai.format_delta_report(report)) ``` ## simulations.score.hack\_scan What will the policy learn from this reward? Name it before training. ### hack\_scan ```python theme={"theme":"vitesse-dark"} hack_scan( rows: Sequence[dict], endorsed: Sequence[str] = (), features: Mapping[str, Callable[[dict], float | None]] | None = None, auto: bool = True, top_k: int = 200, n_perm: int = 100, min_obs: int = 20, seeds: int = 12, reward: str = 'reward', seed: int = 0, top_features: int | None = 20, alpha: float = 0.05, ) -> HackScanReport ``` Defined in [`whileai/simulations/score/hack_scan.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/hack_scan.py). Rank the features that separate reward within each ask, against a permutation noise floor, and say what a grouped update would learn. Reach for it before an RL run, and again after, to check that the reward tracks the behavior you meant rather than a shortcut. It returns a `HackScanReport`, a dict that prints itself: `regime` (`train`, `reward_hack`, `pool_exhausted`, `no_signal`, `degenerate`, `unknown`), `tau` (the floor), `features` ranked by |within-ask correlation| with the pooled correlation beside each, `top_feature`, `endorsed_on_top`, `integrity` (the share of the above-floor signal that sits on an endorsed feature), the support numbers (asks all-pass, all-fail, mixed, gradient capacity), and `warnings` in one line each. * `rows`: graded rollouts, several per ask (`mode="rl"`); the reward under `reward` may be 0/1 or partial credit. * `endorsed`: the features the reward is supposed to track, as substrings of feature names (`"lookup_order"` matches `tool:lookup_order` and `contains:lookup_order`; `"marker:grounded"` a marker). Without it the scan still ranks and floors, but cannot call a hack a hack. * `features`: hand-tier columns to add, as `{"name": lambda row: value}`, beside the built-in ones (reply length, tool calls, turns, truncation, one indicator per tool called, every numeric marker). `auto` (`True`) adds the auto tier: presence of the `top_k` (200) most common words and word pairs in the agent's text, the tier that finds the hack nobody listed. * `alpha` (`ALPHA`, 0.05): sets the floor. `tau` is the `1 - alpha` quantile of the strongest feature's |rho| when reward is shuffled within ask (`n_perm` shuffles, 100), so a feature above it clears chance at that rate. * `top_features` (20): caps the ranking in the report (`None` lists all). `min_obs` (20) is the fewest observations a feature needs to be ranked. `degenerate` is the refusal: an ask holds fewer than `MIN_DISTINCT_PER_ASK` distinct rollouts at the median (`distinct_per_ask`) and two or more features sit at |rho| at or above `DEGENERATE_RHO`, exactly collinear with reward and with each other because nothing else could happen at that variety. The ranking cannot separate them and the noise floor is no help (it tells signal from noise, not one perfect explanation from another), so `top_feature` and `integrity` are `None`, `inverted` is empty, no hack is claimed, and `collinear` lists the tied features. The direction is withheld with the name: at that variety an endorsed feature is negative exactly when it fell on the failing trajectory, so the sign is the same coin flip. Collinear features on a varied pool are left alone: there the ranking found two names for one behavior, and a genuinely inverted endorsed feature is still reported. ```python theme={"theme":"vitesse-dark"} scan = wai.hack_scan(data.rows(), endorsed=["tool:lookup_order"]) print(scan["regime"], scan["top_feature"], scan["integrity"]) ``` ## simulations.score.judge\_trust Can the judge be trusted? The reward is only as good as the judge. ### judge\_trust ```python theme={"theme":"vitesse-dark"} judge_trust( rows: Sequence[dict], judge: Callable[[dict], Any] | None = None, gold: str = 'gold_reward', sample: int = 40, seed: int = 0, concurrency: int = 8, probes: str | Sequence[str] | None = None, rubric: str | None = None, min_agreement: float = 0.8, min_kappa: float = 0.6, allow_model_gold: bool = False, length_gap_flag: float = 0.15, flip_flag: float = 0.1, max_skipped_share: float = 0.1, ) -> JudgeTrustReport ``` Defined in [`whileai/simulations/score/judge_trust.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/judge_trust.py). Measure whether the judge can be trusted, against human labels and under attack. Reach for it before training on a judge's rewards: the reward is only as good as the judge. It returns a `JudgeTrustReport`: print it for the block, read it as the dict it has always been. The keys a caller reads first: `ok` (measured and clean), `agreement["agreement"]` and `agreement["ci95"]` (the number and its Wilson interval, not `ci`), `agreement["n"]` (labels compared), `gold_kind` (where the labels came from), and `warnings`, where every line names its own fix. The rest: `held_out_halves` (agreement on two task-hash halves; if they diverge the rubric is fit to its examples), `length_sensitivity` (judge pass rate on short versus long replies among rows humans agreed on, a length bias the labels rule out as real), `perturbation` and `probes` when a judge callable is given, `disagreements` (the review queue of rows the judge and the humans disagree on), `floors`, `n_labeled` and `n_rows`. `print(report)` writes the whole thing (`format_judge_trust(report)` is the same string). The module docstring lays out each check and its source. The floors and flags are keywords with their defaults in `whileai.simulations.defaults`: `min_agreement` (0.8, the human-human agreement of MT-Bench, arXiv:2306.05685), `min_kappa` (0.6, Landis and Koch "substantial"), `length_gap_flag` (0.15), `flip_flag` (0.10) and `max_skipped_share` (0.10, the share of labeled rows the judge may leave out of the agreement count with a fractional reward before `ok` is false; `report["skipped"]` carries the counts). * `rows`: graded rows carrying the judge's `reward`. Rows that also carry `gold` (0/1, default column `gold_reward`, what `attach_labels` writes) feed the agreement, held-out and length checks. Those checks read the `reward` already on the row, so when `judge` is given and the rows' `judge_name` (what `run_judge` and `data.grade` stamp) names another scorer, the report warns and `ok` is false: the agreement would be that scorer's, not the judge's (#683). * `judge`: the judge callable. With it the report re-judges up to `sample` rows twice more, as-is for consistency and with neutral filler appended; flips on the filler run mean the judge pays for length. * `probes`: `"all"` (or a list of names from `PROBES`) adds `judge_probes`, one more pass over the sample per probe; `rubric` feeds the keyword probe. * `min_agreement` (0.8, the human-human agreement of MT-Bench, arXiv:2306.05685) and `min_kappa` (0.6, Landis and Koch "substantial"): the floors `ok` requires. `length_gap_flag` (0.15) and `flip_flag` (0.10) are the flags. All four live in `whileai.simulations.defaults`. * `allow_model_gold`: `False` by default, so model or unknown gold makes `ok` false with the reason; only a person's labels count as a measurement. `ok` is true only when a gold-labeled check ran against a person's labels, the Wilson lower bound of agreement reached `min_agreement`, kappa reached `min_kappa`, and nothing else was flagged. With no labels every check has `n=0`, so `ok` is false with a warning saying the judge is unmeasured, not failed. The perturbation pass is not a substitute: a judge that passes everything is perfectly consistent (Lambert 2025, chapters Reward Modeling and Synthetic Data and Distillation). ```python theme={"theme":"vitesse-dark"} >>> rows = [{"task_id": str(i), "reward": i % 2, "gold_reward": i % 2} for i in range(20)] >>> wai.judge_trust(rows)["agreement"]["agreement"] 1.0 ``` ## simulations.score.judging One judge contract for grading and evaluation, and the loop around it. ### ScoredData ```python theme={"theme":"vitesse-dark"} class ScoredData( rows: list[dict], run_id: str, source: str, judge_name: str, model: str | None = None, ) ``` Defined in [`whileai/simulations/score/judging.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/judging.py). Scored trajectories: the one representation grade and eval share. Iterates as plain dicts, so it feeds `simulate(traces=...)`, `mine_traces`, `export_training` and JSONL writers directly — no conversion scripts. `.rows` is a `RowList`: a list that also answers to being called, so both `scored.rows` and `scored.rows()` give the scored rows. `SimulationData.rows`, what `simulate()` returns, behaves the same way, so the two spellings are interchangeable across `simulate() -> run_judge()`. `.warnings` is the list of hollow-run notes `run_judge` filled; print it before reading any number. #### ScoredData.agreement ```python theme={"theme":"vitesse-dark"} agreement( self, gold: str | Sequence[dict] = 'gold_reward', reward: str = 'reward', ) -> dict[str, Any] ``` Agreement of this run's rewards with a trusted label. See `judge_agreement`. #### ScoredData.compare\_judges ```python theme={"theme":"vitesse-dark"} compare_judges( self, judges: Mapping[str, Any] | Sequence[Any], gold: str = 'gold_reward', allow_model_gold: bool = False, concurrency: int = 8, floors: tuple[float, float] = (0.8, 0.6), ) ``` Grade these rows with several judges and rank them against the gold labels. `judges` maps a name to a spec string (`"typesafe:jev-latest"`), a backend object, a `wai.Judge` or any judge callable. Each grades its own copy of the rows under this run's system prompt and tools, then is scored the way `judge_trust` scores one judge: agreement with a Wilson interval, kappa, leak rate, unsure and unjudged counts, seconds per row. Returns a `JudgeComparison` that prints as a table ranked by kappa; `whileai.judge_comparison.compare_judges` has the full account and takes a bare row list. #### ScoredData.failed\_traces ```python theme={"theme":"vitesse-dark"} failed_traces(self) -> list[dict] ``` Failures, ready to hand to `simulate(traces=...)`. #### ScoredData.partials ```python theme={"theme":"vitesse-dark"} partials(self) -> list[dict] ``` Rows with a continuous reward strictly between 0 and 1. The scalar lane: 1 pass, 0 fail, partials here, None unjudged - every contract-legal reward is visible in exactly one view. #### ScoredData.push ```python theme={"theme":"vitesse-dark"} push(self, name: str, **kwargs: Any) -> dict ``` Upload the scored rows to the platform: `push_rows(self.rows, name, ...)`. Same keywords as `push_rows` (`gate=`, `mode=`, `agent=`, `purpose=`, `parent=`, `endorsed=`, `strict_hacks=`, `holdout=` for a linked holdout set split by task, `publish=` for a public card, `timeout=` for the upload). The graded copy is what a gated RL push needs, and `SimulationData.push` cannot see it: `grade(judge=)` leaves the run's trajectories ungraded on purpose. #### ScoredData.select ```python theme={"theme":"vitesse-dark"} select( self, mode: str = 'rl', target: int = 1000, band: tuple[float, float] | None = None, endorsed: Sequence[str] = (), truncated: str = 'drop', ) ``` The rows worth training on, as a `Selection` that prints its report: `optimize` over the graded copies, `mode="rl"` by default. `band`, `endorsed` and `truncated` as there. #### ScoredData.select\_by\_reward\_range ```python theme={"theme":"vitesse-dark"} select_by_reward_range( self, lo: float, hi: float, inclusive: bool = True, ) -> list[dict] ``` Rows whose numeric reward falls in \[lo, hi] (or (lo, hi)). #### ScoredData.select\_for\_preference ```python theme={"theme":"vitesse-dark"} select_for_preference( self, max_pairs_per_prompt: int = 1, min_margin: float = 1.0, length_match: bool = True, ) -> tuple[list[dict], dict[str, Any]] ``` Chosen/rejected pairs from same-task contrast. Failures earn here. #### ScoredData.select\_for\_rl ```python theme={"theme":"vitesse-dark"} select_for_rl( self, target: int = 1000, lo: float = 0.2, hi: float = 0.8, has_tools: bool = True, ) -> tuple[list[dict], dict[str, Any]] ``` Whole mixed-reward groups for RL; groups never split. `lo` and `hi` default to `DIFFICULTY_BAND` (0.2, 0.8), the same band `select_for_rl` and `optimize` use; they used to be 0.3 and 0.7 here alone. #### ScoredData.select\_for\_sft ```python theme={"theme":"vitesse-dark"} select_for_sft(self, target: int = 1000) -> tuple[list[dict], dict[str, Any]] ``` Diverse correct demonstrations: 1-labeled, deduped by behavior. #### ScoredData.unjudged ```python theme={"theme":"vitesse-dark"} unjudged(self) -> list[dict] ``` Rows the judge could not score. Never treated as failures. ## simulations.score.passat pass\@1, pass^k and pass\@k from the same graded groups. ### pass\_at ```python theme={"theme":"vitesse-dark"} pass_at( rows: Sequence[dict] | Any, k: int | None = None, min_k: int = 4, unanimous_short: bool = False, ) -> PassAt ``` Defined in [`whileai/simulations/score/passat.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/passat.py). Compute pass\@1, pass^k and pass\@k from graded rows, grouped by task. Reach for it after grading a `mode="rl"` run to read the three numbers a task family gives: pass\@1 is the mean per-task pass rate (the headline), pass^k the chance all k repeats pass (reliability), pass\@k the chance at least one of k passes; headroom is pass\@k minus pass\@1. It returns a `PassAt` with those three, their task-bootstrap `ci95` intervals, `n_groups`, `n_rows`, `per_task` (a dict keyed by task), a `note` when a number is missing and why, and `config` (how the rows were made); `str(result)` prints the line and `to_dict()` gives the keys. A task is a situation, not a string. Rows group under `task_key`: the engine's `scenario_id` when the row has one, else `task_id`, else the prompt text. In `mode="rl"` the repeats of one opener share a `scenario_id`, and so do the textured phrasings of one situation, so those phrasings pool into one task on purpose: the question is whether the agent handles the situation, not one wording of it. `compare_runs`, `delta_report`, `eval_variance`, `curriculum` and `group_signal` count tasks with the same key, so `pass_at(rows).n_groups` and `delta_report(...)["n_paired_tasks"]` agree on the same rows. Only binary `reward` (or `qwen_reward`) rows count, the same rule `group_signal` uses; a row whose reward is between 0 and 1 is counted in `n_partial` and named in `note`, and an unjudged row is skipped. The intervals resample tasks, never rows (Miller 2024, arXiv:2411.00640), so they need at least `MIN_CI_TASKS` (3) tasks. Under that, `ci95` is `None` and the `note` says why and what to change. Ten rows that all carry one `task_id` are one task, not ten, and get no interval; when they are ten separate items, give each its own `task_id`. * `rows`: graded rows, or the `SimulationData` holding them. * `k`: the draw size for the k-way numbers. It defaults to the smallest group of two or more repeats, so every such group contributes; groups with fewer than `k` graded repeats are left out of pass^k and pass\@k and counted in `n_groups_at_k`. pass\@1 always averages every group. * `min_k`: 4 (`ROLLOUTS_PER_TASK`), the smallest k tau-bench and tau2-bench report a pass^k on (arXiv:2406.12045 and arXiv:2506.07982). Below it the k-way numbers are `None` with a `note` instead of a number too noisy to act on. * `unanimous_short`: `True` counts a unanimous group shorter than `k` as if it stayed unanimous (pass^k and pass\@k equal to its pass rate, 1 or 0). That is the assumption a successive-allocation run stopped on, and leaving those groups out would score only the tasks that split and inflate the headroom. Mixed short groups still stay out. `.config` says how the rows were produced (`run_config`): task count, k, temperature, max\_tokens, policy and judge versions, prompt hash, with a `mixed` list naming any the rows disagree on. ```python theme={"theme":"vitesse-dark"} >>> import whileai.simulations as wai >>> rows = [{"task_id": t, "reward": r} for t in "abcd" for r in (1, 1, 0, 1)] >>> wai.pass_at(rows).pass_at_1 0.75 ``` ## simulations.score.preflight Inspect the agent before spending simulation budget, and report after. ### preflight ```python theme={"theme":"vitesse-dark"} preflight( tools: Sequence[dict], system_prompt: str = '', rule_cap: int | None = None, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/preflight.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/preflight.py). Spec-quality report for an agent. Report only; nothing is changed. `warnings` is the list a developer should read before generating thousands of rows; `cells` is the covering-grid size the same way `recommend` counts it. `rules` is every clause of the policy (`rule_cap=None`, the default `RULE_AXIS_CAP_REPORT`); a number keeps the first that many in document order, `n_rules_total` says how many the policy has, `rules_truncated` whether any were left off, and a `warnings` line names the count (#391). The generation grid keeps its own cap (`RULE_AXIS_CAP_GRID`); `cells` is counted on the same axis `rules` shows. ## simulations.score.stats Confidence intervals, paired run comparison, and decontamination. ### decontaminate ```python theme={"theme":"vitesse-dark"} decontaminate( rows: Sequence[dict], against: Sequence[Any] | Any, n: int = 8, fields: Sequence[str] = ('prompt',), overlap: float = 0.8, embedder: Callable[[list[str]], Sequence[Sequence[float]]] | None = None, similarity: float = 0.85, ) -> tuple[list[dict], dict[str, Any]] ``` Defined in [`whileai/simulations/score/stats.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/stats.py). Drop training rows whose prompt overlaps an evaluation set. Reach for it before any train-versus-holdout comparison: a held-out task that also sits in the training data measures memory, not the change (Lambert 2025, chapter Evaluation). It returns `(clean_rows, report)`: the rows that survived (a list that also carries the system prompt and tools the input carried, so `select(clean_rows).export()` writes them), and a report with the count under each rule (`n_contaminated` in total), hits per field, the eval text count, and the first offenders with their coverage (or `similarity` for semantic hits). `rules_skipped` names each rule that could not run on these inputs and why (empty when every rule ran), and `notes` says it in a sentence: a zero under a rule that never ran is not a clearance. * `rows`: the training rows. * `against`: one or more evaluation sources: row lists, JSONL paths, or platform dataset ids (`ds_...`). Evaluation prompts, answers and references are the texts compared (not the eval set's own replies). * `fields` (`("prompt",)`): which row texts are checked; prompts only is what Lambert 2025, chapter Evaluation, checks. Add `"final_text"` to ask the stricter question of whether replies reproduce eval answers or references. * `n` (8) and `overlap` (0.8): the near-copy rule, the Llama 2 rule of 8-grams covering 80% of tokens. `overlap=0` restores any-n-gram. * `embedder` and `similarity` (0.85): a callable from a list of texts to one vector per text turns on the semantic rule at that cosine threshold; nothing here imports a model. Four rules, applied in this order, and a row flagged by one is not counted again by the next, so `n_contaminated` is the number of rows dropped: * `same_task` (`n_same_task`): the row's `scenario_id` or `task_id` is an evaluation row's. A task is a situation, not a string (`task_key`), so a rephrasing of an eval situation is the eval situation whatever the words say. It needs an id on both sides: when no evaluation row (or no training row) carries one, the rule does not run, `rules_skipped["same_task"]` says so, and only the text rules stand between the sets. Every eval set not written by `simulate()` (GSM8K, a Hub set, logged traces) is in that case, so read `n_same_task: 0` next to `rules_skipped`. * `exact` (`n_exact`): one of the row's `fields` is an evaluation text verbatim after normalization (case and whitespace). * near copy (`n_near`): one evaluation text covers at least `overlap` of the row's words with shared word `n`-grams. Texts shorter than `n` words match verbatim only. * `semantic` (`n_semantic`), only with `embedder`: the cosine similarity between the row's text and an evaluation prompt is at least `similarity`, and the two carry different task ids or none. It needs evaluation prompts to embed: when no evaluation row has a `prompt`, the rule does not run, `rules_skipped["semantic"]` says so, and a `UserWarning` is raised because you asked for it. One shared n-gram is the test Lambert 2025, chapter Evaluation, uses for free-form sets. Situations written from templates share whole sentences that say nothing about which question was asked, so any-n-gram flags every row of a template-written set; the coverage rule counts a row when one eval text accounts for most of it. Word overlap does not see a paraphrase. A holdout written by re-running the generator on the same briefs was 70% within 0.85 cosine of the training batch and 5 of 133 byte-identical; the 8-gram rule flagged 4 of 101 prompts and the semantic pass 16. With sentence-transformers: ```python theme={"theme":"vitesse-dark"} from sentence_transformers import SentenceTransformer model = SentenceTransformer("BAAI/bge-small-en-v1.5") clean, report = wai.decontaminate( train, against=[holdout], embedder=lambda texts: model.encode(texts, normalize_embeddings=True).tolist(), ) ``` A semantic flag means the two prompts read alike, not that they are the same task: "cancel one reservation" and "cancel three reservations" for different customers scored 0.932 with no shared answer. So where task identity is recorded the `same_task` rule decides and the semantic pass only looks across different tasks, and the report's `notes` say the flag is a question to check, not a verdict. The default stays lexical: `similarity` 0.85 was read off BGE (unrelated prompts score about 0.55 there) and does not transfer to every model, so the pass calibrates it for yours when it can. With eval rows that carry task ids, the 99th percentile of similarity over eval-prompt pairs with different task ids is how alike distinct tasks read to this embedder, and `notes` says it; a threshold below that number flags tasks that merely share a domain, and the note says so when `similarity` is. ```python theme={"theme":"vitesse-dark"} >>> train = [{"prompt": "Where is order 4473?"}, {"prompt": "Cancel order 9911."}] >>> clean, report = wai.decontaminate(train, against=[[{"prompt": "Cancel order 9911."}]]) >>> len(clean), report["n_contaminated"] (1, 1) ``` ## simulations.simulation `simulate()`: the public entry point. ### simulate ```python theme={"theme":"vitesse-dark"} simulate( agent: Any = None, spec: Any = None, tools: list[dict] | None = None, system_prompt: str | None = None, budget: int | None = 1000, time_budget: float | None = None, until: str = 'compute', mode: str = 'explore', situations: int | None = None, requests_per_situation: int | None = None, rollouts_per_request: int | None = None, unique_situations: bool = False, reproducible: bool | None = None, grade: bool | str = False, llm_grade: bool = False, traces: Any = None, grader: Any = None, rubric: str | None = None, strategy: str = 'auto', seeds: list | None = None, scaffold: str | None = None, execute: Callable | None = None, output: str | None = None, tasks: Any = None, runs: int = 1, checkpoint: str | None = None, on_progress: Callable[[dict], None] | None = None, advanced: dict | None = None, repeats: int | None = None, phrasings: int | None = None, repeat_policy: str | None = None, concurrency: int | None = None, simulator: Any = None, user_model: Any = None, backend: Any = None, seed: int | None = None, sampling: dict | None = None, max_turns: int | None = None, avg_turns: float | None = None, fault_rate: float | None = None, temperature: float | None = None, timeout: float | None = None, logprobs: bool | None = None, hard_share: float | None = None, patience: str | None = None, **passed: Any, ) -> SimulationData ``` Defined in [`whileai/simulations/simulation.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/simulation.py). Generate situations for an agent, roll them out, and return the rows. Reach for it first: it is the run everything else reads. Give it the agent and it writes a grid of human asks (ordinary, vague, complex, adversarial), plays each one against the agent, and returns a `SimulationData`: `rows()` (one row per rollout, with the prompt, the tool calls, the final reply and its lineage), `warnings` (plain-words notes, each naming the call that changes it), `report()`, `search` (how the budget was spent), `pass_at` once graded, and `save(path)`. Nothing is graded unless you ask: pass `grade=True` to grade against the rubric with the judge, the same `wai.Judge(rubric=...)` that `data.grade` runs, so rows carry `reward`, `judge_status` and `judge_name` (no key stops before any budget is spent, nothing is substituted), `grade="conduct"` for the deterministic conduct check by name (what the agent did, not whether it did the job; rows carry `label_source="conduct"`), a callable `grader=` to score inside the loop, or grade later with `data.grade(...)` or `grade()`. The agent and the budget: * `agent`: a callable `message -> trajectory`, played single-turn (one message in, one trajectory out); or a backend object (`wai.OpenAI("gpt-4.1-mini")`) or spec string (`"openai:gpt-4.1-mini"`), which the SDK plays multi-turn from `tools` and `system_prompt`. Leave it `None` to play the model `configure(agent=)` set, else the model While hosts, from `tools`, `system_prompt` (alias `policy`) and `backend` (its model); `spec=` is the third way in. * `budget`: rows the run may produce, 1000 by default, a per-run cap when `runs` is above 1 (`runs=3, budget=100` returns up to 300 rows, and `report()["budget_per_run"]` carries the cap). `time_budget` is the same cap in seconds. * `mode`: `"explore"` (default) spends the budget on new coverage. `"rl"` gives every ask several repeats, so pass rates and RL groups exist, and defaults `repeat_policy` to `"successive"`, which stops early on unanimous asks; `"fixed"` gives every ask all k repeats. * `situations` (N, distinct worlds), `phrasings` (n, wordings of one world; alias `requests_per_situation`) and `repeats` (k, independent runs of one wording; alias `rollouts_per_request`) are three independent counts. Do not collapse them. Under `mode="rl"` `repeats` is a floor, not a count: dynamic sampling (Yu et al. 2025 (DAPO), arXiv:2503.14476) re-rolls uneven groups, so some asks end with more than k rollouts and `pass_at` reports the smallest k. Follow-ups branch on the run. `unique_situations=True` (alias `unique`) keeps picking new worlds (n=1, k=1 unless you set them). With `situations=N` the run stops once all N have their rollouts (`stopped_because="situations_exhausted"`), whatever `budget` still allows; a budget above `situations x phrasings x repeats` is not spent. * `concurrency` (32 parallel rollouts), `seed`, `sampling`, `temperature`, `timeout`, `logprobs`, `fault_rate`, `max_turns` / `avg_turns` (model-backed agents only; `avg_turns=1` is one user line and one reply, the follow-up branch never runs): each is `None` unless you set it, and a misspelled keyword is a `TypeError`, never silently ignored. Writer completions are `advanced["completions_per_request"]`; seed openers are `advanced["seed_prompts"]`. * `timeout`: seconds one agent call may take. Unset, it is 300 s or the reply budget at 4 tokens a second, whichever is longer (`max(300, agent_max_tokens / 4)`: 1,024 s at `agent_max_tokens=4096`), so a long reply is not re-rolled for taking the time it was allowed; a call that runs past it is an agent error and is re-rolled up to `repeats` times. Set it when you know the server's rate: `timeout >= agent_max_tokens / tokens-per-second-per-request`. What the situations come from: * `simulator`: the situation writer. `"hosted"` is the default written out, the same as leaving it unset; `False` is the offline template writer, no key needed. `user_model` plays the simulated person (a backend spec; `None` means the writer's model, the agent's own by default). * `seeds`: opening asks the writer keeps and varies. Every seed is run and becomes at least one situation: `situations` is sized up to `len(seeds)` when you pass a smaller number, and the search never spends a seed's slot on an ask it wrote itself. The one thing that can still drop a seed is `budget`, which pays for `len(seeds) * repeats` rows before anything else; when it cannot, `warnings` says which seeds were dropped and `search["seeds_dropped"]` lists them before rolling out. With a callable agent whose world has real ids (order numbers, account names), put those ids in the seeds or the tool descriptions, or the writer invents ids and every rollout is "not found". Seeds are asks to build a run around, not the eval set: to check that a fixed list of asks all ran and how each scored, use `evaluate(eval_set=asks)`. * `traces`: rows or a JSONL path of production traces. The grid then aims at the tools, faults and world states those traces show instead of the whole space (without it the grid comes from the agent's tools and policy alone, a cold start), and any generated row that near-copies a source trace is dropped, so held-out traces stay out of training. Traces reproduce world-visible situations. A failure that lives in how a reply is worded (an unsupported claim, an estimate not labelled as one, two questions where one was asked for) has no world-visible trigger, so traces alone cannot aim at it; put a grader in the loop for those. * `hard_share`: the difficulty dial, the share of situations drawn from the ambiguous, boundary and adversarial tiers, 0.40 by default, where a base fails most often. `search["tier_mix"]` reports the share asked for and the share drawn; `dimensions={"stance": [...]}` pins one axis and keeps the other axes of the grid. * `tasks`: a previous run (a `SimulationData`, its rows, or its JSONL path) whose task set is replayed instead of drawing a new one. Every distinct prompt is rolled out again, on its own `scenario_id` and `scenario_dimensions`, under the same faults and world state, and nothing else is generated. An unpinned re-run draws by seed and, above `concurrency: 1`, by completion order, so it shares only part of its tasks with the first and `compare_runs` drops the rest; pinning is how an A/B (a prompt edit, a model swap, another seed) keeps every pair. k comes from this call's `repeats` when given, otherwise from the pinned run (the most rollouts any of its prompts has), never from this call's `mode`, so `pass_at` reports the same k on both sides. The run stops when every pinned prompt has its rollouts (`stopped_because="tasks_done"`) or the budget is spent. * `runs`: replay the same task set that many times in one call and stamp `lineage.eval_run` (0, 1, 2, ...) on every row, which is what `delta_report` needs before it will call a change real (Lambert 2025, chapter Evaluation and its evaluation-variance appendix: one evaluation is a draw, three give a standard deviation). `simulate(tasks=base, runs=3)` is the usual form; without `tasks=` the first run draws the task set (from `seeds=` when given) and the rest replay it. Between runs nothing changes but the agent's own sampling (same tasks, faults, world state and seed), so a deterministic agent gives identical runs and a zero re-run band. All rows come back in one `SimulationData` (`output=` holds them all); `search["eval_runs"]` lists the rows and stop reason per run, and `eval_variance(data.rows())` splits by `eval_run` on its own. Replayed rows keep the writer of the run they replay on `writer_model` and say `lineage.replayed_from_run`, so `delta_report` on two runs of one call sees one writer. Watching and resuming a long run: * Progress goes to the `whileai.simulations` logger at INFO (and to stderr when nothing listens) every 10 finished rollouts or 10 s, re-rolls and losses counted as events too, so a run that only re-rolls still speaks: `120/2404 rollouts, 601 situations written, 1h2m elapsed, ~19h left, 96 re-rolled, 3 lost (3 agent error)`. `on_progress=` is a callable that receives the same numbers as a dict on every line, whatever the budget: `rows` (in `data.rows()` so far, resumed ones included), `cap`, `landed` (this call), `resumed`, `rerolled` and `rerolled_by` (`agent_error`, `empty_reply`, `tool_markup`), `timed_out` (the agent errors that were call timeouts), `lost` and `lost_by`, `inflight`, `situations`, `elapsed_s`. The run's final counts are `search["rollouts"]`, and `warnings` (plus a `UserWarning`) says so when more rollouts were re-rolled than landed, since the run then spent most of its time on calls that never became rows. * `checkpoint`: a JSONL path every row is appended to the moment it lands, so a killed run keeps its rows. Call again with the same `checkpoint=` and `tasks=` to resume: the rows on disk are loaded, a task with its `repeats` rows is skipped, one with fewer gets only the missing rollouts, and the returned `SimulationData` is the union (`search["rollouts"]["resumed"]` counts the loaded rows; `lineage.resumed` marks each). Without `tasks=` the rows on disk are loaded and count toward `budget`, and the run draws new situations for the rest. `output=` still writes the whole run at the end; `checkpoint=` is the file that survives a kill. What steers the search and answers the tools: * `grader`: a callable judge. Its verdict steers the search the way a tool fault already does: a row the grader failed is re-rolled and its ask is mutated into new ones, so the budget moves toward what the grader catches, not only toward broken tools. A grader that fails reply-form rules is exactly the signal traces cannot give (measured with a 12-rule grader: every rule with a tool-result trigger was reproduced and every rule about the reply's wording was not), and without a grader there is no verdict to steer by, so there is no switch to set: the grader is the switch. A graded failure is a reward under 0.5 (a 0 from a 0/1 judge, a failed verifier, a rubric below half); markers ride along on the row but do not aim on their own, since their direction differs per marker. `search["mutation_aims"]` counts the parents and the mutated rows per aim, `world_fault` and `graded_failure`. To grade beside the loop and still steer by tool faults alone, pass `advanced={"mutate_graded_failures": False}`. * `execute`: your world, a function `(tool, arguments) -> result` that answers every tool call for real, against your repo, database or service. Without it the mock world answers, which fits record-shaped tools and not code. Scheduled faults still apply first. `whileai.simulations.generate.agents.current_rollout` is a thread-local set before each rollout with `prompt`, `rollout_index` and `seed`, so `execute` can tell which run it is answering. * `patience`: how long the simulated person keeps answering the agent's questions. `"normal"` (the default) always tries to answer the first question, and from the second on may walk away (35% on the second, 60% on each after that, drawn per thread so a seeded run reproduces); at any question the person may also leave when it asks for something they could not or would not know. `"short"` walks away sooner (60% then 90%); `"endless"` never walks away, so the person answers every question until the depth cap and no rubric criterion about asking can fail. The odds are a default, not a measurement: to ground them, fit a Kaplan-Meier hazard per question index on source traces and set the levels from it. A row the person left carries `ended_by="user_left"` and ends on the agent's question; `search["ended_on_question"]` is `\{"share", "n", "user_left"\}`: of `n` rows, the share that ended on a question and how many of those the person left. * `scaffold`: generation-only guidance appended to the system prompt of the model-backed teacher during rollout (and to the scene writer). It never enters `profile.policy`, so exports and evals stay on the plain policy; it is ignored for callable agents. Measured to help some agents and hurt others, so there is no default: configure it per agent. * `reproducible`: the default, `None`, is `True` unless `time_budget` is set. `True` makes a seeded run bit-for-bit at any concurrency, apart from timing fields and per-invocation identity (with `grader=` every row's `lineage.scoring_run_id` names that one scoring pass, a fresh id per call; pass `run_id=` to `run_judge` to pin it): each batch of rollouts finishes, and every verdict of the batch lands, before the next is chosen, so every round sees the same state. A slow rollout holds its batch, so uneven latency costs throughput; pass `False` to trade the same task set on every machine for that throughput. A clock turns it off on its own, since a clock stop lands wherever the run happens to be. It was `False` before 0.111: three runs of one seed at the default concurrency then drew three different task sets (59, 58 and 58 tasks from one `budget=160`), because which rows land before the cap depended on thread timing, and a lesson that wanted the same file on every machine had to pin `concurrency=1`. Either way the draw is a function of the seed and the inputs alone, not of the interpreter: every float sum on the row-selection path is correctly rounded (`math.fsum`), so one seed picks the same rows on CPython 3.10 through 3.13, and a golden-value test pins that draw. Releases before this fix used the builtin `sum`, whose algorithm changed in CPython 3.12, and could swap one row per run between minor versions (issue #410). What every row records. `sampling` (`temperature`, `max_tokens` and `model` as the model backend resolved them), because a result is only comparable with its sampling settings on record (Lambert 2025, chapter Evaluation); a callable agent samples however it samples, so its rows carry `sampling: None` unless you pass `sampling={...}`, which is recorded as given. Three models can take part, the agent (`agent=` or `backend=`), the situation writer (`simulator=`) and the simulated user (`user_model=`), and every row names all three next to `model_version`: `writer_model`, `user_model`, and `judge_meta.model` once graded. When the agent model also wrote the situations or played the user, the run's `degraded` list carries `same_model` and `warnings` says which call separates them, since training on a model's own unfiltered output teaches it its own habits (Lambert 2025, chapter Synthetic Data and Distillation). ```python theme={"theme":"vitesse-dark"} import whileai.simulations as wai TOOLS = [{"type": "function", "function": { "name": "lookup_order", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}}, "required": ["order_id"]}}}] agent = wai.seeded_agent(TOOLS) # an offline demo agent, no key base = wai.simulate(agent, tools=TOOLS, simulator=False, seed=0, mode="rl", repeats=4, budget=32) rerun = wai.simulate(agent, tools=TOOLS, simulator=False, seed=0, tasks=base, mode="rl", # same tasks, k=4 inherited checkpoint="rerun.jsonl", # rows land here as they finish on_progress=print) # or watch the whileai.simulations logger print(base.stopped_because, len(base.rows()), base.warnings) print(rerun.search["rollouts"]) # landed, resumed, rerolled_by, lost_by ``` ## simulations.tools `@tool`: a typed Python function is the tool. ### tool ```python theme={"theme":"vitesse-dark"} tool( fn: Callable[..., Any] | None = None, name: str | None = None, description: str | None = None, ) -> Any ``` Defined in [`whileai/simulations/tools.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/tools.py). Turn a typed function into a `Tool`: signature to schema, docstring to text. Use bare (`@wai.tool`) or with arguments (`@wai.tool(name="lookup")`). `self` and `cls` are skipped, `*args`/`**kwargs` are ignored, a parameter with a default is optional, and `Annotated[T, "note"]` or a docstring `Args:` block gives a parameter its description. ## simulations.verify Verifiers: programmatic, verifiable rewards (Lambert et al. 2024, arXiv:2411.15124; Lambert 2025, chapters Reasoning and Tool Use). ### verify Module `whileai.simulations.verify`. Verifiers: programmatic, verifiable rewards (Lambert et al. 2024, arXiv:2411.15124; Lambert 2025, chapters Reasoning and Tool Use). A verifier is a checker, not a judge: it reads a rollout and decides pass, fail, or a partial score in \[0, 1], with no model call. Every verifier honors the judge contract (`callable(row) -> {"reward", "reason"}`), so it drops into `data.grade(judge=v)`, `evaluate`, `optimize` and a gated `push` exactly where an LLM judge would go. import whileai.simulations as wai from whileai.simulations.verify import MathEqual, All, Regex v = All(\[MathEqual(), Regex(r"\")]) # right answer, and it closed its reasoning scored = data.grade(judge=v) # verifier IS the reward rows, \_ = wai.optimize(scored, mode="rl") # GRPO data with a verifiable reward The gold answer is read from the row's `privileged.reference` (never exported to training rows), with flat fields (`answer`, `target`, ...) as a fallback. Point any verifier at a different column with `field=`. ## simulations.verify.base Verifiers: rewards that are programs, not judges. ### Verifier ```python theme={"theme":"vitesse-dark"} class Verifier(field: str | None = None, name: str | None = None) ``` Defined in [`whileai/simulations/verify/base.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/verify/base.py). Base class. Subclasses implement `check(candidate, reference, row)` and return a float in \[0, 1] (or a bool), or a `(score, reason)` pair. Instances are callables honoring the judge contract, and carry `name` and `kind` so the scored row's `ScorerRef` records what graded it. ### verifier ```python theme={"theme":"vitesse-dark"} verifier(fn: Callable[[str, Any, dict], Any]) -> FunctionVerifier ``` Defined in [`whileai/simulations/verify/base.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/verify/base.py). Decorator: turn `fn(candidate, reference, row) -> score` into a Verifier.