> ## Documentation Index > Fetch the complete documentation index at: https://docs.while.ai/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > Install with `uv add whileai`; import as `import whileai as wai`. > Run the offline path first (`simulator=False`, `wai.seeded_agent`, a callable judge); no key is needed for it. > Report every pass rate with its interval and n, as `scored.pass_at` prints it. # whileai.simulations.score > Grading, judge checks, pass@k, selection for SFT and RL, hack scans. 99 public names. `import whileai.simulations as wai`, then `wai.name`. | Name | What it does | | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`Criterion`](#criterion) | One rubric item. | | [`PassAt`](#passat) | pass\@1 / pass^k / pass\@k over graded groups. | | [`PublishGateError`](#publishgateerror) | The dataset must not be published as it stands. | | [`Rubric`](#rubric) | *no docstring* | | [`ScoredData`](#scoreddata) | Scored trajectories: the one representation grade and eval share. | | [`Spec`](#spec) | A versioned model spec. | | [`Trait`](#trait) | One named expectation. | | [`annotator_agreement`](#annotator_agreement) | How the annotators on `gold_labels` agree with each other. | | [`argument_grounding`](#argument_grounding) | 1.0 when every string argument of every tool call is grounded in the conversation (a rollout with no calls is grounded), else 0.0. | | [`attach_labels`](#attach_labels) | Write hand labels onto rows (in place) and return `(rows, report)`. | | [`attach_rubric`](#attach_rubric) | Put a rubric on each row's `privileged` block (in place). | | [`audit_grades`](#audit_grades) | Estimate the verifier's false-negative rate from a judged sample. | | [`behavior_signature`](#behavior_signature) | Hash of tool sequence, argument provenance, statuses, and outcome shape. | | [`behavioral_markers`](#behavioral_markers) | Rate of each stock marker over `rows` (fraction of rollouts that trip it). | | [`build_preference_pairs`](#build_preference_pairs) | Build same-task chosen/rejected pairs for preference training (DPO-style). | | [`calibrate`](#calibrate) | Stamp `calibration` on every graded row, in place. | | [`classify_failure`](#classify_failure) | Fixed-vocabulary class for a failing row, from its reason and shape. | | [`compare_runs`](#compare_runs) | Test whether run `b` differs from run `a` on one metric, paired by task. | | [`conduct_grade`](#conduct_grade) | Score agent conduct. | | [`coverage_gap`](#coverage_gap) | List the parts of an agent's policy that the asks you already send never reach. | | [`curriculum`](#curriculum) | Split graded tasks into a training curriculum by measured difficulty. | | [`dataset_report`](#dataset_report) | One report a developer reads after simulate/grade: size, signal, mix. | | [`decontaminate`](#decontaminate) | Drop training rows whose prompt overlaps an evaluation set. | | [`dedupe_groups`](#dedupe_groups) | Drop repeat trajectories within one ask. | | [`delta_report`](#delta_report) | Compare an `after` run to a `before` run on pass\@1 and every shared marker, and say whether the change is real. | | [`detectable_effect`](#detectable_effect) | The smallest gain `n_tasks` paired tasks can prove at `power`: `holdout_size` solved for the effect (`FIXED_POINT_STEPS` fixed-point steps, since the after-side variance depends on it). | | [`eval_variance`](#eval_variance) | How much an evaluation moves when the same model is evaluated again (Lambert 2025, chapter Evaluation). | | [`evaluate`](#evaluate) | Judge held-out rollouts under the exact contract `grade` uses. | | [`expected_outcome`](#expected_outcome) | What the checklist will look for on this task, in one sentence. | | [`filter_rl_rows`](#filter_rl_rows) | Split keep/drop. | | [`format_audit`](#format_audit) | The block a person reads: the summary, then the reasons. | | [`format_coverage_gap`](#format_coverage_gap) | The gap report as the block a person actually reads. | | [`format_curriculum`](#format_curriculum) | One-line-per-fact summary for a terminal. | | [`format_delta_report`](#format_delta_report) | The block a person reads: headline, then one line per metric. | | [`format_hack_scan`](#format_hack_scan) | The block a person reads: the regime, the floor, the ranking. | | [`format_hack_scan_diff`](#format_hack_scan_diff) | The block a person reads: what was learned, then the shifts. | | [`format_judge_trust`](#format_judge_trust) | *no docstring* | | [`format_leak_report`](#format_leak_report) | One line per fact, the summary first. | | [`format_markers`](#format_markers) | One line per marker, highest rate first. | | [`format_stages`](#format_stages) | *no docstring* | | [`grounding_report`](#grounding_report) | Over a row set: the share of rollouts with every argument grounded, the share with any call at all, and the most common invented values by tool and key, for a reviewer to look at. | | [`group_signal`](#group_signal) | Within-ask contrast. | | [`hack_scan`](#hack_scan) | Rank the features that separate reward within each ask, against a permutation noise floor, and say what a grouped update would learn. | | [`hack_scan_diff`](#hack_scan_diff) | What the policy learned: the scan before training against the scan after, on rollouts scored by the same reward. | | [`holdout_size`](#holdout_size) | How many paired tasks a holdout needs to prove a gain of `effect`. | | [`judge_agreement`](#judge_agreement) | Agreement between the judge's `reward` and a trusted label. | | [`judge_pairs`](#judge_pairs) | Ask a judge which side of each pair is better, both ways round. | | [`judge_probes`](#judge_probes) | Try the reward hacks a policy finds first on the judge, on purpose. | | [`judge_trust`](#judge_trust) | Measure whether the judge can be trusted, against human labels and under attack. | | [`leak_report`](#leak_report) | Which rows quote their own `privileged` block in the agent's text. | | [`length_report`](#length_report) | Truncated rollouts and asks whose reply lengths are far apart. | | [`load_spec`](#load_spec) | Build a `Spec` from a constitution dict, a list of traits, or a path to a JSON file with either shape. | | [`logprob_report`](#logprob_report) | Coverage and shape of the captured logprobs. | | [`mark_grounding`](#mark_grounding) | Copies of `rows` with `markers["argument_grounding"]` stamped, so `marker_summary`, `delta_report` and the run page read it. | | [`mark_rows`](#mark_rows) | Return copies of `rows` with the stock markers merged into each row's `markers` dict, ready for `marker_summary` / `delta_report`. | | [`marker_names`](#marker_names) | *no docstring* | | [`marker_summary`](#marker_summary) | `metric_summary` for every marker on the rows (or `names`). | | [`mean_kl`](#mean_kl) | Sampled `KL(pi \|\| pi_ref)` per generated token, overall and per task. | | [`metric_summary`](#metric_summary) | Mean over tasks with a task-bootstrap 95% interval. | | [`near_duplicate_prompts`](#near_duplicate_prompts) | Pairs of distinct asks whose token sets overlap at or above `threshold`. | | [`next_round`](#next_round) | The prompt set for the next round, from the last round's graded rollouts. | | [`normalize_judge_result`](#normalize_judge_result) | Coerce one judge return into the contract; never invent a reward. | | [`optimize`](#optimize) | Select the rows worth training on, for SFT or RL, one call after grading. | | [`outcome_check`](#outcome_check) | (outcome, reason, checks). | | [`pairwise_judge`](#pairwise_judge) | A model judge for `judge_pairs`: `judge(a_row, b_row) -> \{"winner": "A" \| "B" \| "tie" \| None, "reason": str\}`. | | [`pass_at`](#pass_at) | Compute pass\@1, pass^k and pass\@k from graded rows, grouped by task. | | [`preflight`](#preflight) | Spec-quality report for an agent. | | [`privileged_context`](#privileged_context) | The teacher's block for a row at birth: `hidden_state` (what the world knows that the ask does not say) and `reference` (what the checklist expects). | | [`publish_gate`](#publish_gate) | Check, calibrate, and report. | | [`rank_rows`](#rank_rows) | Score each row in place. | | [`recommend`](#recommend) | How much data this agent needs, from its own grid. | | [`reference_logprobs`](#reference_logprobs) | Stamp `ref_logprob` on every row: the reference's summed logprob over the tokens the policy generated. | | [`refusal_report`](#refusal_report) | Over-refusal on a benign set (Lambert 2025, chapter Over-optimization, "Over-Refusal"). | | [`retire_solved`](#retire_solved) | Return the rows with every solved task removed. | | [`reward_correlations`](#reward_correlations) | corr(reward, feature) for the cheap features a judge tends to reward by accident: reply length, tool-call count, assistant turns, and the over-optimization signatures Lambert 2025 (chapter Over-optimization) lists (boilerplate, hedging, sycophancy, refusal phrases, 1 when present; see `score.style`), plus every trajectory flag that fired on any row (`lie.*`, `hack.*`, `risk.*`; see `score.trace`). | | [`row_markers`](#row_markers) | The stock markers for one row's final text. | | [`rubric_judge`](#rubric_judge) | A judge for `run_judge` / `data.grade(judge=)` that scores the rubric item by item. | | [`rubric_of`](#rubric_of) | *no docstring* | | [`run_judge`](#run_judge) | Score trajectories with any judge. | | [`score_row`](#score_row) | Score one row. | | [`select_for_rl`](#select_for_rl) | Whole mixed groups up to roughly `target` rows. | | [`select_for_sft`](#select_for_sft) | Diverse correct demonstrations, at most `target` rows. | | [`spec_version`](#spec_version) | The content version of a Spec (or anything `load_spec` accepts). | | [`stage_of`](#stage_of) | The stamped stage, or None. | | [`stage_report`](#stage_report) | Rows per stage, tasks per stage, and the cross-stage leaks: any task used both in `eval` and in a training stage (sft/rm/rl/mid). | | [`staleness_report`](#staleness_report) | Which policies produced these rows, and can an update still use them. | | [`stamp_spec`](#stamp_spec) | Return copies of `rows` tagged with the spec they were produced or graded against: `spec_id` and `spec_version`. | | [`stamp_stage`](#stamp_stage) | Return copies of `rows` with `row["stage"] = stage`. | | [`style_markers`](#style_markers) | Stamp the style markers on every row's `markers` (in place) and return the rows. | | [`style_report`](#style_report) | How much of each signature the replies carry, and whether the reward pays for it. | | [`task_checklist`](#task_checklist) | Judge contract: conduct gated by the task's checkable outcome. | | [`task_key`](#task_key) | The one name every report groups a row's rollouts under. | | [`trace_flag_report`](#trace_flag_report) | How often each flag fires, what a reviewer should read, and whether the reward pays for it. | | [`trace_flags`](#trace_flags) | The flags that fire on one rollout: `{flag: evidence}`. | | [`trace_markers`](#trace_markers) | Stamp the trace markers on every row's `markers` (in place) and return the rows: 1.0 when the family is clean, 0.0 when a flag fired. | | [`trim_out_of_band`](#trim_out_of_band) | Difficulty band filter. | | [`trim_unanimous_groups`](#trim_unanimous_groups) | Drop asks whose k >= `min_k` rollouts all landed 0 or all landed 1. | | [`ungrounded_arguments`](#ungrounded_arguments) | The string arguments of the rollout's tool calls that appear nowhere in the context the call could draw on. | | [`write_rubrics`](#write_rubrics) | Draft one rubric per distinct prompt with a model and attach it to every row of that prompt (`privileged.rubric`, `source="model"`). | ## agreement Does the judge agree with labels you trust? ### judge\_agreement ```python theme={"theme":"vitesse-dark"} judge_agreement( rows: Sequence[dict], gold: str | Sequence[dict] = 'gold_reward', reward: str = 'reward', allow_model_gold: bool = False, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/agreement.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/agreement.py). Agreement between the judge's `reward` and a trusted label. `gold` is either a key on the same rows (default `gold_reward`, the field `attach_labels` fills when you hand-label a sample) or a second row list from another scoring pass, matched by rollout id, scenario id plus rollout index, or prompt plus final text. Only exact 0/1 labels on both sides count; partial scores and unjudged rows are reported as skipped (`n_skipped`), not guessed. A fractional reward is what a `Rubric` of plain principles returns (the mean of its criteria), so a judge built that way loses every partially met row here; `judge_trust` counts that share against `MAX_SKIPPED_SHARE` and says so, since the rows kept are the ones the judge was sure about and agreement over them reads high by construction (#345). Returns `n`, `agreement`, `kappa` (Cohen, chance-corrected), the confusion counts, `pass_when_gold_fail` (the leak rate: gold failures the judge passed) and `fail_when_gold_pass`, both pass rates, `gold_kind` (where the labels came from: `"human"`, `"program"`, `"model"`, `"unknown"` for rows with no record), `ok` (rows were compared and the labels are a person's or a program's), and `warnings`. A second judge pass is model gold; rows with `gold_reward` but no `gold_kind` are unknown; either makes `ok` false with the reason unless `allow_model_gold=True`. A program's labels (`attach_labels(kind="program")`: a verifier, a unit test, a rule over tool calls) are trusted like a person's, since a deterministic rule is at least as strong a gold as a rater (Lambert 2025, chapter Evaluation, verifiable rewards). ## audit Is the verifier failing answers that are right? ### audit\_grades ```python theme={"theme":"vitesse-dark"} audit_grades( rows: Sequence[dict], judge: Callable[[dict], Any], sample: int = 40, passes: int = 0, question: str = "The reference answer is the answer key for this task. Is the reply's final answer correct, meaning it gives the same result as the reference? Differences in formatting, rounding, ordering (unless the task asks for an order), column or field naming, or an equivalent way of expressing the same result do not make it wrong. A missing, different, or partial result does.", seed: int = 0, concurrency: int = 4, timeout: float = 120, judge_name: str | None = None, fn_warn: float = 0.1, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/audit.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/audit.py). Estimate the verifier's false-negative rate from a judged sample. `rows` are graded by the verifier (`run_judge(rows, verifier)` or `data.grade(judge=verifier)`): `reward` 0/1, `reason` from the rule. `sample` failed rows (reward 0, judge ok) are drawn with `seed` and each is put to `judge` (any judge in the `run_judge` contract: `rubric_judge()`, `grade_llm`, your own callable) with the reference in place and `question` as the rubric when the row carries none. A judge reward at or above `PASS_THRESHOLD` (0.5) on a failed row is a false negative. `passes` samples passed rows the same way for the false-positive side. Returns `fn_rate` with `fn_ci95` (Wilson), `estimated_wrong_fails` (the rate over every failed row), `reasons` (the verifier's failure kinds in the sample, each with how many the judge overturned), a few `examples`, `fp_rate` when `passes` > 0, and `warnings`. Above `fn_warn` (`FN_WARN`, 0.10) the summary says to fix the verifier before training; `select_for_rl(audit=report)` and `optimize(audit=)` carry the same warning into the selection. ### format\_audit ```python theme={"theme":"vitesse-dark"} format_audit(report: dict[str, Any]) -> str ``` Defined in [`whileai/simulations/score/audit.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/audit.py). The block a person reads: the summary, then the reasons. ## checklist A task-specific checklist reward, derived from what the world knows. ### expected\_outcome ```python theme={"theme":"vitesse-dark"} expected_outcome(dims: dict | None) -> str | None ``` Defined in [`whileai/simulations/score/checklist.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/checklist.py). What the checklist will look for on this task, in one sentence. The same branches as `outcome_check`, read before the agent has acted. `None` when no rule applies (a compound ask). This is the teacher's `privileged.reference`: the answer key the student must never be shown. ### outcome\_check ```python theme={"theme":"vitesse-dark"} outcome_check(row: dict) -> tuple[int | None, str, dict[str, Any]] ``` Defined in [`whileai/simulations/score/checklist.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/checklist.py). (outcome, reason, checks). `None` when no rule applies to this task. ### privileged\_context ```python theme={"theme":"vitesse-dark"} privileged_context(dims: dict | None, faults: dict | None = None) -> dict ``` Defined in [`whileai/simulations/score/checklist.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/checklist.py). The teacher's block for a row at birth: `hidden_state` (what the world knows that the ask does not say) and `reference` (what the checklist expects). Empty when the task carries neither. ### task\_checklist ```python theme={"theme":"vitesse-dark"} task_checklist( row: dict, declared_tools: set[str] | None = None, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/checklist.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/checklist.py). Judge contract: conduct gated by the task's checkable outcome. ## curriculum Curriculum: order tasks easy-to-hard and retire the solved ones. ### curriculum ```python theme={"theme":"vitesse-dark"} curriculum( rows: Sequence[dict], solved: float = 0.8, floor: float = 0.2, band: tuple[float, float] = (0.2, 0.8), tiers: int = 3, min_rollouts: int = 2, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/curriculum.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/curriculum.py). Split graded tasks into a training curriculum by measured difficulty. A task is *solved* when its pass rate is above `solved` (retire it: an all-pass task is dead gradient). It is *not ready* when its pass rate is below `floor` (hold it: no signal until the policy can sometimes solve it). Everything from `floor` to `solved` inclusive is *trainable*, ordered easy to hard (highest pass rate first) and split into `tiers` difficulty buckets for a staged schedule. The defaults are the two edges of `DEFAULT_BAND` (20% and 80%), the same band `select_for_rl` keeps, so a task at 1 of 8 is not ready here and out of band there for the same reason. Tasks with fewer than `min_rollouts` graded rollouts cannot have a difficulty and are reported separately. Returns a report; nothing is mutated. `band` is recorded and used only to count how many trainable tasks sit in the reasoning-recipe 20-80% sweet spot, so you can see whether the set has usable signal. ### format\_curriculum ```python theme={"theme":"vitesse-dark"} format_curriculum(report: dict[str, Any]) -> str ``` Defined in [`whileai/simulations/score/curriculum.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/curriculum.py). One-line-per-fact summary for a terminal. ### retire\_solved ```python theme={"theme":"vitesse-dark"} retire_solved( rows: Sequence[dict], solved: float = 0.8, min_rollouts: int = 2, ) -> list[dict] ``` Defined in [`whileai/simulations/score/curriculum.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/curriculum.py). Return the rows with every solved task removed. A task above the `solved` pass rate teaches nothing, so its rollouts are dropped; tasks with too few rollouts to judge are kept. ## delta Did training move the behavior, and did anything else slip? ### delta\_report ```python theme={"theme":"vitesse-dark"} delta_report( before: Sequence[dict], after: Sequence[dict], target: str | None = None, must_not_regress: Sequence[str] = (), markers: Sequence[str] | None = None, by: str | Callable[[dict], Any] | None = None, run_std: float | Mapping[str, float | None] | None = None, run_std_runs: int | None = None, train_runs: TrainRuns | None = None, proxy: str | None = None, n_boot: int = 2000, seed: int = 0, balance_rollouts: bool = False, alpha: float = 0.05, power: float = 0.8, ceiling_pass_rate: float = 0.9, answered_gap_points: float = 0.1, answered_alpha: float = 0.01, ) -> DeltaReport ``` Defined in [`whileai/simulations/score/delta.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/delta.py). Compare an `after` run to a `before` run on pass\@1 and every shared marker, and say whether the change is real. Reach for it after a change (a prompt edit, a trained adapter, a model swap): both sides are graded rows, ideally on the same pinned tasks (`simulate(tasks=before)`) with the same rollouts per task. It returns a `DeltaReport`, a dict that prints itself. The keys a caller reads first: `headline_verdict` (`PASS` only for a gain the report supports, `NO DIFFERENCE` for an interval over zero, `NOT COMPARABLE (causes)` when the arms cannot be compared, `FAIL` for a regression, a failed guard, or over-optimization), `ok` (the gate: no regression, no failed guard, comparable arms; it does not say the change helped), `metrics` (one entry per metric with its delta, interval and verdict), `warnings` (each naming its fix), `not_comparable`, `n_paired_tasks` and `n_unpaired_tasks`. `print(report)` writes it with `headline_verdict` on the first line (`format_delta_report(report)` is the same string). Arguments that matter: * `target`: the metric the run was meant to move (`"pass_at_1"` or `"marker:name"`); its verdict is the headline. * `proxy`: the metric the run was actually trained on (the training reward as a marker, such as `"marker:first_action"`). When the proxy moved up and the target did not, or the proxy's interval sits entirely above the target's, the report is `over_optimized` and fails: the policy learned something the target does not credit (Gao et al. 2022, arXiv:2210.10760). * `must_not_regress`: metrics whose significant drop fails the report. Marker metrics go by marker name; pass\@1 is `"pass_at_1"`. * `by`: split the target by a group on each row (a top-level row key, a marker name, or a callable `row -> group`). The report gains `groups`, the target compared within each, so a headline that moved cannot hide a kind of prompt that moved the other way. A group whose target dropped significantly is listed in `groups_down` and warned about; it does not flip `ok`, which stays the `must_not_regress` contract (name the group's metric there if it should). * `run_std` and `run_std_runs`: the evaluation's own re-run standard deviation, per metric or as one number, and how many re-runs it was computed from. See the noise floor below. * `train_runs`: the rows of every independent training seed of each arm, when the two sides are separately trained models: a list of row sets for the after arm (the before arm untrained), or `{"before": [...], "after": [...]}` with `None` for an arm that was not trained. See training seeds below. * `alpha` (0.05): the false-positive rate every verdict runs at. Each interval is at `1 - alpha` (`ci95` at the default), the re-run band uses the same quantile, and `family_error` is `1 - (1 - alpha) ** n_metrics`. `power` (0.8) feeds the sizing line (`detectable_effect`, `holdout_size`). `tasks_needed` is sized from the task sd measured on the paired rows in hand (`holdout_size(before=, after=)`), and `tasks_needed_source` says so (`"rows"`); the binomial model, which cannot see the covariance pairing buys, asked for about twice the tasks (#733). * `balance_rollouts` (off): trim every paired task to the rows both sides have, chosen by `seed`, so pass^k and pass\@k share one k; `balanced` says how many rows each side gave up. * `ceiling_pass_rate` (`CEILING_PASS_RATE`, 0.9), `answered_gap_points` (`ANSWERED_GAP_POINTS`, 0.1) and `answered_alpha` (`ANSWERED_P_MAX`, 0.01): the thresholds of the `ceiling` and `answered` flags below. Pairing. Tasks pair by the key `pass_at` groups on; tasks on one side only do not pair, their count is `n_unpaired_tasks`, and when any were dropped a warning says so. `situations` is the cause in `not_comparable` when fewer than half the tasks are on both sides (`paired_share` under 0.5 with tasks on one side only): the arms drew different situation sets, so the delta over the few that pair is between two evals, and the fix is to pin the after side to the before run's tasks (`tasks=`) or compare per tier with `dataset_report`. Unequal rollouts. When a run lost rollouts (`data.report()["rollouts_lost"]`), one arm can sit at k=4 and the other at k=2; the report warns, next to the sizing line, naming both. Unequal k is a precision issue, not a bias: a task's pass rate is its mean over however many rows it has, so rows lost at random leave the paired delta unbiased and only widen its interval (simulated, k=4 against k=2 on half the tasks: mean delta on the true value, interval about 10% wider). Rows lost for a reason are the problem: a timeout that takes the hard runs, an empty reply on the long ones, and the surviving rows on that arm score higher than the arm does. No trimming fixes that; `balance_rollouts` costs precision (another 10% on the interval in the same simulation) and removes no bias (failures dropped on one arm: delta 0.32 untrimmed, 0.32 trimmed, true 0.05). Only re-running the short arm on its short tasks does, and `data.report()["rollouts_lost_by"]` says why the rows went missing. Noise floor. One evaluation is a draw, not a distribution (Lambert 2025, chapter Evaluation, "why many comparisons are unreliable", and its evaluation-variance appendix). With one run on either side and no `run_std`, a target that moved reads `moved_unreplicated` and a warning says how to fix it. Pass `eval_variance(...)["run_std_by_metric"]` as `run_std` so pass\@1 and each marker are judged against their own floor: a marker on a subset of tasks is several times noisier than pass\@1, and pass\@1's floor reads a re-run draw of it as a regression. A scalar applies one floor to every metric. A metric the mapping lacks, or carries as `None`, is never given another metric's floor: it gets `noise_note: "no_replicate_floor"`, a warning, and its verdict rests on the task interval alone. A metric whose delta is inside `noise_band(floor, n_a, n_b, df)` is `within_noise`: not improved, not slipped, not a regression, and a target there reads `within_eval_noise` rather than moved, because re-running the eval moves it that much on its own. The band is `floor * sqrt(1/n_a + 1/n_b)` (the delta is a mean of `n_a` runs against a mean of `n_b`) times 1.96 for a given floor, which is taken as the eval's spread. A floor that came from re-runs is an estimate, not the spread: pass `run_std_runs` (`eval_variance(...)["n_runs"]`) and the band uses the two-sided t quantile at `df = run_std_runs - 1` instead (three re-runs: 4.30 x floor x sqrt(2) with one run per side, not 1.96; under pure noise the 1.96 band lets about one delta in five through at df=2). A given `run_std` without `run_std_runs` keeps 1.96 and a warning names the fix. When both row sets carry two or more `lineage.eval_run` values (`simulate(tasks=..., runs=3)`) the report computes each metric's floor itself, pooled over the two sides, and uses the t quantile at `df = sum(runs - 1)` (three runs per side: 2.78 x floor x sqrt(2/3)); `run_std` is then the headline metric's floor, `run_std_by_metric` has them all, `noise_band` is the headline band, `noise_rule` spells it out, and `eval_runs` says how many runs each side had. An arm handed in through `train_runs` as N row sets averages N eval draws, so `eval_runs` counts those too (three seeds a side: `sqrt(1/3 + 1/3)`); the band used to read only the lineage and came out 1.73x too wide for three seeds (#750). Training seeds. The noise floor measures the eval; a delta between two separately trained models also carries training variance, which the floor cannot see (#356: one recipe read -0.065 \[-0.117, -0.013] on one run and +0.050 on the next, at one seed per arm). Pass `train_runs` and the headline metric gains a between-seed term: each trained arm's per-seed means give a between-seed standard deviation `train_std[arm]`, the delta's variance adds `std**2 / n_seeds` per arm, and `train_ci95` is the interval centred on the across-seed delta `train_delta` and widened in quadrature by the two-sided t quantile at `train_df = sum(n_seeds - 1)` (Lambert 2025, chapter Evaluation; Miller 2024, arXiv:2411.00640, on the variance components a claim rests on). `moved` then needs that interval to exclude zero as well; when it covers zero the verdict is `no_change_detected` and a warning says the seed spread ate the delta. Fewer than `MIN_TRAIN_SEEDS` (2) seeds on a trained arm resolves nothing: the verdict is `unresolved`, the interval and floor lines still print, and the line says "one training seed per arm; add a seed to resolve". Without `train_runs` the report says nothing about training seeds (a prompt edit or a model swap has none); the paper-recipe contract (`recipes/papers/check.py`) reads a one-seed delta as unresolved. Comparability. `config` says what each side was produced with (`pass_at(...).config` per side: task count, k, temperature, max\_tokens, policy and judge versions, prompt hash). A warning names each setting the two sides disagree on, and says so when both sides are the same policy version (Lambert 2025, chapter Evaluation: a comparison is only as good as the settings it was run under). `config[side]["answered_share"]` is the share of rows per side with a spoken reply once `` markup is gone, and every rate is conditional on it. The two shares are compared with a pooled two-proportion z test; when it clears `answered_alpha` the warning states p and the gap, and when the gap also exceeds the re-run band (or `answered_gap_points` with no band) the report fails with `answered` in `not_comparable` and names the mechanism: a reasoning base against a reasoning-suppressed adapter under one shared `max_tokens` runs out of budget inside `` and never answers, so the adapter wins every row the base did not reply to. `not_comparable` lists every such cause under one prefix, `NOT COMPARABLE:`; none are raised here. A replay (`simulate(tasks=...)` or `runs=N`) keeps the writer of the run it replays on `writer_model`, so two runs of one call compare as one writer. Situations nobody's model wrote (a `seeds=` ask, the offline template writer, or a replay of either) count as one writer for this check: nothing there could have moved with the weights. `ceiling` is set when the before side already passes `ceiling_pass_rate` of its tasks, or when fewer than `CEILING_MIN_TASKS_WITH_ROOM` paired tasks (and under half) are not already passed every time: there is little room left for an improvement to show, whatever the training did. ```python theme={"theme":"vitesse-dark"} report = wai.delta_report(base.rows(), tuned.rows(), target="pass_at_1") print(wai.format_delta_report(report)) ``` ### format\_delta\_report ```python theme={"theme":"vitesse-dark"} format_delta_report(report: dict[str, Any]) -> str ``` Defined in [`whileai/simulations/score/delta.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/delta.py). The block a person reads: headline, then one line per metric. ## grading Deterministic conduct rules for any tool-using agent. ### behavior\_signature ```python theme={"theme":"vitesse-dark"} behavior_signature(trajectory: dict) -> str ``` Defined in [`whileai/simulations/score/grading.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/grading.py). Hash of tool sequence, argument provenance, statuses, and outcome shape. ### conduct\_grade ```python theme={"theme":"vitesse-dark"} conduct_grade(trajectory: dict, declared_tools: set[str] | None = None) -> dict ``` Defined in [`whileai/simulations/score/grading.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/grading.py). Score agent conduct. Tool/sandbox faults are a flag, not a zero. ## grounding Argument grounding: did every tool argument come from the conversation? ### argument\_grounding ```python theme={"theme":"vitesse-dark"} argument_grounding(row: dict, **kwargs: Any) -> float ``` Defined in [`whileai/simulations/score/grounding.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/grounding.py). 1.0 when every string argument of every tool call is grounded in the conversation (a rollout with no calls is grounded), else 0.0. Keyword arguments are those of `ungrounded_arguments`. ### grounding\_report ```python theme={"theme":"vitesse-dark"} grounding_report(rows: Sequence[dict], **kwargs: Any) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/grounding.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/grounding.py). Over a row set: the share of rollouts with every argument grounded, the share with any call at all, and the most common invented values by tool and key, for a reviewer to look at. ### mark\_grounding ```python theme={"theme":"vitesse-dark"} mark_grounding(rows: Sequence[dict], **kwargs: Any) -> list[dict] ``` Defined in [`whileai/simulations/score/grounding.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/grounding.py). Copies of `rows` with `markers["argument_grounding"]` stamped, so `marker_summary`, `delta_report` and the run page read it. ### ungrounded\_arguments ```python theme={"theme":"vitesse-dark"} ungrounded_arguments( row: dict, ignore_keys: Sequence[str] = (), allow: Sequence[str] = (), min_len: int = 3, ) -> list[dict[str, str]] ``` Defined in [`whileai/simulations/score/grounding.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/grounding.py). The string arguments of the rollout's tool calls that appear nowhere in the context the call could draw on. Each entry is `\{"tool", "key", "value"\}`. `ignore_keys` skips argument names that are free text by design (a note, a message body); `allow` lists values that are legal without appearing in the conversation (an enum, a default); strings shorter than `min_len` are skipped. ## hack\_scan What will the policy learn from this reward? Name it before training. ### format\_hack\_scan ```python theme={"theme":"vitesse-dark"} format_hack_scan(report: dict[str, Any], top: int = 12) -> str ``` Defined in [`whileai/simulations/score/hack_scan.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/hack_scan.py). The block a person reads: the regime, the floor, the ranking. ### format\_hack\_scan\_diff ```python theme={"theme":"vitesse-dark"} format_hack_scan_diff(report: dict[str, Any]) -> str ``` Defined in [`whileai/simulations/score/hack_scan.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/hack_scan.py). The block a person reads: what was learned, then the shifts. ### hack\_scan ```python theme={"theme":"vitesse-dark"} hack_scan( rows: Sequence[dict], endorsed: Sequence[str] = (), features: Mapping[str, Callable[[dict], float | None]] | None = None, auto: bool = True, top_k: int = 200, n_perm: int = 100, min_obs: int = 20, seeds: int = 12, reward: str = 'reward', seed: int = 0, top_features: int | None = 20, alpha: float = 0.05, ) -> HackScanReport ``` Defined in [`whileai/simulations/score/hack_scan.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/hack_scan.py). Rank the features that separate reward within each ask, against a permutation noise floor, and say what a grouped update would learn. Reach for it before an RL run, and again after, to check that the reward tracks the behavior you meant rather than a shortcut. It returns a `HackScanReport`, a dict that prints itself: `regime` (`train`, `reward_hack`, `pool_exhausted`, `no_signal`, `degenerate`, `unknown`), `tau` (the floor), `features` ranked by |within-ask correlation| with the pooled correlation beside each, `top_feature`, `endorsed_on_top`, `integrity` (the share of the above-floor signal that sits on an endorsed feature), the support numbers (asks all-pass, all-fail, mixed, gradient capacity), and `warnings` in one line each. * `rows`: graded rollouts, several per ask (`mode="rl"`); the reward under `reward` may be 0/1 or partial credit. * `endorsed`: the features the reward is supposed to track, as substrings of feature names (`"lookup_order"` matches `tool:lookup_order` and `contains:lookup_order`; `"marker:grounded"` a marker). Without it the scan still ranks and floors, but cannot call a hack a hack. * `features`: hand-tier columns to add, as `{"name": lambda row: value}`, beside the built-in ones (reply length, tool calls, turns, truncation, one indicator per tool called, every numeric marker). `auto` (`True`) adds the auto tier: presence of the `top_k` (200) most common words and word pairs in the agent's text, the tier that finds the hack nobody listed. * `alpha` (`ALPHA`, 0.05): sets the floor. `tau` is the `1 - alpha` quantile of the strongest feature's |rho| when reward is shuffled within ask (`n_perm` shuffles, 100), so a feature above it clears chance at that rate. * `top_features` (20): caps the ranking in the report (`None` lists all). `min_obs` (20) is the fewest observations a feature needs to be ranked. `degenerate` is the refusal: an ask holds fewer than `MIN_DISTINCT_PER_ASK` distinct rollouts at the median (`distinct_per_ask`) and two or more features sit at |rho| at or above `DEGENERATE_RHO`, exactly collinear with reward and with each other because nothing else could happen at that variety. The ranking cannot separate them and the noise floor is no help (it tells signal from noise, not one perfect explanation from another), so `top_feature` and `integrity` are `None`, `inverted` is empty, no hack is claimed, and `collinear` lists the tied features. The direction is withheld with the name: at that variety an endorsed feature is negative exactly when it fell on the failing trajectory, so the sign is the same coin flip. Collinear features on a varied pool are left alone: there the ranking found two names for one behavior, and a genuinely inverted endorsed feature is still reported. ```python theme={"theme":"vitesse-dark"} scan = wai.hack_scan(data.rows(), endorsed=["tool:lookup_order"]) print(scan["regime"], scan["top_feature"], scan["integrity"]) ``` ### hack\_scan\_diff ```python theme={"theme":"vitesse-dark"} hack_scan_diff( before: Sequence[dict], after: Sequence[dict], endorsed: Sequence[str] = (), top: int = 10, **scan_kwargs: Any, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/hack_scan.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/hack_scan.py). What the policy learned: the scan before training against the scan after, on rollouts scored by the same reward. A feature that clears the floor after and did not before is what the update moved toward; one that dropped out is what it moved away from. `gained` and `lost` list them with both correlations, `moved` the largest shifts either way, and `learned` is the one line to read: the top gained feature, and whether it is endorsed. `scan_kwargs` reach both `hack_scan` calls. When either side comes back `degenerate`, every feature there is above the floor at |rho| 1 and no feature can be said to have gained it. `learned` says which side could not be read and why, and no hack is claimed; the rows are still listed so the shift is visible. ## hygiene Row hygiene a grouped RL update or a rejection-sampling pass cares about. ### dedupe\_groups ```python theme={"theme":"vitesse-dark"} dedupe_groups(rows: Sequence[dict]) -> tuple[list[dict], dict[str, Any]] ``` Defined in [`whileai/simulations/score/hygiene.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/hygiene.py). Drop repeat trajectories within one ask. Keeps the first of each (behavior signature, normalized reply) pair. Reports how many pairs disagreed on reward, which is label noise the judge introduced. ### length\_report ```python theme={"theme":"vitesse-dark"} length_report(rows: Sequence[dict], max_spread: float = 4.0) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/hygiene.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/hygiene.py). Truncated rollouts and asks whose reply lengths are far apart. `max_spread` is max / median within one ask. ### near\_duplicate\_prompts ```python theme={"theme":"vitesse-dark"} near_duplicate_prompts( rows: Sequence[dict], threshold: float = 0.8, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/hygiene.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/hygiene.py). Pairs of distinct asks whose token sets overlap at or above `threshold`. Quadratic in the number of asks; fine for the few thousand a run produces. Report only. ### reward\_correlations ```python theme={"theme":"vitesse-dark"} reward_correlations( rows: Sequence[dict], threshold: float = 0.3, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/hygiene.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/hygiene.py). corr(reward, feature) for the cheap features a judge tends to reward by accident: reply length, tool-call count, assistant turns, and the over-optimization signatures Lambert 2025 (chapter Over-optimization) lists (boilerplate, hedging, sycophancy, refusal phrases, 1 when present; see `score.style`), plus every trajectory flag that fired on any row (`lie.*`, `hack.*`, `risk.*`; see `score.trace`). Any |corr| at or above `threshold` is flagged. A negative tool-count correlation means the reward pays the policy to do less; a positive phrase or flag correlation means it pays for the tic or the fake. ## judge\_trust Can the judge be trusted? The reward is only as good as the judge. ### format\_judge\_trust ```python theme={"theme":"vitesse-dark"} format_judge_trust(report: dict[str, Any]) -> str ``` Defined in [`whileai/simulations/score/judge_trust.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/judge_trust.py). ### judge\_probes ```python theme={"theme":"vitesse-dark"} judge_probes( rows: Sequence[dict], judge: Callable[[dict], Any], probes: str | Sequence[str] = 'all', rubric: str | None = None, sample: int = 40, seed: int = 0, concurrency: int = 8, flip_flag: float = 0.1, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/judge_trust.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/judge_trust.py). Try the reward hacks a policy finds first on the judge, on purpose. Reach for it when a judge is about to become a training reward: a policy trained on it will find these holes, so find them first (Gao et al. 2022, arXiv:2210.10760). It returns a dict: `probes` (per probe: `n`, `kind`, `pass_before`, `pass_after`, `flips_up`, `flips_down`, `net_flips`, `denominator`, `exploit_rate`, `ci95` (its Wilson interval), `low_power`, `resolves`, `flagged`, `errors`; a probe that applies to no row is `skipped` with the reason), `exploitable_by` (the probes at or over `flip_flag`), one `warnings` line per exploit, and one `notes` line per probe that had too few rows to say. Each probe mutates up to `sample` graded rows one way and re-judges them. An additive probe (filler, the rubric's words, a success claim, the ask echoed, a sycophantic opener) reports `exploit_rate`: the net share of originally failing replies that pass once the text is added, `max(0, flips_up - flips_down) / originally failing`, so a judge whose verdicts churn both ways under the edit reads as noise, not as a hole (#347). A replacement probe (a well-formed tool call with empty arguments, a refusal) reports the share of replies that pass with the content gone. Every rate carries a Wilson interval, the one `agreement` gets, and `flagged` needs at least `PROBE_MIN_N` rows in the denominator: below it one flipped row on ten originally failing is already `flip_flag`, and the interval on 1 of 10 runs 0.02 to 0.40. A probe under the floor is `low_power` with `resolves`, the smallest exploit rate that many rows can tell from `flip_flag` at `POWER` (the normal approximation for one proportion, Miller 2024, section 5), and `flagged` stays false whatever the rate. * `probes`: `"all"` (the default) or names from `PROBES`. * `rubric`: the text the keyword probe draws words from; without it the row's system prompt is used. The `Rubric` handed to `rubric_judge` is not seen here; pass its text as `rubric=` (`judge_trust(rubric=)` forwards it). * `sample` (40), `seed` (0), `concurrency` (8): how many rows to re-judge, which ones, and how many judge calls run at once. * `flip_flag` (`FLIP_FLAG`, 0.10): the exploit rate at which a probe is flagged. `PROBE_MIN_N` (20) is the denominator floor; it is a module default, since this call already carries eight parameters (style rule 3). ### judge\_trust ```python theme={"theme":"vitesse-dark"} judge_trust( rows: Sequence[dict], judge: Callable[[dict], Any] | None = None, gold: str = 'gold_reward', sample: int = 40, seed: int = 0, concurrency: int = 8, probes: str | Sequence[str] | None = None, rubric: str | None = None, min_agreement: float = 0.8, min_kappa: float = 0.6, allow_model_gold: bool = False, length_gap_flag: float = 0.15, flip_flag: float = 0.1, max_skipped_share: float = 0.1, ) -> JudgeTrustReport ``` Defined in [`whileai/simulations/score/judge_trust.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/judge_trust.py). Measure whether the judge can be trusted, against human labels and under attack. Reach for it before training on a judge's rewards: the reward is only as good as the judge. It returns a `JudgeTrustReport`: print it for the block, read it as the dict it has always been. The keys a caller reads first: `ok` (measured and clean), `agreement["agreement"]` and `agreement["ci95"]` (the number and its Wilson interval, not `ci`), `agreement["n"]` (labels compared), `gold_kind` (where the labels came from), and `warnings`, where every line names its own fix. The rest: `held_out_halves` (agreement on two task-hash halves; if they diverge the rubric is fit to its examples), `length_sensitivity` (judge pass rate on short versus long replies among rows humans agreed on, a length bias the labels rule out as real), `perturbation` and `probes` when a judge callable is given, `disagreements` (the review queue of rows the judge and the humans disagree on), `floors`, `n_labeled` and `n_rows`. `print(report)` writes the whole thing (`format_judge_trust(report)` is the same string). The module docstring lays out each check and its source. The floors and flags are keywords with their defaults in `whileai.simulations.defaults`: `min_agreement` (0.8, the human-human agreement of MT-Bench, arXiv:2306.05685), `min_kappa` (0.6, Landis and Koch "substantial"), `length_gap_flag` (0.15), `flip_flag` (0.10) and `max_skipped_share` (0.10, the share of labeled rows the judge may leave out of the agreement count with a fractional reward before `ok` is false; `report["skipped"]` carries the counts). * `rows`: graded rows carrying the judge's `reward`. Rows that also carry `gold` (0/1, default column `gold_reward`, what `attach_labels` writes) feed the agreement, held-out and length checks. Those checks read the `reward` already on the row, so when `judge` is given and the rows' `judge_name` (what `run_judge` and `data.grade` stamp) names another scorer, the report warns and `ok` is false: the agreement would be that scorer's, not the judge's (#683). * `judge`: the judge callable. With it the report re-judges up to `sample` rows twice more, as-is for consistency and with neutral filler appended; flips on the filler run mean the judge pays for length. * `probes`: `"all"` (or a list of names from `PROBES`) adds `judge_probes`, one more pass over the sample per probe; `rubric` feeds the keyword probe. * `min_agreement` (0.8, the human-human agreement of MT-Bench, arXiv:2306.05685) and `min_kappa` (0.6, Landis and Koch "substantial"): the floors `ok` requires. `length_gap_flag` (0.15) and `flip_flag` (0.10) are the flags. All four live in `whileai.simulations.defaults`. * `allow_model_gold`: `False` by default, so model or unknown gold makes `ok` false with the reason; only a person's labels count as a measurement. `ok` is true only when a gold-labeled check ran against a person's labels, the Wilson lower bound of agreement reached `min_agreement`, kappa reached `min_kappa`, and nothing else was flagged. With no labels every check has `n=0`, so `ok` is false with a warning saying the judge is unmeasured, not failed. The perturbation pass is not a substitute: a judge that passes everything is perfectly consistent (Lambert 2025, chapters Reward Modeling and Synthetic Data and Distillation). ```python theme={"theme":"vitesse-dark"} >>> rows = [{"task_id": str(i), "reward": i % 2, "gold_reward": i % 2} for i in range(20)] >>> wai.judge_trust(rows)["agreement"]["agreement"] 1.0 ``` ## judging One judge contract for grading and evaluation, and the loop around it. ### ScoredData ```python theme={"theme":"vitesse-dark"} class ScoredData( rows: list[dict], run_id: str, source: str, judge_name: str, model: str | None = None, ) ``` Defined in [`whileai/simulations/score/judging.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/judging.py). Scored trajectories: the one representation grade and eval share. Iterates as plain dicts, so it feeds `simulate(traces=...)`, `mine_traces`, `export_training` and JSONL writers directly — no conversion scripts. `.rows` is a `RowList`: a list that also answers to being called, so both `scored.rows` and `scored.rows()` give the scored rows. `SimulationData.rows`, what `simulate()` returns, behaves the same way, so the two spellings are interchangeable across `simulate() -> run_judge()`. `.warnings` is the list of hollow-run notes `run_judge` filled; print it before reading any number. #### ScoredData.agreement ```python theme={"theme":"vitesse-dark"} agreement( self, gold: str | Sequence[dict] = 'gold_reward', reward: str = 'reward', ) -> dict[str, Any] ``` Agreement of this run's rewards with a trusted label. See `judge_agreement`. #### ScoredData.compare\_judges ```python theme={"theme":"vitesse-dark"} compare_judges( self, judges: Mapping[str, Any] | Sequence[Any], gold: str = 'gold_reward', allow_model_gold: bool = False, concurrency: int = 8, floors: tuple[float, float] = (0.8, 0.6), ) ``` Grade these rows with several judges and rank them against the gold labels. `judges` maps a name to a spec string (`"typesafe:jev-latest"`), a backend object, a `wai.Judge` or any judge callable. Each grades its own copy of the rows under this run's system prompt and tools, then is scored the way `judge_trust` scores one judge: agreement with a Wilson interval, kappa, leak rate, unsure and unjudged counts, seconds per row. Returns a `JudgeComparison` that prints as a table ranked by kappa; `whileai.judge_comparison.compare_judges` has the full account and takes a bare row list. #### ScoredData.failed\_traces ```python theme={"theme":"vitesse-dark"} failed_traces(self) -> list[dict] ``` Failures, ready to hand to `simulate(traces=...)`. #### ScoredData.partials ```python theme={"theme":"vitesse-dark"} partials(self) -> list[dict] ``` Rows with a continuous reward strictly between 0 and 1. The scalar lane: 1 pass, 0 fail, partials here, None unjudged - every contract-legal reward is visible in exactly one view. #### ScoredData.push ```python theme={"theme":"vitesse-dark"} push(self, name: str, **kwargs: Any) -> dict ``` Upload the scored rows to the platform: `push_rows(self.rows, name, ...)`. Same keywords as `push_rows` (`gate=`, `mode=`, `agent=`, `purpose=`, `parent=`, `endorsed=`, `strict_hacks=`, `holdout=` for a linked holdout set split by task, `publish=` for a public card, `timeout=` for the upload). The graded copy is what a gated RL push needs, and `SimulationData.push` cannot see it: `grade(judge=)` leaves the run's trajectories ungraded on purpose. #### ScoredData.select ```python theme={"theme":"vitesse-dark"} select( self, mode: str = 'rl', target: int = 1000, band: tuple[float, float] | None = None, endorsed: Sequence[str] = (), truncated: str = 'drop', ) ``` The rows worth training on, as a `Selection` that prints its report: `optimize` over the graded copies, `mode="rl"` by default. `band`, `endorsed` and `truncated` as there. #### ScoredData.select\_by\_reward\_range ```python theme={"theme":"vitesse-dark"} select_by_reward_range( self, lo: float, hi: float, inclusive: bool = True, ) -> list[dict] ``` Rows whose numeric reward falls in \[lo, hi] (or (lo, hi)). #### ScoredData.select\_for\_preference ```python theme={"theme":"vitesse-dark"} select_for_preference( self, max_pairs_per_prompt: int = 1, min_margin: float = 1.0, length_match: bool = True, ) -> tuple[list[dict], dict[str, Any]] ``` Chosen/rejected pairs from same-task contrast. Failures earn here. #### ScoredData.select\_for\_rl ```python theme={"theme":"vitesse-dark"} select_for_rl( self, target: int = 1000, lo: float = 0.2, hi: float = 0.8, has_tools: bool = True, ) -> tuple[list[dict], dict[str, Any]] ``` Whole mixed-reward groups for RL; groups never split. `lo` and `hi` default to `DIFFICULTY_BAND` (0.2, 0.8), the same band `select_for_rl` and `optimize` use; they used to be 0.3 and 0.7 here alone. #### ScoredData.select\_for\_sft ```python theme={"theme":"vitesse-dark"} select_for_sft(self, target: int = 1000) -> tuple[list[dict], dict[str, Any]] ``` Diverse correct demonstrations: 1-labeled, deduped by behavior. #### ScoredData.unjudged ```python theme={"theme":"vitesse-dark"} unjudged(self) -> list[dict] ``` Rows the judge could not score. Never treated as failures. ### build\_preference\_pairs ```python theme={"theme":"vitesse-dark"} build_preference_pairs( rows: Sequence[dict], max_pairs_per_prompt: int = 1, min_margin: float = 1.0, length_match: bool = True, ) -> tuple[list[dict], dict[str, Any]] ``` Defined in [`whileai/simulations/score/judging.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/judging.py). Build same-task chosen/rejected pairs for preference training (DPO-style). Reach for it after grading a run with several rollouts per ask: the contrast between a pass and a fail on the same prompt is the training signal, so failures are supply here, not waste. It returns `(pairs, report)`: each pair carries `prompt`, `chosen`, `rejected`, the scores and lineage listed below, and both parents' lineage; the report counts the pairs, the prompts that had a contrast, and how often chosen is still the longer side. A pair exists only where the same prompt has two trajectories whose rewards differ by at least `min_margin`. Rows without a valid judge result never pair. * `min_margin` (1.0): pairs 1-labeled with 0-labeled rows only; `0.5` also admits partial-credit rows against a full pass or fail. * `max_pairs_per_prompt` (1): how many pairs one prompt may contribute. * `length_match` (`True`): each chosen row takes the rejected row closest to it in length. DPO exploits a length gap faster than it learns the behavior (Lambert 2025, chapter Direct Alignment). Each pair keeps what the trainer and the reviewer need to trust it: * `chosen_score` / `rejected_score` / `margin`: the raw scores and their gap, so a margin-aware loss (Llama 2 style) can use them and a reviewer can see how far apart the two really are. * `chosen_model` / `rejected_model` / `same_policy`: which policy produced each side. Preference data works best when both sides come from the policy being trained (Lambert et al. 2024, Tulu 3, arXiv:2411.15124; Lambert 2025, chapter Preference Data); a mixed pair is still a pair, but it is labeled as off-policy. * `length_delta`: chosen reply chars minus rejected, and the report says how often chosen is still the longer side. ```python theme={"theme":"vitesse-dark"} pairs, report = wai.build_preference_pairs(data.rows(), min_margin=1.0) print(len(pairs), report) ``` ### evaluate ```python theme={"theme":"vitesse-dark"} evaluate( rows: Sequence[dict] | Any = None, judge: Callable[[dict], Any] | None = None, grader: Callable[[dict], Any] | None = None, model: str | None = None, eval_set: Sequence[Any] | None = None, judge_name: str | None = None, run_id: str | None = None, concurrency: int = 8, timeout: float | None = None, scale: tuple[float, float] | None = None, tools: Sequence[dict] | Sequence[str] | None = None, ) -> ScoredData ``` Defined in [`whileai/simulations/score/judging.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/judging.py). Judge held-out rollouts under the exact contract `grade` uses. Reach for it to score rows that did not come from the run in hand: production traces, another run's rollouts, a frozen eval set. Same engine, same schema; only the lineage source differs. It returns a `ScoredData`: `rows` (scored copies, each with `reward`, `reason`, `judge_status`, `judge_meta` and a `lineage` record; a judge error is marked on its row, never coerced to 0), `warnings` (read them before the numbers: no rollout called a tool, a declared tool none touched, a marker that fired on no row), `eval_coverage` when `eval_set` was given, and `traces`, which feeds `simulate(traces=...)` to close the loop. * `rows`: the rollouts, or the `SimulationData` holding them, which also supplies `tools`. * `judge` or `grader`: the judge callable, either spelling, not both. `grader` is the doctrine-sketch name. * `eval_set`: the frozen evaluation set, as prompt strings or rows. The result's `eval_coverage` reports which asks the rollouts actually covered and the gap, instead of letting a silent partial eval pass as a full one. * `tools`: the agent's declared tool list (schemas or names), so a declared tool no rollout called is warned about. * `judge_name`, `model`, `run_id`: recorded in each row's `lineage` as the judge, the judged model, and the scoring run (a fresh id per call unless pinned). * `scale`: `(lo, hi)` reads a numeric verdict as a rating on that scale and maps it to a 0 to 1 reward, keeping the raw rating in `judge_meta`. * `concurrency` (8) and `timeout`: judge calls in flight and seconds per call. ```python theme={"theme":"vitesse-dark"} scored = wai.evaluate(traces, judge=my_judge, eval_set=asks) print(scored.warnings, wai.pass_at(scored.rows).pass_at_1) ``` ### normalize\_judge\_result ```python theme={"theme":"vitesse-dark"} normalize_judge_result( raw: Any, scale: tuple[float, float] | None = None, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/judging.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/judging.py). Coerce one judge return into the contract; never invent a reward. `scale=(lo, hi)` reads the judge's number as a rating on that scale (a 1 to 5 Likert, a 0 to 10 score): the row's `reward` is the rating mapped onto \[0, 1] and `judge_meta` keeps `rating` and `scale`. A rating outside the scale is a contract break, as a reward outside \[0, 1] is without one. A dict may carry the number as `rating` instead of `score` when a scale is set. ### run\_judge ```python theme={"theme":"vitesse-dark"} run_judge( rows: Sequence[dict] | Any, judge: Callable[[dict], Any], judge_name: str | None = None, source: str = 'grade', model: str | None = None, run_id: str | None = None, concurrency: int = 8, timeout: float | None = None, version: str | None = None, scale: tuple[float, float] | None = None, tools: Sequence[dict] | Sequence[str] | None = None, ) -> ScoredData ``` Defined in [`whileai/simulations/score/judging.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/judging.py). Score trajectories with any judge. Originals are left unmodified. `tools=` is the agent's declared tool list (or names); with it the result's `warnings` also say which declared tools no rollout called. Passing the `SimulationData` itself as `rows` supplies it. Each scored row is a copy of the input row plus `reward`, `reason`, `judge_status`, `judge_meta`, and a `lineage` record naming the scoring run, its source (grade or eval), the judged model, and the parent trajectory. Rows whose judge result breaks the contract keep `reward=None` and a non-ok status; they are counted, not hidden. `version` names the judge's version (model, prompt hash, whatever would change its labels); it lands in `lineage.judge_version` and reads back as `Judgment.scorer.version`. ## labels Human labels on rows: who said what, and do they agree. ### annotator\_agreement ```python theme={"theme":"vitesse-dark"} annotator_agreement(rows: Sequence[dict]) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/labels.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/labels.py). How the annotators on `gold_labels` agree with each other. `per_annotator`: labels given and pass share. `multi_labeled`: rows with two or more annotators; `unanimous` the share of those where every label matched; `kappa` Cohen's kappa when exactly two annotators labeled the same rows (`pair` names them), else `None`. `disagreements` lists the split rows (prompt, labels) so a person can read the ones the guideline did not settle (Lambert 2025, chapter Preference Data). ### attach\_labels ```python theme={"theme":"vitesse-dark"} attach_labels( rows: Sequence[dict], labels: Any, annotator: str | None = None, kind: str = 'human', replace: bool = False, ) -> tuple[list[dict], dict[str, Any]] ``` Defined in [`whileai/simulations/score/labels.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/labels.py). Write hand labels onto rows (in place) and return `(rows, report)`. `labels` is a JSONL path, a list of dicts, or a `{key: label}` mapping. A dict label names its row by `key` / `rollout_id`, `scenario_id` + `rollout_index`, or `prompt` (+ `final_text`), and carries `label` (or `reward` / `gold_reward`: 0 or 1), and optionally `annotator`, `note`, `ts`. `annotator` here is the default for labels that name none. `kind` is recorded on each label: `"human"` for a person's, `"program"` (or its alias `"verifier"`) for a deterministic rule's (execution match, a unit test, a rule over tool calls), `"model"` for a stronger model's. Any other string raises `ValueError` naming the accepted set, so a typo cannot silently downgrade the gold (#343). Each row gains `gold_labels` (every label, appended unless `replace`), `gold_reward`, the majority of its labels, and `gold_kind`: the one kind every label on the row shares, else `"mixed"`. `judge_trust` and `judge_agreement` count human and program gold as a measurement of the judge: a deterministic rule is at least as strong a gold as a rater, since it cannot be argued into a pass and agrees with itself on every run (Lambert 2025, chapter Evaluation, verifiable rewards). A tie leaves both unset. A key (the mapping's key, or a dict's `key`) is spelled the way the row is: its `rollout_id` when it has one, else `'#'` (what a `simulate` row carries), and a bare `scenario_id` names its one rollout (#751). Labels that name no row, or carry no 0/1 value, are counted and listed, and `warnings` names the key form; when no label at all names a row the call raises, naming the first keys and the keys the rows carry, since a judge check over zero gold reads as "not labeled yet" and not as "wrong key". A key that names several rows (rows pooled from several runs on one pinned task grid share `scenario_id` and `rollout_index`; a bare `scenario_id` on a scenario with several rollouts) raises before any row is changed, naming the fix: a unique `rollout_id` per row, or the composite key (#759); it used to land every label on the first such row and report a clean match. A list or file holding anything but dicts (a bare `[0, 1, 1, 0]`) raises naming the item and the accepted shapes, since a label with no row identity cannot be attached (#685). ## logprobs What the policy's own log-probabilities buy you. ### logprob\_report ```python theme={"theme":"vitesse-dark"} logprob_report(rows: Sequence[dict]) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/logprobs.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/logprobs.py). Coverage and shape of the captured logprobs. `mean_token_logprob` is total logprob over total tokens. The per-row quantiles are of each row's own mean, so one long rollout does not dominate. `corr_reward_confidence` is Pearson between the 0/1 reward and the per-row mean over graded rows: a strong positive value says the judge rewards fluency, not behavior. ### mean\_kl ```python theme={"theme":"vitesse-dark"} mean_kl( rows: Sequence[dict], ref: str | Sequence[dict] = 'ref_logprob', ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/logprobs.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/logprobs.py). Sampled `KL(pi || pi_ref)` per generated token, overall and per task. `ref` is either a key on the same rows holding the reference model's summed logprob over the same tokens (default `ref_logprob`), or a second row list scored under the reference, matched by rollout id, scenario id plus rollout index, or prompt plus final text, carrying `logprob`. Rows missing either side are skipped and counted. Per task the estimate pools tokens across that task's rollouts, which is what a per-task difficulty record wants. ### staleness\_report ```python theme={"theme":"vitesse-dark"} staleness_report( rows: Sequence[dict], base_model: str | None = None, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/logprobs.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/logprobs.py). Which policies produced these rows, and can an update still use them. Noukhovitch et al. 2024, arXiv:2410.18252 (asynchronous RL, truncated importance sampling): rows sampled by an older policy are usable when the row carries the sampler's version and its logprobs so the ratio can be formed; rows from an unknown sampler are not. `versions` counts rows per `policy_version` (`model_version` when the row predates it); `base_model` names the model about to be trained, and rows whose `model_version` differs are `stale`. Coverage says how many rows carry `sampling`, `logprob` and `token_logprobs`. ## markers Stock behavioral markers for the over-optimization signatures (Lambert 2025, chapter Over-optimization). ### behavioral\_markers ```python theme={"theme":"vitesse-dark"} behavioral_markers( rows: Sequence[dict], names: Sequence[str] | None = None, ) -> dict[str, float] ``` Defined in [`whileai/simulations/score/markers.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/markers.py). Rate of each stock marker over `rows` (fraction of rollouts that trip it). The over-optimization dashboard in one call. Deprecated: use `score.style.style_report` for the delta-ready view. ### format\_markers ```python theme={"theme":"vitesse-dark"} format_markers(report: dict[str, float]) -> str ``` Defined in [`whileai/simulations/score/markers.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/markers.py). One line per marker, highest rate first. ### mark\_rows ```python theme={"theme":"vitesse-dark"} mark_rows( rows: Sequence[dict], names: Sequence[str] | None = None, extra: dict[str, Callable[[dict], float]] | None = None, ) -> list[dict] ``` Defined in [`whileai/simulations/score/markers.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/markers.py). Return copies of `rows` with the stock markers merged into each row's `markers` dict, ready for `marker_summary` / `delta_report`. `extra` adds custom named detectors `row -> value`. Existing marker values are kept; stock names overwrite only themselves. Deprecated: presence polarity (1 = tic present) reads a `delta_report` paired comparison backwards. Use `score.style.style_markers`. ### row\_markers ```python theme={"theme":"vitesse-dark"} row_markers(row: dict, names: Sequence[str] | None = None) -> dict[str, int] ``` Defined in [`whileai/simulations/score/markers.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/markers.py). The stock markers for one row's final text. ## optimize Concentrate a big simulated batch into the dataset post-training needs. ### filter\_rl\_rows ```python theme={"theme":"vitesse-dark"} filter_rl_rows( rows: Sequence[dict], has_tools: bool = True, text_gates: bool = True, ) -> tuple[list[dict], dict[str, Any]] ``` Defined in [`whileai/simulations/score/optimize.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/optimize.py). Split keep/drop. Does not mutate `rows`. `text_gates` as in `drop_reason`. ### group\_signal ```python theme={"theme":"vitesse-dark"} group_signal( rows: Sequence[dict], lo: float = 0.2, hi: float = 0.8, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/optimize.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/optimize.py). Within-ask contrast. Signal is a group whose k rollouts disagree. A grouped RL update learns from a mix of 0 and 1 on the same ask, ideally with pass rate p in \[`lo`, `hi`]. Unanimous groups are dead gradient. Groups of one rollout cannot mix and are counted separately, not blamed. ### next\_round ```python theme={"theme":"vitesse-dark"} next_round( prior: Sequence[dict], tasks: Sequence[Any] | None = None, lo: float = 0.2, hi: float = 0.8, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/optimize.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/optimize.py). The prompt set for the next round, from the last round's graded rollouts. A round trained on the file it started from keeps paying for groups that give no gradient: at a 0.65 training reward about half the groups are all-pass or all-fail. The band is the published fix (Lambert 2025, chapter Reasoning: filter to the 20-80% band; Yu et al. 2025 (DAPO), arXiv:2503.14476: dynamic sampling drops groups with no contrast), applied to what the *current* policy does rather than what the base did. `prior` is round N's graded rollouts (`simulate(tasks=..., repeats=k)` on the round-N policy, or the trainer's own sampled rows); each task's pass rate over them decides: inside `[lo, hi]` it is kept, above `hi` it is solved and dropped, below `lo` it is unsolved and dropped. `tasks` restricts the candidates (rows, task dicts with a `prompt`, or prompt strings); a task with no prior rollouts is `unknown` and kept, since nothing says it is flat. `prior` takes either shape: per-rollout rows with a binary `reward`, or one row per task carrying `pass_rate` and `n` (a trainer's per-task table, or the `calibration` stamp this function and `select_for_rl` write). No reply text is read. A `prior` that carries neither is a `UserWarning` and an all-unknown plan, not a silent empty one. Returns `tasks` (one representative row per kept task: the prior row, with `calibration.pass_rate` and the band), the counts `kept`, `dropped_solved`, `dropped_unsolved`, `unknown`, `pass_rates` per task, `band`, `from_policy` (the policy versions the prior rows came from) and `prompt_set_sha`: the identity of the kept set, for lineage on the run. Push the kept rows as the next train set with `parent=` the last one. ### optimize ```python theme={"theme":"vitesse-dark"} optimize( source, mode: str | None = None, target: int = 1000, output: str | None = None, band: tuple[float, float] = (0.2, 0.8), enforce_band: bool = True, select: str = 'top_per_prompt', min_reward: float = 1.0, endorsed: Sequence[str] = (), truncated: str = 'drop', order: str = 'spread', audit: dict[str, Any] | None = None, ) -> tuple[list[dict], dict[str, Any]] ``` Defined in [`whileai/simulations/score/optimize.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/optimize.py). Select the rows worth training on, for SFT or RL, one call after grading. Reach for it once rows carry `reward`. It returns `(rows, report)`: the kept rows in training order, and a report saying which mode ran, what each gate dropped and why, and for RL a `hack_scan` of what the reward is actually tracking. It writes the rows to `output` when given, or to `..jsonl` next to a path source, and never overwrites the source file unless `output` names it explicitly. * `source`: a `SimulationData`, a row list, or a JSONL path. * `mode`: `"sft"` or `"rl"`. Defaults to the run's own mode for a `SimulationData` and to `"rl"` otherwise. SFT picks diverse correct demonstrations (`select_for_sft`); RL keeps whole mixed groups, never a split one (`select_for_rl`). Both drop a row whose reply quotes its own privileged context first (`drop_privileged_leaks`; `privileged_leaks_dropped` in the report), so `export_dataset` never refuses what was kept. * `target`: about how many rows to keep, 1000 by default. * `band`: the RL difficulty band as a pass-rate range, `(0.2, 0.8)` by default: asks the policy always or never solves carry no advantage (Lambert 2025, chapter Reasoning, difficulty filtering at 20 to 80 percent; DAPO's dynamic sampling, arXiv:2503.14476). `enforce_band=False` only ranks out-of-band asks last instead of dropping them. `order` is `"spread"` across pass rates (default) or `"middle"` first. * `select` (`"top_per_prompt"`) and `min_reward` (1.0): the SFT picker and the reward a demonstration needs, as in `select_for_sft`. * `endorsed`: what the reward should track, as substrings of feature names (`"tool:lookup_order"`), so the RL report's `hack_scan` can call a shortcut a hack. * `truncated`: what happens to a rollout cut at the token cap (DAPO's overlong handling, Yu et al. 2025, arXiv:2503.14476): `"drop"` removes it (the default), `"keep"` leaves it in with `overlong=True` and its own reward, `"penalize"` keeps it as a failure that counts (reward 0, the judged score under `reward_before_penalty`). A row counts as truncated when the engine stamped it so (`finish_reason` `"length"`, or a step marked `truncated`), when the grader's `reason` says truncated or cut off, or when the reply text stops without reaching its end; the stamp is read first, since the backend trims a capped reply to its last sentence and a re-grade overwrites the grader's reason (`hygiene.is_truncated`). ```python theme={"theme":"vitesse-dark"} rows, report = wai.optimize(data, mode="rl", endorsed=["tool:lookup_order"]) print(report["mode"], len(rows)) ``` ### recommend ```python theme={"theme":"vitesse-dark"} recommend( tools: Sequence[dict] | None = None, policy: str = '', system_prompt: str | None = None, mode: str = 'sft', target: int | None = None, mixed_rate: float = 0.5, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/optimize.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/optimize.py). How much data this agent needs, from its own grid. No guessing. `system_prompt=` is the same text under `simulate`'s spelling; `policy=` and `system_prompt=` are interchangeable here as there. Grounded two ways: the agent's measured covering grid (every cell wants `SATURATION_COPIES` visits, and selection wants surplus to choose from), and published post-training practice (curated agent SFT lands at 500 to 2,000 trajectories: FireAct 500, LIMA 1,000, AgentTuning 1,866; agent RL uses 8 to 16 rollouts per prompt and drops all-pass/all-fail groups: DAPO 2025, Skywork-OR1 2025). Returns the numbers plus `simulate_kwargs` ready to splat, and `reasoning` lines that show the arithmetic. ### select\_for\_rl ```python theme={"theme":"vitesse-dark"} select_for_rl( rows: Sequence[dict], target: int = 1000, lo: float = 0.2, hi: float = 0.8, enforce_band: bool = True, has_tools: bool = True, dedupe: bool = True, drop_truncated: bool = True, endorsed: Sequence[str] = (), truncated: str = 'drop', order: str = 'spread', prior: Sequence[dict] | None = None, audit: dict[str, Any] | None = None, text_gates: str = 'auto', ) -> tuple[list[dict], dict[str, Any]] ``` Defined in [`whileai/simulations/score/optimize.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/optimize.py). Whole mixed groups up to roughly `target` rows. Groups never split. `text_gates` says whether the gates that read the reply run: the junk and do-nothing checks and the duplicate trim, which key on `final_text`. `"auto"` (the default) runs them when any row carries a reply (`final_text`, an assistant `messages` turn, or a tool step) and skips them when none does, because a trainer's state holds a task and a binary reward per sample and nothing else, and an empty reply there is the shape of the data, not a finding; the report's `text_gates` block and a `hygiene_warnings` line say the gates were skipped. `"require"` runs them regardless (a row with no reply is `incomplete_junk`, the behavior before 0.121), `"skip"` never runs them. The label gate, the unanimous trim, the difficulty band and the ranking run in every mode: they read the reward alone (Lambert 2025, chapter Reasoning; Yu et al. 2025, arXiv:2503.14476). `audit` is an `audit_grades` report on these rows' verifier; when it found the verifier rejecting right answers more than `FN_WARN` of the time, `hygiene_warnings` says to fix the verifier before training on the selection (#255). `prior` is the previous round's graded rollouts: tasks the round-N policy already solves (pass rate above `hi` on `prior`) or never solves (below `lo`) are dropped before anything else, so round N+1 trains on what that policy gets right 20-80% of the time rather than on the file round 1 started from (`next_round`; Lambert 2025, chapter Reasoning). The report's `prior` block counts kept, dropped\_solved, dropped\_unsolved and unknown. `truncated` says what happens to a rollout cut at the token cap (DAPO's overlong handling, Yu et al. 2025, arXiv:2503.14476; overlong filtering, Lambert 2025, chapter Reasoning): `"drop"` removes it (the default; `drop_truncated=False` is the old spelling of `"keep"`), `"keep"` leaves it in with `overlong=True` and its own reward, riding with its ask rather than deciding it (the ask is unanimous, in band and ranked exactly as under `"drop"`, so `"keep"` never returns fewer rows than `"drop"`; a cut rollout's reward is not the contrast an ask is kept for), and `"penalize"` keeps it as a failure that does count: reward 0, the judged score under `reward_before_penalty`, so running past the cap is a negative signal instead of a rollout that vanished. A conduct-grade advisory 0.5 for truncation is unusable under `"keep"` and a 0 under `"penalize"`. After the row gates, duplicate and truncated rollouts (`dedupe`, `truncated`), the unanimous trim, and (`enforce_band`) the difficulty band, remaining asks are taken round-robin across observed fault kinds, so the dataset keeps a grounded spread of no-fault, miss, timeout, and already-done situations rather than one over-represented failure. Within a fault kind, `order="spread"` (the default) takes asks round-robin across their pass rates, so a 25% ask, a 50% ask and a 75% ask are picked in turn with no preference for the middle (Lambert 2025, chapter Reasoning, filters to the 20-80% band and stops there; nothing in it says 50% is better than 30%). `order="middle"` is the older ranking by closeness to a 50% pass rate. The last group may overshoot `target`; an RL update wants the complete group or none of it. `enforce_band=False` keeps out-of-band asks and only ranks them last. The report's `hack_scan` block is the reward-hack scan over the selection (`hack_scan`: what separates reward within an ask, against a permutation floor; `endorsed` names what it should be), `correlations` the older pooled scan. Reward tracking a shortcut is a judge problem, flagged in `hygiene_warnings`, not pruned. Selected rows are stamped in place with the `calibration` measured on the rows as they arrived, before dedupe and the trims: the pass rate over the k repeats the grader saw is the task's difficulty, and re-measuring it on the survivors would report the post-dedup k under that name. `publish_gate` keeps the carried stamp. The k-way reliability numbers do not survive the prune, and `hygiene_warnings` says so when they were available before it. ### select\_for\_sft ```python theme={"theme":"vitesse-dark"} select_for_sft( rows: Sequence[dict], target: int = 1000, select: str = 'top_per_prompt', k: int | None = None, min_reward: float = 1.0, seed: int = 0, ) -> tuple[list[dict], dict[str, Any]] ``` Defined in [`whileai/simulations/score/optimize.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/optimize.py). Diverse correct demonstrations, at most `target` rows. Imitation clones what it sees, so only rows whose reward reaches `min_reward` (default 1.0: judge-approved) and that are not junk qualify; unanimity is not a problem here. A grader with partial credit ranks by its score: lower `min_reward` to admit it. `select` is the rejection-sampling rule (Lambert 2025, chapter Rejection Sampling, "Scoring Completions"): `"top_per_prompt"` keeps each prompt's highest-reward completion and then round-robins across behavior signatures (tool sequence, argument provenance, outcome shape) so every distinct way of being right appears before any repeats; `"top_k_overall"` keeps the `k` highest-reward completions across all prompts, several per prompt allowed; the two `random_*` rules are the control that chapter asks for (same counts, seeded random picks) so a claimed gain from selection can be checked against chance. `k` defaults to `target`. ### trim\_out\_of\_band ```python theme={"theme":"vitesse-dark"} trim_out_of_band( rows: Sequence[dict], lo: float = 0.2, hi: float = 0.8, min_k: int = 2, ) -> tuple[list[dict], dict[str, Any]] ``` Defined in [`whileai/simulations/score/optimize.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/optimize.py). Difficulty band filter. Nothing to do with topic or relevance. "Out of band" here means outside the *difficulty* band `[lo, hi]` (default `DEFAULT_BAND`, 0.2 to 0.8): an ask is dropped when its pass rate over k >= `min_k` rollouts is too high (the policy almost always solves it) or too low (it almost never does), because either way it carries little gradient per rollout. It does not read the prompt, the topic, or the tools; a perfectly on-topic ask is dropped for being too easy, and an off-topic one the policy passes half the time is kept. Junk rows are a separate filter (`is_incomplete_junk`, applied by `optimize`), and nothing here filters by topic at all. Unanimous asks are `trim_unanimous_groups`'s job and are left alone here; singles always stay. Rows are per-rollout rows with a binary `reward`, or one row per task carrying `pass_rate` and `n` (a trainer's state, or the `calibration` stamp); `min_k` reads `n`, and a rate row that does not say its `n` is taken at its word. The report's `from_rates` counts the tasks measured from a carried rate rather than rollouts. ### trim\_unanimous\_groups ```python theme={"theme":"vitesse-dark"} trim_unanimous_groups( rows: Sequence[dict], min_k: int = 2, ) -> tuple[list[dict], dict[str, Any]] ``` Defined in [`whileai/simulations/score/optimize.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/optimize.py). Drop asks whose k >= `min_k` rollouts all landed 0 or all landed 1. The basic optimizer from the working decision: trim zeros and ones from tasks, then rerun the simulator and check the variance. Groups smaller than `min_k` (unique-situation runs) always stay; trimming them would gut an explore dataset, and they carry no group gradient either way. ## pairwise Pairwise judging: which of two replies to the same request is better, asked both ways round. ### judge\_pairs ```python theme={"theme":"vitesse-dark"} judge_pairs( pairs: Sequence[dict], judge: Callable[[dict, dict], Any] | None = None, spec: str | None = None, swap: bool = True, concurrency: int = 8, api_key: str | None = None, examples: int = 10, position_flip_flag: float = 0.2, prefers_rejected_flag: float = 0.2, ) -> tuple[list[dict], dict[str, Any]] ``` Defined in [`whileai/simulations/score/pairwise.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/pairwise.py). Ask a judge which side of each pair is better, both ways round. `judge(a_row, b_row)` returns `\{"winner": "A" | "B" | "tie", "reason"\}` (or that JSON as a string); without one the hosted model judge from `pairwise_judge(spec)` is used. With `swap=True` each pair is judged twice with A and B exchanged; a pair the judge decides differently in the two orders is recorded as a tie with `position_consistent=False`. Writes on each pair (in place, and returned): `pairwise` with `winner` (`"chosen"` | `"rejected"` | `"tie"` | `None` when the judge failed), `position_consistent`, `reasons`, `judge`; and `tie` (bool). Report: `position_flip_rate` (position bias: the judge's answer changed with the order), `tie_rate`, `agrees_with_scores` (the pairwise winner is the pointwise `chosen`), `prefers_rejected` (the two disagree outright, the rows a person should read), `failed`. A `position_flip_rate` at or over `position_flip_flag` (`POSITION_FLIP_FLAG`, 0.2: Zheng et al. arXiv:2306.05685 measured 35% of GPT-4 verdicts flipping with the order) and a prefers-rejected share at or over `prefers_rejected_flag` each add a warning. ### pairwise\_judge ```python theme={"theme":"vitesse-dark"} pairwise_judge( spec: str | None = None, api_key: str | None = None, prompt: str | None = None, policy: str = '', tools: Sequence | None = None, timeout: float = 120, max_tokens: int = 120, request_chars: int = 4000, ) -> Callable[[dict, dict], Verdict] ``` Defined in [`whileai/simulations/score/pairwise.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/pairwise.py). A model judge for `judge_pairs`: `judge(a_row, b_row) -> \{"winner": "A" | "B" | "tie" | None, "reason": str\}`. `spec` is a backend spec (default the hosted judge); `prompt` replaces the pairwise system prompt. The judge's name is `@` so a prompt edit is a new judge. `max_tokens` is the judge's reply budget and `request_chars` how much of the request it is shown. ## passat pass\@1, pass^k and pass\@k from the same graded groups. ### PassAt ```python theme={"theme":"vitesse-dark"} class PassAt( k: int, pass_at_1: float | None, pass_pow_k: float | None, pass_at_k: float | None, n_groups: int, n_rows: int, n_groups_at_k: int = 0, n_groups_imputed: int = 0, n_partial: int = 0, per_task: dict[str, float] = , note: str = '', config: dict[str, Any] = , ci95: tuple[float, float] | None = None, pass_pow_k_ci95: tuple[float, float] | None = None, pass_at_k_ci95: tuple[float, float] | None = None, ) -> None ``` Defined in [`whileai/simulations/score/passat.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/passat.py). pass\@1 / pass^k / pass\@k over graded groups. See module docstring. Every field, and the name it prints as in `str(...)`. The printed line and the attribute are not spelled the same: pass^k is `pass_pow_k` (not `pass_hat_k`), and the print says both once so the attribute is readable off it. | attribute | prints as | what it is | | ------------------ | ---------------- | ------------------------------------------------------------ | | `k` | `k=4` | draw size the k-way numbers used | | `pass_at_1` | `pass@1` | mean per-task pass rate, the headline | | `pass_pow_k` | `pass^4` | chance all k repeats pass (reliability) | | `pass_at_k` | `pass@4` | chance at least one of k passes | | `headroom` | `headroom` | property: pass\@k minus pass\@1 | | `ci95` | `[lo..hi]` | interval on pass\@1; `None` under 3 groups, reason in `note` | | `pass_pow_k_ci95` | `[lo..hi]` | same for pass^k; `None` under 3 groups | | `pass_at_k_ci95` | `[lo..hi]` | same for pass\@k; `None` under 3 groups | | `n_groups` | `N groups` | tasks pass\@1 averaged over | | `n_rows` | not printed | graded rows behind those tasks | | `n_groups_at_k` | not printed | tasks the k-way numbers used | | `n_groups_imputed` | not printed | short unanimous tasks counted in | | `n_partial` | in the note | rows whose reward is not 0 or 1, left out | | `per_task` | not printed | `{task key: pass rate}`, a dict | | `note` | tail of the line | why a number is missing, and the fix | | `config` | token-cap share | how the rows were made (`run_config`) | `to_dict()` uses these same keys, with `headroom` added and the intervals as lists. ### pass\_at ```python theme={"theme":"vitesse-dark"} pass_at( rows: Sequence[dict] | Any, k: int | None = None, min_k: int = 4, unanimous_short: bool = False, ) -> PassAt ``` Defined in [`whileai/simulations/score/passat.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/passat.py). Compute pass\@1, pass^k and pass\@k from graded rows, grouped by task. Reach for it after grading a `mode="rl"` run to read the three numbers a task family gives: pass\@1 is the mean per-task pass rate (the headline), pass^k the chance all k repeats pass (reliability), pass\@k the chance at least one of k passes; headroom is pass\@k minus pass\@1. It returns a `PassAt` with those three, their task-bootstrap `ci95` intervals, `n_groups`, `n_rows`, `per_task` (a dict keyed by task), a `note` when a number is missing and why, and `config` (how the rows were made); `str(result)` prints the line and `to_dict()` gives the keys. A task is a situation, not a string. Rows group under `task_key`: the engine's `scenario_id` when the row has one, else `task_id`, else the prompt text. In `mode="rl"` the repeats of one opener share a `scenario_id`, and so do the textured phrasings of one situation, so those phrasings pool into one task on purpose: the question is whether the agent handles the situation, not one wording of it. `compare_runs`, `delta_report`, `eval_variance`, `curriculum` and `group_signal` count tasks with the same key, so `pass_at(rows).n_groups` and `delta_report(...)["n_paired_tasks"]` agree on the same rows. Only binary `reward` (or `qwen_reward`) rows count, the same rule `group_signal` uses; a row whose reward is between 0 and 1 is counted in `n_partial` and named in `note`, and an unjudged row is skipped. The intervals resample tasks, never rows (Miller 2024, arXiv:2411.00640), so they need at least `MIN_CI_TASKS` (3) tasks. Under that, `ci95` is `None` and the `note` says why and what to change. Ten rows that all carry one `task_id` are one task, not ten, and get no interval; when they are ten separate items, give each its own `task_id`. * `rows`: graded rows, or the `SimulationData` holding them. * `k`: the draw size for the k-way numbers. It defaults to the smallest group of two or more repeats, so every such group contributes; groups with fewer than `k` graded repeats are left out of pass^k and pass\@k and counted in `n_groups_at_k`. pass\@1 always averages every group. * `min_k`: 4 (`ROLLOUTS_PER_TASK`), the smallest k tau-bench and tau2-bench report a pass^k on (arXiv:2406.12045 and arXiv:2506.07982). Below it the k-way numbers are `None` with a `note` instead of a number too noisy to act on. * `unanimous_short`: `True` counts a unanimous group shorter than `k` as if it stayed unanimous (pass^k and pass\@k equal to its pass rate, 1 or 0). That is the assumption a successive-allocation run stopped on, and leaving those groups out would score only the tasks that split and inflate the headroom. Mixed short groups still stay out. `.config` says how the rows were produced (`run_config`): task count, k, temperature, max\_tokens, policy and judge versions, prompt hash, with a `mixed` list naming any the rows disagree on. ```python theme={"theme":"vitesse-dark"} >>> import whileai.simulations as wai >>> rows = [{"task_id": t, "reward": r} for t in "abcd" for r in (1, 1, 0, 1)] >>> wai.pass_at(rows).pass_at_1 0.75 ``` ## preflight Inspect the agent before spending simulation budget, and report after. ### classify\_failure ```python theme={"theme":"vitesse-dark"} classify_failure(row: dict) -> str | None ``` Defined in [`whileai/simulations/score/preflight.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/preflight.py). Fixed-vocabulary class for a failing row, from its reason and shape. Returns None for passing or unlabeled rows and for failures the heuristics cannot place (leave those for a person, do not guess). ### coverage\_gap ```python theme={"theme":"vitesse-dark"} coverage_gap( asks: Any, tools: Sequence[dict], system_prompt: str = '', rows: Sequence[dict] | None = None, rule_cap: int | None = None, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/preflight.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/preflight.py). List the parts of an agent's policy that the asks you already send never reach. Reach for it before writing situations, with the test suite you already have: it says which tools and which policy rules no ask exercises, in the engine's own vocabulary. It returns a dict: `untested_rules` and `untested_tools` (the lists worth reading), `rules` (the policy clauses found), `axes` (each axis with the count per value), `stances`, `pressure_asks`, `single_shot`, `per_ask` (where each ask landed), `notes`, `summary` and `n_asks`. `format_coverage_gap(report)` prints it. * `asks`: what a suite asks the agent: a list of prompt strings, a list of rows carrying `prompt`, or a path to a `.py` or `.jsonl` file holding either. * `tools` and `system_prompt`: the agent's tool schemas and policy. The axes come from `build_dimensions`, the same grid `simulate` covers: which tool, which policy rule, what stance the person takes, what the world looks like, what condition the tool is in, what happened before. * `rows`: graded rollouts from a run. With them the report also checks the world side: rules whose rows all ended in the same tool fault are rules the asks reach but the fixtures never let happen (`rules_the_world_never_triggers`, with `rows_per_rule` and `rules_with_no_rows`). Each ask is placed on the axes it touches with text heuristics, not a model: the tools its words name or imply, the rule clauses it shares words with, and the stance its words show. Two axes (`world_state`, `tool_condition`) cannot be read from an ask at all: a prompt never says the order is missing or the tool timed out, so a hand-written suite leaves them at one point and `notes` says so. With `rows` (graded rollouts from a run) the report also checks the world side: rules whose rows all ended in the same tool fault are rules the asks reach but the fixtures never let happen. The rule axis is every clause of the policy (`rule_cap=None`, the default `RULE_AXIS_CAP_REPORT`): a report over an existing suite has no grid to bound. A number keeps the first that many clauses in document order; `n_rules_total` and `rules_truncated` say what was left off and `notes` carries the count (#391). ```python theme={"theme":"vitesse-dark"} gap = wai.coverage_gap(["Where is order 4473?", "Cancel order 9911."], tools=TOOLS, system_prompt=POLICY) print(gap["untested_rules"], gap["untested_tools"]) ``` ### dataset\_report ```python theme={"theme":"vitesse-dark"} dataset_report( rows: Sequence[dict], tools: Sequence[dict] | None = None, system_prompt: str = '', hard_share_floor: float = 0.3, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/preflight.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/preflight.py). One report a developer reads after simulate/grade: size, signal, mix. `hard_share_floor` (`HARD_SHARE_FLOOR`, 0.3) is the share of hard- tier rows under which the set is called easy. ### format\_coverage\_gap ```python theme={"theme":"vitesse-dark"} format_coverage_gap(report: dict[str, Any]) -> str ``` Defined in [`whileai/simulations/score/preflight.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/preflight.py). The gap report as the block a person actually reads. ### preflight ```python theme={"theme":"vitesse-dark"} preflight( tools: Sequence[dict], system_prompt: str = '', rule_cap: int | None = None, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/preflight.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/preflight.py). Spec-quality report for an agent. Report only; nothing is changed. `warnings` is the list a developer should read before generating thousands of rows; `cells` is the covering-grid size the same way `recommend` counts it. `rules` is every clause of the policy (`rule_cap=None`, the default `RULE_AXIS_CAP_REPORT`); a number keeps the first that many in document order, `n_rules_total` says how many the policy has, `rules_truncated` whether any were left off, and a `warnings` line names the count (#391). The generation grid keeps its own cap (`RULE_AXIS_CAP_GRID`); `cells` is counted on the same axis `rules` shows. ## privileged Did the agent say what only the teacher was told? ### format\_leak\_report ```python theme={"theme":"vitesse-dark"} format_leak_report(report: dict[str, Any]) -> str ``` Defined in [`whileai/simulations/score/privileged.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/privileged.py). One line per fact, the summary first. ### leak\_report ```python theme={"theme":"vitesse-dark"} leak_report(rows: Any, min_len: int = 12) -> LeakReport ``` Defined in [`whileai/simulations/score/privileged.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/privileged.py). Which rows quote their own `privileged` block in the agent's text. Takes the `SimulationData` itself, `data.trajectories`, or any list of rows. Given the data object it reads the trajectories, which still carry the block; `data.rows()` is the scrubbed export and checks nothing (the report says so). Checks every row that carries `privileged` (`reference`, `principle`, and every string in `hidden_state` at least `min_len` characters long) against the final reply and every assistant turn. Returns `n_rows`, `n_checked`, `n_leaked`, `rate` (over checked rows), `checked` (False when no row carried the block, so the result is vacuous), `leaked` (up to 20 rows: `scenario_id`, `rollout_index`, `field`, `needle`) and `summary`. Does not mutate `rows`. Pass `data.trajectories`, not `data.rows()`: the export scrubs `privileged` at any depth, so exported rows carry nothing to check and the report is vacuous. When it can tell the rows came through the export, `summary` says so and names the accessor to use instead. ## publish\_gate The gate a dataset passes before it leaves for the platform. ### PublishGateError Defined in [`whileai/simulations/score/publish_gate.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/publish_gate.py). The dataset must not be published as it stands. The message says why. ### calibrate ```python theme={"theme":"vitesse-dark"} calibrate( rows: Sequence[dict], policy: PolicyRef | dict | str | None = None, model: str | None = None, ref: str | Sequence[dict] | None = None, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/publish_gate.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/publish_gate.py). Stamp `calibration` on every graded row, in place. The per-task pass rate is over the binary rewards grouped by prompt, the same grouping `group_signal` and `pass_at` use. Rows without a 0/1 reward are left alone and counted. Returns a report with the number of tasks and rows stamped plus the `pass_at` summary. `ref` (a key holding the reference model's summed logprob, or rows scored under it) fills `mean_kl` per task from the captured logprobs; see `mean_kl`. A row whose carried stamp counts more repeats than these rows hold keeps it (`n_carried` in the report): `optimize(mode="rl")` drops duplicate trajectories, and recomputing here would report the post-dedup k as the policy's pass rate over k repeats. The producing policy and `mean_kl` are still filled in from this call. ### publish\_gate ```python theme={"theme":"vitesse-dark"} publish_gate( rows: Sequence[dict], mode: str | None = None, band: tuple[float, float] = (0.2, 0.8), policy: PolicyRef | dict | str | None = None, model: str | None = None, strict: bool = True, endorsed: Sequence[str] = (), strict_hacks: bool = False, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/publish_gate.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/publish_gate.py). Check, calibrate, and report. Raises `PublishGateError` when `strict` and the rows are RL-shaped but ungraded or carry no mixed group, or when `strict_hacks` and `hack_scan` (with `endorsed` naming what the reward should track) finds the reward best explained by something else. Never mutates anything except the `calibration` stamp. `judge_trust` in the report is the summary `grade` stamped on the rows when they carried human labels, else `None`. ## quality Second-pass conversation quality ranker. Scores rows; does not rewrite them. ### rank\_rows ```python theme={"theme":"vitesse-dark"} rank_rows(rows: Sequence[dict]) -> list[dict] ``` Defined in [`whileai/simulations/score/quality.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/quality.py). Score each row in place. Returns the same list when given a list. ### score\_row ```python theme={"theme":"vitesse-dark"} score_row(row: dict) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/quality.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/quality.py). Score one row. Returns quality, quality\_reason, quality\_scores. No mutate. ## reference Score rollouts under a reference model, so `mean_kl` has its other side. ### reference\_logprobs ```python theme={"theme":"vitesse-dark"} reference_logprobs( source, ref: str, system_prompt: str | None = None, tools: Sequence[dict] | None = None, api_key: str | None = None, concurrency: int = 4, timeout: float = 600.0, chat_template_kwargs: Mapping[str, Any] | None = None, transport: Callable[[dict], dict] | None = None, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/reference.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/reference.py). Stamp `ref_logprob` on every row: the reference's summed logprob over the tokens the policy generated. Rows are modified in place; the report says what was scored. `ref` is a backend spec, `vllm:@`; on the platform's serving endpoint `` is the base by its own name (`Qwen/Qwen3-4B`: the reference of an SFT/GRPO/DPO run), a hosted model's name, or `run:` for a finished run's adapter, with `WHILEAI_API_KEY` as the key. `source` is a `SimulationData` (system prompt and tools come from its profile), a row list, or a JSONL path; pass `system_prompt=`/`tools=` for the last two so the reference sees the prompt the policy saw. `chat_template_kwargs` must match what the policy sampled with (`\{"enable_thinking": False\}` for Qwen3). Report: `n_rows`, `n_skipped` (no assistant turn or a failed call, with `errors`), `n_tokens`, `model`, and `token_count_gap` (mean |ref\_n\_tokens - n\_tokens| over rows that carry `n_tokens`): near zero when the reference shares the policy's tokenizer, which is when `mean_kl` is a KL and not a length artifact. ## rubric Rubrics: prompt-specific criteria as an object, a judge that scores them one by one, and a writer that drafts them. ### Criterion ```python theme={"theme":"vitesse-dark"} class Criterion( title: str, description: str = '', weight: float = 1.0, kind: str = 'principle', ) -> None ``` Defined in [`whileai/simulations/score/rubric.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/rubric.py). One rubric item. `weight` is a positive magnitude; a `pitfall` subtracts it when the reply exhibits the mistake, a `principle` adds it when met, and a `hard` rule missed fails the whole reply. ### Rubric ```python theme={"theme":"vitesse-dark"} class Rubric( criteria: tuple[Criterion, ...], source: str = 'hand', domain: str = '', notes: str = '', _cache: dict = , ) -> None ``` Defined in [`whileai/simulations/score/rubric.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/rubric.py). #### Rubric.checklist ```python theme={"theme":"vitesse-dark"} checklist(self) -> str ``` The rubric as the judge reads it. #### Rubric.score ```python theme={"theme":"vitesse-dark"} score(self, results: Mapping[str, Any]) -> dict[str, Any] ``` Reward from one verdict per criterion (keyed by title or slug; a truthy value means the reply meets a hard rule or principle, or exhibits a pitfall). A missed hard rule is a 0. Otherwise the reward is the met principle weight minus the exhibited pitfall weight, over the total principle weight, clamped to \[0, 1]; with no principle the reward is 1 minus the pitfall share. Criteria the judge did not answer count as not met (and not exhibited) and are listed. ### attach\_rubric ```python theme={"theme":"vitesse-dark"} attach_rubric( rows: Sequence[dict], rubric: Rubric | Mapping[str, Any] | Sequence[Mapping[str, Any]] | Callable[[dict], Any], overwrite: bool = True, ) -> list[dict] ``` Defined in [`whileai/simulations/score/rubric.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/rubric.py). Put a rubric on each row's `privileged` block (in place). `rubric` is a `Rubric`, its dict / list form, or `row -> Rubric | None` for a per-prompt rubric; `None` leaves that row alone. ### rubric\_judge ```python theme={"theme":"vitesse-dark"} rubric_judge( rubric: Rubric | None = None, spec: str | None = None, api_key: str | None = None, prompt: str | None = None, policy: str = '', tools: Sequence | None = None, timeout: float = 120, ) -> Callable[[dict], dict[str, Any]] ``` Defined in [`whileai/simulations/score/rubric.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/rubric.py). A judge for `run_judge` / `data.grade(judge=)` that scores the rubric item by item. `rubric` applies to every row; without one the row's own `privileged.rubric` is used and a row with none stays ungraded. The result carries `reward` (`Rubric.score`), `reason`, `markers` (`rubric:` = 1.0 met / 0.0 not, and for a pitfall 1.0 clean / 0.0 exhibited), `criteria` (the raw verdicts), `rubric_version` and the score breakdown. The judge's name folds the rubric version in when one is fixed. Two things about the verdict worth knowing before it is trusted. A rubric of plain principles scores the *mean* of its criteria, so three principles return 0, 1/3, 2/3 or 1, and `judge_agreement` / `judge_trust` count exact 0/1 rewards only: every partially met row is skipped, and the agreement number is read off the rows the judge was sure about (#345). Give each `Criterion` `kind="hard"` for a 0/1 verdict (`Rubric.score`), or accept that `judge_trust` reports the skipped share and pulls `ok` when it passes `MAX_SKIPPED_SHARE`. And whether a tool was *called* is handed to the judge as a fact, not left for it to infer: the payload carries `tools_called` (steps that returned a result) and `tools_not_called`, and the system prompt says a reply that announces a call it never made has not made it (#346: without the list, a 4B judge passed 18 of 18 announced-but-never-made escalations). A criterion that must be exact belongs in a `grader=` that reads `steps` itself. The hosted judge scales to zero, so the first row through warms it once (`warm_judge`, a 600s budget) while the rest of the fan-out waits. Without that, `run_judge`'s eight concurrent calls all raced a container that was still loading its weights and every row came back `invalid_result` with a `TimeoutError`. Warm-up failure is not fatal: the rows are judged anyway and report the real error. ### rubric\_of ```python theme={"theme":"vitesse-dark"} rubric_of(row: Mapping[str, Any]) -> Rubric | None ``` Defined in [`whileai/simulations/score/rubric.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/rubric.py). ### write\_rubrics ```python theme={"theme":"vitesse-dark"} write_rubrics( rows: Sequence[dict], spec: str | None = None, domain: str = '', n_criteria: tuple[int, int] = (3, 8), reference_key: str = 'reference', overwrite: bool = False, concurrency: int = 8, api_key: str | None = None, timeout: float = 120, writer: Callable[[str], Any] | None = None, max_hard: int | None = None, ) -> tuple[list[dict], dict[str, Any]] ``` Defined in [`whileai/simulations/score/rubric.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/rubric.py). Draft one rubric per distinct prompt with a model and attach it to every row of that prompt (`privileged.rubric`, `source="model"`). The writer sees the request, the row's reference answer when there is one (`privileged.reference` or `row[reference_key]`), and the `domain` guidance you give it (the general rubric Lambert 2025 seeds from). Rows that already carry a rubric are skipped unless `overwrite`. `writer(user_message) -> str` replaces the model call for tests and for a writer of your own. Report: prompts seen, rubrics written, failures, mean criteria per rubric, the rubric versions. `max_hard` caps the hard rules a written rubric may carry: the heaviest `max_hard` stay hard and the rest become principles with their weight (`demoted_hard` in the report). A model writer marks most of what it wants as Essential, and every Essential item a reply misses is a 0, so an uncapped rubric fails rows a binary judge passes (measured live: 22 of 32 rows). `None` keeps what the writer wrote. ## spec Model spec as a versioned object (Lambert 2025, chapter Model Character and Products). ### Spec ```python theme={"theme":"vitesse-dark"} class Spec( id: str, traits: tuple[Trait, ...] = (), source: dict = , version: str = '', ) -> None ``` Defined in [`whileai/simulations/score/spec.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/spec.py). A versioned model spec. `version` is derived from the content when left empty, so it is stable across processes and changes on any edit. #### Spec.behaviors ```python theme={"theme":"vitesse-dark"} behaviors(self) -> list[str] ``` Trait ids, in order. Hand to `delta_report(must_not_regress=...)`. ### Trait ```python theme={"theme":"vitesse-dark"} class Trait( id: str, name: str = '', principle: str = '', authority: str = 'should', examples: tuple[Any, ...] = (), ) -> None ``` Defined in [`whileai/simulations/score/spec.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/spec.py). One named expectation. `authority` is the strength (`must` / `should` / `may`, following the model-spec convention). ### load\_spec ```python theme={"theme":"vitesse-dark"} load_spec(obj: Any, spec_id: str | None = None) -> Spec ``` Defined in [`whileai/simulations/score/spec.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/spec.py). Build a `Spec` from a constitution dict, a list of traits, or a path to a JSON file with either shape. A trait may be a full dict or a bare principle string. ### spec\_version ```python theme={"theme":"vitesse-dark"} spec_version(spec: Spec | dict | Any) -> str ``` Defined in [`whileai/simulations/score/spec.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/spec.py). The content version of a Spec (or anything `load_spec` accepts). ### stamp\_spec ```python theme={"theme":"vitesse-dark"} stamp_spec(rows: Sequence[dict], spec: Spec) -> list[dict] ``` Defined in [`whileai/simulations/score/spec.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/spec.py). Return copies of `rows` tagged with the spec they were produced or graded against: `spec_id` and `spec_version`. Provenance for the ch. 17 retention question — did adherence hold from one spec version, or model version, to the next. ## stage Stage lineage: which post-training stage consumed each row (Lambert 2025, chapter Training Overview). ### format\_stages ```python theme={"theme":"vitesse-dark"} format_stages(report: dict[str, Any]) -> str ``` Defined in [`whileai/simulations/score/stage.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/stage.py). ### stage\_of ```python theme={"theme":"vitesse-dark"} stage_of(row: dict) -> str | None ``` Defined in [`whileai/simulations/score/stage.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/stage.py). The stamped stage, or None. An eval-sourced row with no stamp reads as `eval` (its `lineage.source`), so a held-out set is never mistaken for training data just because no one stamped it. ### stage\_report ```python theme={"theme":"vitesse-dark"} stage_report(rows: Sequence[dict]) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/stage.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/stage.py). Rows per stage, tasks per stage, and the cross-stage leaks: any task used both in `eval` and in a training stage (sft/rm/rl/mid). That leak means the number you report was optimized against. ### stamp\_stage ```python theme={"theme":"vitesse-dark"} stamp_stage(rows: Sequence[dict], stage: str) -> list[dict] ``` Defined in [`whileai/simulations/score/stage.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/stage.py). Return copies of `rows` with `row["stage"] = stage`. `stage` must be one of `STAGES`; nothing else is touched. ## stats Confidence intervals, paired run comparison, and decontamination. ### compare\_runs ```python theme={"theme":"vitesse-dark"} compare_runs( a: Sequence[dict], b: Sequence[dict], metric: str = 'pass_at_1', n_boot: int = 2000, seed: int = 0, min_paired: int = 5, level: float = 0.95, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/stats.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/stats.py). Test whether run `b` differs from run `a` on one metric, paired by task. Reach for it for a quick A/B on a single number; `delta_report` is the full report with markers, the noise floor and the comparability checks. It returns a dict: `delta` (b minus a), `ci95` (the interval, with `level` beside it), `p_value`, `verdict`, `n_paired`, `n_only_a`, `n_only_b`, `paired_share`, `mean_a`, `mean_b`, and a `note`. Tasks the two runs share are compared as paired differences (b minus a, per task, keyed the way `pass_at` groups); the interval is a `level` bootstrap over those pairs and the p-value is a sign-flip permutation test. `verdict` is one of `"b_better"`, `"a_better"`, `"no_difference_detected"`: the last means the interval covers zero, not that the runs are equal. Tasks on one side only are dropped from a paired comparison, and `note` says how many, since a verdict over a quarter of the tasks is not a verdict over the eval. `paired_share` is the shared fraction of every task either run saw. * `metric`: `"pass_at_1"` (the default, binary reward) or `"marker:name"` for a marker. * `min_paired` (5): with fewer shared tasks the comparison falls back to unpaired task means and says so. * `level` (0.95): the interval's coverage (`ci95` at the default). `n_boot` (2000) and `seed` (0) fix the bootstrap. ```python theme={"theme":"vitesse-dark"} >>> a = [{"task_id": t, "reward": 0} for t in "abcdef"] >>> b = [{"task_id": t, "reward": 1} for t in "abcdef"] >>> wai.compare_runs(a, b)["verdict"] 'b_better' ``` ### decontaminate ```python theme={"theme":"vitesse-dark"} decontaminate( rows: Sequence[dict], against: Sequence[Any] | Any, n: int = 8, fields: Sequence[str] = ('prompt',), overlap: float = 0.8, embedder: Callable[[list[str]], Sequence[Sequence[float]]] | None = None, similarity: float = 0.85, ) -> tuple[list[dict], dict[str, Any]] ``` Defined in [`whileai/simulations/score/stats.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/stats.py). Drop training rows whose prompt overlaps an evaluation set. Reach for it before any train-versus-holdout comparison: a held-out task that also sits in the training data measures memory, not the change (Lambert 2025, chapter Evaluation). It returns `(clean_rows, report)`: the rows that survived (a list that also carries the system prompt and tools the input carried, so `select(clean_rows).export()` writes them), and a report with the count under each rule (`n_contaminated` in total), hits per field, the eval text count, and the first offenders with their coverage (or `similarity` for semantic hits). `rules_skipped` names each rule that could not run on these inputs and why (empty when every rule ran), and `notes` says it in a sentence: a zero under a rule that never ran is not a clearance. * `rows`: the training rows. * `against`: one or more evaluation sources: row lists, JSONL paths, or platform dataset ids (`ds_...`). Evaluation prompts, answers and references are the texts compared (not the eval set's own replies). * `fields` (`("prompt",)`): which row texts are checked; prompts only is what Lambert 2025, chapter Evaluation, checks. Add `"final_text"` to ask the stricter question of whether replies reproduce eval answers or references. * `n` (8) and `overlap` (0.8): the near-copy rule, the Llama 2 rule of 8-grams covering 80% of tokens. `overlap=0` restores any-n-gram. * `embedder` and `similarity` (0.85): a callable from a list of texts to one vector per text turns on the semantic rule at that cosine threshold; nothing here imports a model. Four rules, applied in this order, and a row flagged by one is not counted again by the next, so `n_contaminated` is the number of rows dropped: * `same_task` (`n_same_task`): the row's `scenario_id` or `task_id` is an evaluation row's. A task is a situation, not a string (`task_key`), so a rephrasing of an eval situation is the eval situation whatever the words say. It needs an id on both sides: when no evaluation row (or no training row) carries one, the rule does not run, `rules_skipped["same_task"]` says so, and only the text rules stand between the sets. Every eval set not written by `simulate()` (GSM8K, a Hub set, logged traces) is in that case, so read `n_same_task: 0` next to `rules_skipped`. * `exact` (`n_exact`): one of the row's `fields` is an evaluation text verbatim after normalization (case and whitespace). * near copy (`n_near`): one evaluation text covers at least `overlap` of the row's words with shared word `n`-grams. Texts shorter than `n` words match verbatim only. * `semantic` (`n_semantic`), only with `embedder`: the cosine similarity between the row's text and an evaluation prompt is at least `similarity`, and the two carry different task ids or none. It needs evaluation prompts to embed: when no evaluation row has a `prompt`, the rule does not run, `rules_skipped["semantic"]` says so, and a `UserWarning` is raised because you asked for it. One shared n-gram is the test Lambert 2025, chapter Evaluation, uses for free-form sets. Situations written from templates share whole sentences that say nothing about which question was asked, so any-n-gram flags every row of a template-written set; the coverage rule counts a row when one eval text accounts for most of it. Word overlap does not see a paraphrase. A holdout written by re-running the generator on the same briefs was 70% within 0.85 cosine of the training batch and 5 of 133 byte-identical; the 8-gram rule flagged 4 of 101 prompts and the semantic pass 16. With sentence-transformers: ```python theme={"theme":"vitesse-dark"} from sentence_transformers import SentenceTransformer model = SentenceTransformer("BAAI/bge-small-en-v1.5") clean, report = wai.decontaminate( train, against=[holdout], embedder=lambda texts: model.encode(texts, normalize_embeddings=True).tolist(), ) ``` A semantic flag means the two prompts read alike, not that they are the same task: "cancel one reservation" and "cancel three reservations" for different customers scored 0.932 with no shared answer. So where task identity is recorded the `same_task` rule decides and the semantic pass only looks across different tasks, and the report's `notes` say the flag is a question to check, not a verdict. The default stays lexical: `similarity` 0.85 was read off BGE (unrelated prompts score about 0.55 there) and does not transfer to every model, so the pass calibrates it for yours when it can. With eval rows that carry task ids, the 99th percentile of similarity over eval-prompt pairs with different task ids is how alike distinct tasks read to this embedder, and `notes` says it; a threshold below that number flags tasks that merely share a domain, and the note says so when `similarity` is. ```python theme={"theme":"vitesse-dark"} >>> train = [{"prompt": "Where is order 4473?"}, {"prompt": "Cancel order 9911."}] >>> clean, report = wai.decontaminate(train, against=[[{"prompt": "Cancel order 9911."}]]) >>> len(clean), report["n_contaminated"] (1, 1) ``` ### detectable\_effect ```python theme={"theme":"vitesse-dark"} detectable_effect( n_tasks: int, base: float = 0.6, k: int = 4, power: float = 0.8, alpha: float = 0.05, ) -> float | None ``` Defined in [`whileai/simulations/score/stats.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/stats.py). The smallest gain `n_tasks` paired tasks can prove at `power`: `holdout_size` solved for the effect (`FIXED_POINT_STEPS` fixed-point steps, since the after-side variance depends on it). `None` below `MIN_HOLDOUT_TASKS` tasks. ### eval\_variance ```python theme={"theme":"vitesse-dark"} eval_variance( *runs: Sequence[dict], metric: str = 'pass_at_1', by: str | None = None, ) -> EvalVarianceReport ``` Defined in [`whileai/simulations/score/stats.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/stats.py). How much an evaluation moves when the same model is evaluated again (Lambert 2025, chapter Evaluation). Pass each re-run's rows as its own argument, or one row list whose rows say which run they belong to: `lineage.eval_run` (what `simulate(runs=3)` stamps), else `lineage.scoring_run_id` (what `evaluate(run_id=)` stamps), or a top-level or lineage key named by `by`. Each run's `metric` is a mean over tasks; the report is those means, their mean, the sample standard deviation `run_std`, and `noise_band` = `noise_band(run_std, df=n_runs - 1)`: the two-sided t quantile at `noise_band_df` = `n_runs - 1` times sqrt(2) times `run_std`, because a before/after delta with one run per side is the difference of two re-run draws and `run_std` is an estimate from these very runs, not the eval's exact spread (Lambert 2025, chapter Evaluation). This is the band `compare(run_std=, run_std_runs=)` applies; with three runs the multiplier is 4.30, not 1.96 (the 1.96 band read a three-run estimate as exact and let about one pure-noise delta in five through, #616). A delta inside the band is what re-running the eval does on its own. `run_std_by_metric` reports the same floor for pass\@1 and every marker shared by all runs; hand that mapping to `delta_report(run_std=)` so each metric uses its own re-run variance. The scalar `run_std` remains the selected `metric`'s value for callers comparing only one metric. `stability` places `run_std` on Olmo 3's bands in points. Fewer than three runs is a difference, not a distribution; the report says so and `run_std` is `None` below two. ### holdout\_size ```python theme={"theme":"vitesse-dark"} holdout_size( effect: float, base: float = 0.6, k: int = 4, power: float = 0.8, alpha: float = 0.05, before: Sequence[dict] | None = None, after: Sequence[dict] | None = None, task_std: float | None = None, rows: Sequence[dict] | None = None, ceiling_pass_rate: float = 0.9, ) -> HoldoutSizeReport ``` Defined in [`whileai/simulations/score/stats.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/stats.py). How many paired tasks a holdout needs to prove a gain of `effect`. Models the test `delta_report` runs: each task's pass rate over `k` rollouts on each side, the delta as the mean of the paired differences, the interval from a bootstrap over tasks. The usual two-sided power calculation then gives `n = ((z_\{1-alpha/2\} + z_power) * sd / effect) ** 2` with `sd` the standard deviation of one task's paired difference (Lambert 2025, chapter Evaluation: the point of a better eval is statistical power when comparing training runs). Where `sd` comes from is the whole question, and there are three ways to answer it, best first: * `before` and `after`, the graded arms of a previous eval on the same tasks (the two row lists `delta_report(before, after)` takes): `sd` is measured as the sample sd of the per-task differences, which carries the covariance that pairing buys and whatever shape the gain had. No model. `sd_source` is `"rows"` and `n_paired` says how many tasks it was read off. * `task_std`, a number you measured (the per-task sibling of `delta_report`'s `run_std`): the same quantity read off a previous `delta_report`: `(hi - lo) * sqrt(n_paired_tasks) / 3.92` from `target_ci95` and `n_paired_tasks` (or any `metrics[...]["ci95"]` with its `n_paired`). Agent rubrics sat near 0.38 across five lanes (#288). `sd_source` is `"given"`. `eval_variance`'s `run_std` is a different number (how much a re-run moves the mean) and is not this. * Neither: the binomial model `sqrt((p(1-p) + q(1-q)) / k)` with `p = base` and `q = base + effect`, `sd_source` `"model"`. It assumes two things it cannot check: that the gain is spread evenly across tasks, and that the two arms are independent draws (`Var(A) + Var(B)`, no covariance term). When the gain is carried by a few tasks, most tasks are ties and the paired differences spread far wider than binomial-per-task predicts; a voice trait at 0 -> 0.127, k=4, carried by 19 of 150 tasks, measured sd 0.333 against the model's 0.168 and needed 54 tasks where the model said 14 (#292). So the model path also returns `n_tasks_concentrated`, the count if the gain were carried by the fewest tasks that can carry it (each going from `base` to 1\), and `notes` says which assumption is in play. On a holdout whose tasks differ in difficulty the independence assumption errs the other way: the model puts `p(1-p)` of variance on every task where pairing keeps each task's own `p_i(1-p_i)`, whose mean is `p(1-p) - Var(p_i)`, so it asks for `1 / (1 - Var(p_i) / (p(1-p)))` times the tasks pairing needs (1.19x at spread 0.2 around 0.5, 2.78x at 0.4). `before` alone reports the spread as `base_spread` and puts that ratio in `notes`. `before` on its own (`rows` is the same argument under its old name) reads `base` and `k` off the data. Returns `n_tasks` plus the inputs, `task_std`, `sd_source`, `half_width` (the 95% band on the delta at that `n`), `n_tasks_concentrated`, `base_spread`, `n_paired`, `saturated`, `notes` and `warnings`; every key is present on every path (`None`, `False` or `[]` where it does not apply). The default answer is unchanged; the honest paths are the two that measure. A saturated `base=` cannot size anything (`base` is the before arm's pass rate; there is no `baseline=`). Rows whose tasks all pass give `p = 1`, the binomial variance `p(1-p)` is 0, and both arms all passing give a measured paired sd of 0; the formula then returns the floor, `MIN_HOLDOUT_TASKS`, which is the model collapsing, not evidence that two tasks are enough (#392). When the measured base is at or above `ceiling_pass_rate` (`CEILING_PASS_RATE`, the share `delta_report` flags as `ceiling`) or the measured sd is 0 (the paired difference identical on every task, `DEGENERATE`), the rows are not used: `n_tasks` is the binomial model's answer at `BASE_PASS_RATE` and the rows' `k`, `sd_source` is `"model"`, `saturated` is `True`, and `warnings` names the ceiling and the fix: harder situations, so `base` sits inside the 20-80 difficulty band (Lambert 2025, chapter Reasoning; DAPO, arXiv 2503.14476, drops prompts at accuracy 0 and 1 because they carry no signal), then size again on those rows. The recipe that asked for this had 140 tasks at k=4 around 0.6: a band of about +-0.06, so a real 3-point gain reads `no_change_detected` every round. This says so before training. ### marker\_names ```python theme={"theme":"vitesse-dark"} marker_names(rows: Sequence[dict]) -> list[str] ``` Defined in [`whileai/simulations/score/stats.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/stats.py). ### marker\_summary ```python theme={"theme":"vitesse-dark"} marker_summary( rows: Sequence[dict], names: Sequence[str] | None = None, n_boot: int = 2000, seed: int = 0, ) -> dict[str, dict[str, Any]] ``` Defined in [`whileai/simulations/score/stats.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/stats.py). `metric_summary` for every marker on the rows (or `names`). Each marker's stats are keyed `mean`, `ci95` (not `ci`), `n_tasks`, `n_rows` (not `n`), `n_rows_at_1`, `n_rows_at_0`, `degenerate`, and `note` or `warning` when there is one. `ci95` is `None` below `MIN_CI_TASKS` tasks, and `note` then says how many tasks the marker has and how many the interval needs; a reader who sees only `None` cannot tell that from a bug. ### metric\_summary ```python theme={"theme":"vitesse-dark"} metric_summary( rows: Sequence[dict], metric: str = 'pass_at_1', n_boot: int = 2000, seed: int = 0, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/stats.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/stats.py). Mean over tasks with a task-bootstrap 95% interval. `degenerate` is set when every applicable row scored the same value: the metric has not been shown to be able to come out any other way, so `ci95` is `None` (the way `pass_at` returns `None` below three groups) and `warning` says so. A marker that is silently unfireable (a key-name mismatch) and one that is genuinely always true look identical otherwise, and either one passed to `must_not_regress` is a guard that cannot fail (#270). `n_rows_at_1` and `n_rows_at_0` put the row-level split next to the mean. ### task\_key ```python theme={"theme":"vitesse-dark"} task_key(row: dict) -> str ``` Defined in [`whileai/simulations/score/stats.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/stats.py). The one name every report groups a row's rollouts under. A task is a situation, not a string: `scenario_id` when the row has one (the engine's situation id, shared by the repeats of one opener and by the textured phrasings of one situation), else `task_id` (rows from elsewhere), else the prompt text. `pass_at`, `compare_runs`, `delta_report`, `eval_variance`, `curriculum`, `group_signal` and the exporters all count tasks with this key, so the same rows give the same task count everywhere (Miller 2024, arXiv:2411.00640: intervals and paired comparisons are over tasks, never rows). ## style Over-optimization signatures on replies: the things a reward pays for by accident. ### refusal\_report ```python theme={"theme":"vitesse-dark"} refusal_report( rows: Sequence[dict], phrases: Sequence[str] = ("i can't help with", 'i cannot help with', "i can't assist", 'i cannot assist', "i'm unable to", 'i am unable to', "i won't be able to", "i'm not able to", 'i am not able to', "i can't do that", 'i cannot do that'), examples: int = 5, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/style.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/style.py). Over-refusal on a benign set (Lambert 2025, chapter Over-optimization, "Over-Refusal"). Pass the rows whose asks the agent should have answered; the report is the share it refused anyway, with a Wilson 95% interval, the phrases that fired, and the first few refusals so a person can read them. Refusal rate on a mixed set means nothing, which is why this takes the benign rows rather than finding them. ### style\_markers ```python theme={"theme":"vitesse-dark"} style_markers( rows: Sequence[dict], phrases: Mapping[str, Sequence[str]] | None = None, ) -> list[dict] ``` Defined in [`whileai/simulations/score/style.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/style.py). Stamp the style markers on every row's `markers` (in place) and return the rows. `phrases` overrides or extends `STYLE_MARKERS`: `{"no_boilerplate": [...], "no_brand_voice": [...]}`. Existing markers with other names are kept. ### style\_report ```python theme={"theme":"vitesse-dark"} style_report( rows: Sequence[dict], phrases: Mapping[str, Sequence[str]] | None = None, threshold: float = 0.3, n_boot: int = 2000, seed: int = 0, ) -> StyleReport ``` Defined in [`whileai/simulations/score/style.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/style.py). How much of each signature the replies carry, and whether the reward pays for it. Does not mutate `rows`. Per marker: `clean` (share of rows without a hit, with a task-bootstrap 95% interval), `hits` (rows with a hit), `top_phrases` (the phrases that fired, most common first) and `reward_corr` (Pearson between "phrase present" and the binary reward over graded rows). A positive correlation at or above `threshold` is flagged: the judge is rewarding the tic, and a policy trained on these rewards will produce more of it (Gao et al. 2022, arXiv:2210.10760). `warnings` says so in one line per flag. `print` the report; it is a dict, so every key still reads. A marker that came out the same on every row is `degenerate`: it has no interval, the line says so next to the mean, and one `notes` entry names every such marker and the fix. A phrase list that matches nothing looks exactly like a behavior that never happened, and either one in `must_not_regress=` is a guard that cannot fail (#270). `warnings` stays what it was, the reward-pays-for-a-tic flags and nothing else. This report stamps the phrase signatures and nothing else, so it says what it did not stamp: `not_stamped` is `{call: [marker, ...]}` for the markers `trace_markers` and `mark_grounding` write, and the printed report ends with that line. A row clean on every marker here can still have faked a tool call or invented an argument, which #760 measured at 24.2% \[22.6%, 25.9%] of the rows this report passed. print(wai.style\_report(rows)) # style 10 rows, 10 graded # no\_boilerplate clean 1.000 hits 0 no interval: constant, see the warning below # no\_hedging clean 0.500 \[0.200..0.700] hits 5 corr +1.00 flagged "it depends" 5 # ... # not stamped here: 8 markers in other families. trace\_markers(rows) stamps ... ## trace Did the agent fake the work? Flags read from the trajectory, not the prose. ### trace\_flag\_report ```python theme={"theme":"vitesse-dark"} trace_flag_report( rows: Sequence[dict], kinds: Mapping[str, str] | None = None, threshold: float = 0.3, n_boot: int = 2000, seed: int = 0, examples: int = 3, ) -> dict[str, Any] ``` Defined in [`whileai/simulations/score/trace.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/trace.py). How often each flag fires, what a reviewer should read, and whether the reward pays for it. Does not mutate `rows`. Per flag: `n` (rows it fired on), `rate`, `examples` (evidence with the ask), and `reward_corr` (Pearson between "flag fired" and the binary reward over graded rows); a positive correlation at or over `threshold` is flagged, since a judge that pays for a faked turn trains a policy to fake turns. Per marker: the clean share with a task-bootstrap interval. `warnings` says so, one line per flag. ### trace\_flags ```python theme={"theme":"vitesse-dark"} trace_flags( row: Mapping[str, Any], kinds: Mapping[str, str] | None = None, ) -> dict[str, str] ``` Defined in [`whileai/simulations/score/trace.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/trace.py). The flags that fire on one rollout: `{flag: evidence}`. First match wins per flag, so the evidence points at the earliest cause. ### trace\_markers ```python theme={"theme":"vitesse-dark"} trace_markers( rows: Sequence[dict], kinds: Mapping[str, str] | None = None, evidence: bool = True, ) -> list[dict] ``` Defined in [`whileai/simulations/score/trace.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/score/trace.py). Stamp the trace markers on every row's `markers` (in place) and return the rows: 1.0 when the family is clean, 0.0 when a flag fired. With `evidence` the flags and their fragments land on the row as `trace_flags` for a reviewer.