99 public names. import whileai.simulations as wai, then wai.name.

Name	What it does
`Criterion`	One rubric item.
`PassAt`	pass@1 / pass^k / pass@k over graded groups.
`PublishGateError`	The dataset must not be published as it stands.
`Rubric`	no docstring
`ScoredData`	Scored trajectories: the one representation grade and eval share.
`Spec`	A versioned model spec.
`Trait`	One named expectation.
`annotator_agreement`	How the annotators on `gold_labels` agree with each other.
`argument_grounding`	1.0 when every string argument of every tool call is grounded in the conversation (a rollout with no calls is grounded), else 0.0.
`attach_labels`	Write hand labels onto rows (in place) and return `(rows, report)`.
`attach_rubric`	Put a rubric on each row’s `privileged` block (in place).
`audit_grades`	Estimate the verifier’s false-negative rate from a judged sample.
`behavior_signature`	Hash of tool sequence, argument provenance, statuses, and outcome shape.
`behavioral_markers`	Rate of each stock marker over `rows` (fraction of rollouts that trip it).
`build_preference_pairs`	Build same-task chosen/rejected pairs for preference training (DPO-style).
`calibrate`	Stamp `calibration` on every graded row, in place.
`classify_failure`	Fixed-vocabulary class for a failing row, from its reason and shape.
`compare_runs`	Test whether run `b` differs from run `a` on one metric, paired by task.
`conduct_grade`	Score agent conduct.
`coverage_gap`	List the parts of an agent’s policy that the asks you already send never reach.
`curriculum`	Split graded tasks into a training curriculum by measured difficulty.
`dataset_report`	One report a developer reads after simulate/grade: size, signal, mix.
`decontaminate`	Drop training rows whose prompt overlaps an evaluation set.
`dedupe_groups`	Drop repeat trajectories within one ask.
`delta_report`	Compare an `after` run to a `before` run on pass@1 and every shared marker, and say whether the change is real.
`detectable_effect`	The smallest gain `n_tasks` paired tasks can prove at `power`: `holdout_size` solved for the effect (`FIXED_POINT_STEPS` fixed-point steps, since the after-side variance depends on it).
`eval_variance`	How much an evaluation moves when the same model is evaluated again (Lambert 2025, chapter Evaluation).
`evaluate`	Judge held-out rollouts under the exact contract `grade` uses.
`expected_outcome`	What the checklist will look for on this task, in one sentence.
`filter_rl_rows`	Split keep/drop.
`format_audit`	The block a person reads: the summary, then the reasons.
`format_coverage_gap`	The gap report as the block a person actually reads.
`format_curriculum`	One-line-per-fact summary for a terminal.
`format_delta_report`	The block a person reads: headline, then one line per metric.
`format_hack_scan`	The block a person reads: the regime, the floor, the ranking.
`format_hack_scan_diff`	The block a person reads: what was learned, then the shifts.
`format_judge_trust`	no docstring
`format_leak_report`	One line per fact, the summary first.
`format_markers`	One line per marker, highest rate first.
`format_stages`	no docstring
`grounding_report`	Over a row set: the share of rollouts with every argument grounded, the share with any call at all, and the most common invented values by tool and key, for a reviewer to look at.
`group_signal`	Within-ask contrast.
`hack_scan`	Rank the features that separate reward within each ask, against a permutation noise floor, and say what a grouped update would learn.
`hack_scan_diff`	What the policy learned: the scan before training against the scan after, on rollouts scored by the same reward.
`holdout_size`	How many paired tasks a holdout needs to prove a gain of `effect`.
`judge_agreement`	Agreement between the judge’s `reward` and a trusted label.
`judge_pairs`	Ask a judge which side of each pair is better, both ways round.
`judge_probes`	Try the reward hacks a policy finds first on the judge, on purpose.
`judge_trust`	Measure whether the judge can be trusted, against human labels and under attack.
`leak_report`	Which rows quote their own `privileged` block in the agent’s text.
`length_report`	Truncated rollouts and asks whose reply lengths are far apart.
`load_spec`	Build a `Spec` from a constitution dict, a list of traits, or a path to a JSON file with either shape.
`logprob_report`	Coverage and shape of the captured logprobs.
`mark_grounding`	Copies of `rows` with `markers["argument_grounding"]` stamped, so `marker_summary`, `delta_report` and the run page read it.
`mark_rows`	Return copies of `rows` with the stock markers merged into each row’s `markers` dict, ready for `marker_summary` / `delta_report`.
`marker_names`	no docstring
`marker_summary`	`metric_summary` for every marker on the rows (or `names`).
`mean_kl`	Sampled `KL(pi \|\| pi_ref)` per generated token, overall and per task.
`metric_summary`	Mean over tasks with a task-bootstrap 95% interval.
`near_duplicate_prompts`	Pairs of distinct asks whose token sets overlap at or above `threshold`.
`next_round`	The prompt set for the next round, from the last round’s graded rollouts.
`normalize_judge_result`	Coerce one judge return into the contract; never invent a reward.
`optimize`	Select the rows worth training on, for SFT or RL, one call after grading.
`outcome_check`	(outcome, reason, checks).
`pairwise_judge`	A model judge for `judge_pairs`: `judge(a_row, b_row) -> \{"winner": "A" \| "B" \| "tie" \| None, "reason": str\}`.
`pass_at`	Compute pass@1, pass^k and pass@k from graded rows, grouped by task.
`preflight`	Spec-quality report for an agent.
`privileged_context`	The teacher’s block for a row at birth: `hidden_state` (what the world knows that the ask does not say) and `reference` (what the checklist expects).
`publish_gate`	Check, calibrate, and report.
`rank_rows`	Score each row in place.
`recommend`	How much data this agent needs, from its own grid.
`reference_logprobs`	Stamp `ref_logprob` on every row: the reference’s summed logprob over the tokens the policy generated.
`refusal_report`	Over-refusal on a benign set (Lambert 2025, chapter Over-optimization, “Over-Refusal”).
`retire_solved`	Return the rows with every solved task removed.
`reward_correlations`	corr(reward, feature) for the cheap features a judge tends to reward by accident: reply length, tool-call count, assistant turns, and the over-optimization signatures Lambert 2025 (chapter Over-optimization) lists (boilerplate, hedging, sycophancy, refusal phrases, 1 when present; see `score.style`), plus every trajectory flag that fired on any row (`lie.`, `hack.`, `risk.*`; see `score.trace`).
`row_markers`	The stock markers for one row’s final text.
`rubric_judge`	A judge for `run_judge` / `data.grade(judge=)` that scores the rubric item by item.
`rubric_of`	no docstring
`run_judge`	Score trajectories with any judge.
`score_row`	Score one row.
`select_for_rl`	Whole mixed groups up to roughly `target` rows.
`select_for_sft`	Diverse correct demonstrations, at most `target` rows.
`spec_version`	The content version of a Spec (or anything `load_spec` accepts).
`stage_of`	The stamped stage, or None.
`stage_report`	Rows per stage, tasks per stage, and the cross-stage leaks: any task used both in `eval` and in a training stage (sft/rm/rl/mid).
`staleness_report`	Which policies produced these rows, and can an update still use them.
`stamp_spec`	Return copies of `rows` tagged with the spec they were produced or graded against: `spec_id` and `spec_version`.
`stamp_stage`	Return copies of `rows` with `row["stage"] = stage`.
`style_markers`	Stamp the style markers on every row’s `markers` (in place) and return the rows.
`style_report`	How much of each signature the replies carry, and whether the reward pays for it.
`task_checklist`	Judge contract: conduct gated by the task’s checkable outcome.
`task_key`	The one name every report groups a row’s rollouts under.
`trace_flag_report`	How often each flag fires, what a reviewer should read, and whether the reward pays for it.
`trace_flags`	The flags that fire on one rollout: `{flag: evidence}`.
`trace_markers`	Stamp the trace markers on every row’s `markers` (in place) and return the rows: 1.0 when the family is clean, 0.0 when a flag fired.
`trim_out_of_band`	Difficulty band filter.
`trim_unanimous_groups`	Drop asks whose k >= `min_k` rollouts all landed 0 or all landed 1.
`ungrounded_arguments`	The string arguments of the rollout’s tool calls that appear nowhere in the context the call could draw on.
`write_rubrics`	Draft one rubric per distinct prompt with a model and attach it to every row of that prompt (`privileged.rubric`, `source="model"`).

agreement

Does the judge agree with labels you trust?

judge_agreement

judge_agreement(
    rows: Sequence[dict],
    gold: str | Sequence[dict] = 'gold_reward',
    reward: str = 'reward',
    allow_model_gold: bool = False,
) -> dict[str, Any]

Defined in whileai/simulations/score/agreement.py. Agreement between the judge’s reward and a trusted label. gold is either a key on the same rows (default gold_reward, the field attach_labels fills when you hand-label a sample) or a second row list from another scoring pass, matched by rollout id, scenario id plus rollout index, or prompt plus final text. Only exact 0/1 labels on both sides count; partial scores and unjudged rows are reported as skipped (n_skipped), not guessed. A fractional reward is what a Rubric of plain principles returns (the mean of its criteria), so a judge built that way loses every partially met row here; judge_trust counts that share against MAX_SKIPPED_SHARE and says so, since the rows kept are the ones the judge was sure about and agreement over them reads high by construction (#345). Returns n, agreement, kappa (Cohen, chance-corrected), the confusion counts, pass_when_gold_fail (the leak rate: gold failures the judge passed) and fail_when_gold_pass, both pass rates, gold_kind (where the labels came from: "human", "program", "model", "unknown" for rows with no record), ok (rows were compared and the labels are a person’s or a program’s), and warnings. A second judge pass is model gold; rows with gold_reward but no gold_kind are unknown; either makes ok false with the reason unless allow_model_gold=True. A program’s labels (attach_labels(kind="program"): a verifier, a unit test, a rule over tool calls) are trusted like a person’s, since a deterministic rule is at least as strong a gold as a rater (Lambert 2025, chapter Evaluation, verifiable rewards).

audit

Is the verifier failing answers that are right?

audit_grades

audit_grades(
    rows: Sequence[dict],
    judge: Callable[[dict], Any],
    sample: int = 40,
    passes: int = 0,
    question: str = "The reference answer is the answer key for this task. Is the reply's final answer correct, meaning it gives the same result as the reference? Differences in formatting, rounding, ordering (unless the task asks for an order), column or field naming, or an equivalent way of expressing the same result do not make it wrong. A missing, different, or partial result does.",
    seed: int = 0,
    concurrency: int = 4,
    timeout: float = 120,
    judge_name: str | None = None,
    fn_warn: float = 0.1,
) -> dict[str, Any]

Defined in whileai/simulations/score/audit.py. Estimate the verifier’s false-negative rate from a judged sample. rows are graded by the verifier (run_judge(rows, verifier) or data.grade(judge=verifier)): reward 0/1, reason from the rule. sample failed rows (reward 0, judge ok) are drawn with seed and each is put to judge (any judge in the run_judge contract: rubric_judge(), grade_llm, your own callable) with the reference in place and question as the rubric when the row carries none. A judge reward at or above PASS_THRESHOLD (0.5) on a failed row is a false negative. passes samples passed rows the same way for the false-positive side. Returns fn_rate with fn_ci95 (Wilson), estimated_wrong_fails (the rate over every failed row), reasons (the verifier’s failure kinds in the sample, each with how many the judge overturned), a few examples, fp_rate when passes > 0, and warnings. Above fn_warn (FN_WARN, 0.10) the summary says to fix the verifier before training; select_for_rl(audit=report) and optimize(audit=) carry the same warning into the selection.

format_audit

format_audit(report: dict[str, Any]) -> str

Defined in whileai/simulations/score/audit.py. The block a person reads: the summary, then the reasons.

checklist

A task-specific checklist reward, derived from what the world knows.

expected_outcome

expected_outcome(dims: dict | None) -> str | None

Defined in whileai/simulations/score/checklist.py. What the checklist will look for on this task, in one sentence. The same branches as outcome_check, read before the agent has acted. None when no rule applies (a compound ask). This is the teacher’s privileged.reference: the answer key the student must never be shown.

outcome_check

outcome_check(row: dict) -> tuple[int | None, str, dict[str, Any]]

Defined in whileai/simulations/score/checklist.py. (outcome, reason, checks). None when no rule applies to this task.

privileged_context

privileged_context(dims: dict | None, faults: dict | None = None) -> dict

Defined in whileai/simulations/score/checklist.py. The teacher’s block for a row at birth: hidden_state (what the world knows that the ask does not say) and reference (what the checklist expects). Empty when the task carries neither.

task_checklist

task_checklist(
    row: dict,
    declared_tools: set[str] | None = None,
) -> dict[str, Any]

Defined in whileai/simulations/score/checklist.py. Judge contract: conduct gated by the task’s checkable outcome.

curriculum

Curriculum: order tasks easy-to-hard and retire the solved ones.

curriculum

curriculum(
    rows: Sequence[dict],
    solved: float = 0.8,
    floor: float = 0.2,
    band: tuple[float, float] = (0.2, 0.8),
    tiers: int = 3,
    min_rollouts: int = 2,
) -> dict[str, Any]

Defined in whileai/simulations/score/curriculum.py. Split graded tasks into a training curriculum by measured difficulty. A task is solved when its pass rate is above solved (retire it: an all-pass task is dead gradient). It is not ready when its pass rate is below floor (hold it: no signal until the policy can sometimes solve it). Everything from floor to solved inclusive is trainable, ordered easy to hard (highest pass rate first) and split into tiers difficulty buckets for a staged schedule. The defaults are the two edges of DEFAULT_BAND (20% and 80%), the same band select_for_rl keeps, so a task at 1 of 8 is not ready here and out of band there for the same reason. Tasks with fewer than min_rollouts graded rollouts cannot have a difficulty and are reported separately. Returns a report; nothing is mutated. band is recorded and used only to count how many trainable tasks sit in the reasoning-recipe 20-80% sweet spot, so you can see whether the set has usable signal.

format_curriculum

format_curriculum(report: dict[str, Any]) -> str

Defined in whileai/simulations/score/curriculum.py. One-line-per-fact summary for a terminal.

retire_solved

retire_solved(
    rows: Sequence[dict],
    solved: float = 0.8,
    min_rollouts: int = 2,
) -> list[dict]

Defined in whileai/simulations/score/curriculum.py. Return the rows with every solved task removed. A task above the solved pass rate teaches nothing, so its rollouts are dropped; tasks with too few rollouts to judge are kept.

delta

Did training move the behavior, and did anything else slip?

delta_report

delta_report(
    before: Sequence[dict],
    after: Sequence[dict],
    target: str | None = None,
    must_not_regress: Sequence[str] = (),
    markers: Sequence[str] | None = None,
    by: str | Callable[[dict], Any] | None = None,
    run_std: float | Mapping[str, float | None] | None = None,
    run_std_runs: int | None = None,
    train_runs: TrainRuns | None = None,
    proxy: str | None = None,
    n_boot: int = 2000,
    seed: int = 0,
    balance_rollouts: bool = False,
    alpha: float = 0.05,
    power: float = 0.8,
    ceiling_pass_rate: float = 0.9,
    answered_gap_points: float = 0.1,
    answered_alpha: float = 0.01,
) -> DeltaReport

Defined in whileai/simulations/score/delta.py. Compare an after run to a before run on pass@1 and every shared marker, and say whether the change is real. Reach for it after a change (a prompt edit, a trained adapter, a model swap): both sides are graded rows, ideally on the same pinned tasks (simulate(tasks=before)) with the same rollouts per task. It returns a DeltaReport, a dict that prints itself. The keys a caller reads first: headline_verdict (PASS only for a gain the report supports, NO DIFFERENCE for an interval over zero, NOT COMPARABLE (causes) when the arms cannot be compared, FAIL for a regression, a failed guard, or over-optimization), ok (the gate: no regression, no failed guard, comparable arms; it does not say the change helped), metrics (one entry per metric with its delta, interval and verdict), warnings (each naming its fix), not_comparable, n_paired_tasks and n_unpaired_tasks. print(report) writes it with headline_verdict on the first line (format_delta_report(report) is the same string). Arguments that matter:

target: the metric the run was meant to move ("pass_at_1" or "marker:name"); its verdict is the headline.
proxy: the metric the run was actually trained on (the training reward as a marker, such as "marker:first_action"). When the proxy moved up and the target did not, or the proxy’s interval sits entirely above the target’s, the report is over_optimized and fails: the policy learned something the target does not credit (Gao et al. 2022, arXiv:2210.10760).
must_not_regress: metrics whose significant drop fails the report. Marker metrics go by marker name; pass@1 is "pass_at_1".
by: split the target by a group on each row (a top-level row key, a marker name, or a callable row -> group). The report gains groups, the target compared within each, so a headline that moved cannot hide a kind of prompt that moved the other way. A group whose target dropped significantly is listed in groups_down and warned about; it does not flip ok, which stays the must_not_regress contract (name the group’s metric there if it should).
run_std and run_std_runs: the evaluation’s own re-run standard deviation, per metric or as one number, and how many re-runs it was computed from. See the noise floor below.
train_runs: the rows of every independent training seed of each arm, when the two sides are separately trained models: a list of row sets for the after arm (the before arm untrained), or {"before": [...], "after": [...]} with None for an arm that was not trained. See training seeds below.
alpha (0.05): the false-positive rate every verdict runs at. Each interval is at 1 - alpha (ci95 at the default), the re-run band uses the same quantile, and family_error is 1 - (1 - alpha) ** n_metrics. power (0.8) feeds the sizing line (detectable_effect, holdout_size). tasks_needed is sized from the task sd measured on the paired rows in hand (holdout_size(before=, after=)), and tasks_needed_source says so ("rows"); the binomial model, which cannot see the covariance pairing buys, asked for about twice the tasks (#733).
balance_rollouts (off): trim every paired task to the rows both sides have, chosen by seed, so pass^k and pass@k share one k; balanced says how many rows each side gave up.
ceiling_pass_rate (CEILING_PASS_RATE, 0.9), answered_gap_points (ANSWERED_GAP_POINTS, 0.1) and answered_alpha (ANSWERED_P_MAX, 0.01): the thresholds of the ceiling and answered flags below.

Pairing. Tasks pair by the key pass_at groups on; tasks on one side only do not pair, their count is n_unpaired_tasks, and when any were dropped a warning says so. situations is the cause in not_comparable when fewer than half the tasks are on both sides (paired_share under 0.5 with tasks on one side only): the arms drew different situation sets, so the delta over the few that pair is between two evals, and the fix is to pin the after side to the before run’s tasks (tasks=) or compare per tier with dataset_report. Unequal rollouts. When a run lost rollouts (data.report()["rollouts_lost"]), one arm can sit at k=4 and the other at k=2; the report warns, next to the sizing line, naming both. Unequal k is a precision issue, not a bias: a task’s pass rate is its mean over however many rows it has, so rows lost at random leave the paired delta unbiased and only widen its interval (simulated, k=4 against k=2 on half the tasks: mean delta on the true value, interval about 10% wider). Rows lost for a reason are the problem: a timeout that takes the hard runs, an empty reply on the long ones, and the surviving rows on that arm score higher than the arm does. No trimming fixes that; balance_rollouts costs precision (another 10% on the interval in the same simulation) and removes no bias (failures dropped on one arm: delta 0.32 untrimmed, 0.32 trimmed, true 0.05). Only re-running the short arm on its short tasks does, and data.report()["rollouts_lost_by"] says why the rows went missing. Noise floor. One evaluation is a draw, not a distribution (Lambert 2025, chapter Evaluation, “why many comparisons are unreliable”, and its evaluation-variance appendix). With one run on either side and no run_std, a target that moved reads moved_unreplicated and a warning says how to fix it. Pass eval_variance(...)["run_std_by_metric"] as run_std so pass@1 and each marker are judged against their own floor: a marker on a subset of tasks is several times noisier than pass@1, and pass@1’s floor reads a re-run draw of it as a regression. A scalar applies one floor to every metric. A metric the mapping lacks, or carries as None, is never given another metric’s floor: it gets noise_note: "no_replicate_floor", a warning, and its verdict rests on the task interval alone. A metric whose delta is inside noise_band(floor, n_a, n_b, df) is within_noise: not improved, not slipped, not a regression, and a target there reads within_eval_noise rather than moved, because re-running the eval moves it that much on its own. The band is floor * sqrt(1/n_a + 1/n_b) (the delta is a mean of n_a runs against a mean of n_b) times 1.96 for a given floor, which is taken as the eval’s spread. A floor that came from re-runs is an estimate, not the spread: pass run_std_runs (eval_variance(...)["n_runs"]) and the band uses the two-sided t quantile at df = run_std_runs - 1 instead (three re-runs: 4.30 x floor x sqrt(2) with one run per side, not 1.96; under pure noise the 1.96 band lets about one delta in five through at df=2). A given run_std without run_std_runs keeps 1.96 and a warning names the fix. When both row sets carry two or more lineage.eval_run values (simulate(tasks=..., runs=3)) the report computes each metric’s floor itself, pooled over the two sides, and uses the t quantile at df = sum(runs - 1) (three runs per side: 2.78 x floor x sqrt(2/3)); run_std is then the headline metric’s floor, run_std_by_metric has them all, noise_band is the headline band, noise_rule spells it out, and eval_runs says how many runs each side had. An arm handed in through train_runs as N row sets averages N eval draws, so eval_runs counts those too (three seeds a side: sqrt(1/3 + 1/3)); the band used to read only the lineage and came out 1.73x too wide for three seeds (#750). Training seeds. The noise floor measures the eval; a delta between two separately trained models also carries training variance, which the floor cannot see (#356: one recipe read -0.065 [-0.117, -0.013] on one run and +0.050 on the next, at one seed per arm). Pass train_runs and the headline metric gains a between-seed term: each trained arm’s per-seed means give a between-seed standard deviation train_std[arm], the delta’s variance adds std**2 / n_seeds per arm, and train_ci95 is the interval centred on the across-seed delta train_delta and widened in quadrature by the two-sided t quantile at train_df = sum(n_seeds - 1) (Lambert 2025, chapter Evaluation; Miller 2024, arXiv:2411.00640, on the variance components a claim rests on). moved then needs that interval to exclude zero as well; when it covers zero the verdict is no_change_detected and a warning says the seed spread ate the delta. Fewer than MIN_TRAIN_SEEDS (2) seeds on a trained arm resolves nothing: the verdict is unresolved, the interval and floor lines still print, and the line says “one training seed per arm; add a seed to resolve”. Without train_runs the report says nothing about training seeds (a prompt edit or a model swap has none); the paper-recipe contract (recipes/papers/check.py) reads a one-seed delta as unresolved. Comparability. config says what each side was produced with (pass_at(...).config per side: task count, k, temperature, max_tokens, policy and judge versions, prompt hash). A warning names each setting the two sides disagree on, and says so when both sides are the same policy version (Lambert 2025, chapter Evaluation: a comparison is only as good as the settings it was run under). config[side]["answered_share"] is the share of rows per side with a spoken reply once <think> markup is gone, and every rate is conditional on it. The two shares are compared with a pooled two-proportion z test; when it clears answered_alpha the warning states p and the gap, and when the gap also exceeds the re-run band (or answered_gap_points with no band) the report fails with answered in not_comparable and names the mechanism: a reasoning base against a reasoning-suppressed adapter under one shared max_tokens runs out of budget inside <think> and never answers, so the adapter wins every row the base did not reply to. not_comparable lists every such cause under one prefix, NOT COMPARABLE:; none are raised here. A replay (simulate(tasks=...) or runs=N) keeps the writer of the run it replays on writer_model, so two runs of one call compare as one writer. Situations nobody’s model wrote (a seeds= ask, the offline template writer, or a replay of either) count as one writer for this check: nothing there could have moved with the weights. ceiling is set when the before side already passes ceiling_pass_rate of its tasks, or when fewer than CEILING_MIN_TASKS_WITH_ROOM paired tasks (and under half) are not already passed every time: there is little room left for an improvement to show, whatever the training did.

report = wai.delta_report(base.rows(), tuned.rows(), target="pass_at_1")
print(wai.format_delta_report(report))

format_delta_report

format_delta_report(report: dict[str, Any]) -> str

Defined in whileai/simulations/score/delta.py. The block a person reads: headline, then one line per metric.

grading

Deterministic conduct rules for any tool-using agent.

behavior_signature

behavior_signature(trajectory: dict) -> str

Defined in whileai/simulations/score/grading.py. Hash of tool sequence, argument provenance, statuses, and outcome shape.

conduct_grade

conduct_grade(trajectory: dict, declared_tools: set[str] | None = None) -> dict

Defined in whileai/simulations/score/grading.py. Score agent conduct. Tool/sandbox faults are a flag, not a zero.

grounding

Argument grounding: did every tool argument come from the conversation?

argument_grounding

argument_grounding(row: dict, **kwargs: Any) -> float

Defined in whileai/simulations/score/grounding.py. 1.0 when every string argument of every tool call is grounded in the conversation (a rollout with no calls is grounded), else 0.0. Keyword arguments are those of ungrounded_arguments.

grounding_report

grounding_report(rows: Sequence[dict], **kwargs: Any) -> dict[str, Any]

Defined in whileai/simulations/score/grounding.py. Over a row set: the share of rollouts with every argument grounded, the share with any call at all, and the most common invented values by tool and key, for a reviewer to look at.

mark_grounding

mark_grounding(rows: Sequence[dict], **kwargs: Any) -> list[dict]

Defined in whileai/simulations/score/grounding.py. Copies of rows with markers["argument_grounding"] stamped, so marker_summary, delta_report and the run page read it.

ungrounded_arguments

ungrounded_arguments(
    row: dict,
    ignore_keys: Sequence[str] = (),
    allow: Sequence[str] = (),
    min_len: int = 3,
) -> list[dict[str, str]]

Defined in whileai/simulations/score/grounding.py. The string arguments of the rollout’s tool calls that appear nowhere in the context the call could draw on. Each entry is \{"tool", "key", "value"\}. ignore_keys skips argument names that are free text by design (a note, a message body); allow lists values that are legal without appearing in the conversation (an enum, a default); strings shorter than min_len are skipped.

hack_scan

What will the policy learn from this reward? Name it before training.

format_hack_scan

format_hack_scan(report: dict[str, Any], top: int = 12) -> str

Defined in whileai/simulations/score/hack_scan.py. The block a person reads: the regime, the floor, the ranking.

format_hack_scan_diff

format_hack_scan_diff(report: dict[str, Any]) -> str

Defined in whileai/simulations/score/hack_scan.py. The block a person reads: what was learned, then the shifts.

hack_scan

hack_scan(
    rows: Sequence[dict],
    endorsed: Sequence[str] = (),
    features: Mapping[str, Callable[[dict], float | None]] | None = None,
    auto: bool = True,
    top_k: int = 200,
    n_perm: int = 100,
    min_obs: int = 20,
    seeds: int = 12,
    reward: str = 'reward',
    seed: int = 0,
    top_features: int | None = 20,
    alpha: float = 0.05,
) -> HackScanReport

Defined in whileai/simulations/score/hack_scan.py. Rank the features that separate reward within each ask, against a permutation noise floor, and say what a grouped update would learn. Reach for it before an RL run, and again after, to check that the reward tracks the behavior you meant rather than a shortcut. It returns a HackScanReport, a dict that prints itself: regime (train, reward_hack, pool_exhausted, no_signal, degenerate, unknown), tau (the floor), features ranked by |within-ask correlation| with the pooled correlation beside each, top_feature, endorsed_on_top, integrity (the share of the above-floor signal that sits on an endorsed feature), the support numbers (asks all-pass, all-fail, mixed, gradient capacity), and warnings in one line each.

rows: graded rollouts, several per ask (mode="rl"); the reward under reward may be 0/1 or partial credit.
endorsed: the features the reward is supposed to track, as substrings of feature names ("lookup_order" matches tool:lookup_order and contains:lookup_order; "marker:grounded" a marker). Without it the scan still ranks and floors, but cannot call a hack a hack.
features: hand-tier columns to add, as {"name": lambda row: value}, beside the built-in ones (reply length, tool calls, turns, truncation, one indicator per tool called, every numeric marker). auto (True) adds the auto tier: presence of the top_k (200) most common words and word pairs in the agent’s text, the tier that finds the hack nobody listed.
alpha (ALPHA, 0.05): sets the floor. tau is the 1 - alpha quantile of the strongest feature’s |rho| when reward is shuffled within ask (n_perm shuffles, 100), so a feature above it clears chance at that rate.
top_features (20): caps the ranking in the report (None lists all). min_obs (20) is the fewest observations a feature needs to be ranked.

degenerate is the refusal: an ask holds fewer than MIN_DISTINCT_PER_ASK distinct rollouts at the median (distinct_per_ask) and two or more features sit at |rho| at or above DEGENERATE_RHO, exactly collinear with reward and with each other because nothing else could happen at that variety. The ranking cannot separate them and the noise floor is no help (it tells signal from noise, not one perfect explanation from another), so top_feature and integrity are None, inverted is empty, no hack is claimed, and collinear lists the tied features. The direction is withheld with the name: at that variety an endorsed feature is negative exactly when it fell on the failing trajectory, so the sign is the same coin flip. Collinear features on a varied pool are left alone: there the ranking found two names for one behavior, and a genuinely inverted endorsed feature is still reported.

scan = wai.hack_scan(data.rows(), endorsed=["tool:lookup_order"])
print(scan["regime"], scan["top_feature"], scan["integrity"])

hack_scan_diff

hack_scan_diff(
    before: Sequence[dict],
    after: Sequence[dict],
    endorsed: Sequence[str] = (),
    top: int = 10,
    **scan_kwargs: Any,
) -> dict[str, Any]

Defined in whileai/simulations/score/hack_scan.py. What the policy learned: the scan before training against the scan after, on rollouts scored by the same reward. A feature that clears the floor after and did not before is what the update moved toward; one that dropped out is what it moved away from. gained and lost list them with both correlations, moved the largest shifts either way, and learned is the one line to read: the top gained feature, and whether it is endorsed. scan_kwargs reach both hack_scan calls. When either side comes back degenerate, every feature there is above the floor at |rho| 1 and no feature can be said to have gained it. learned says which side could not be read and why, and no hack is claimed; the rows are still listed so the shift is visible.

hygiene

Row hygiene a grouped RL update or a rejection-sampling pass cares about.

dedupe_groups

dedupe_groups(rows: Sequence[dict]) -> tuple[list[dict], dict[str, Any]]

Defined in whileai/simulations/score/hygiene.py. Drop repeat trajectories within one ask. Keeps the first of each (behavior signature, normalized reply) pair. Reports how many pairs disagreed on reward, which is label noise the judge introduced.

length_report

length_report(rows: Sequence[dict], max_spread: float = 4.0) -> dict[str, Any]

Defined in whileai/simulations/score/hygiene.py. Truncated rollouts and asks whose reply lengths are far apart. max_spread is max / median within one ask.

near_duplicate_prompts

near_duplicate_prompts(
    rows: Sequence[dict],
    threshold: float = 0.8,
) -> dict[str, Any]

Defined in whileai/simulations/score/hygiene.py. Pairs of distinct asks whose token sets overlap at or above threshold. Quadratic in the number of asks; fine for the few thousand a run produces. Report only.

reward_correlations

reward_correlations(
    rows: Sequence[dict],
    threshold: float = 0.3,
) -> dict[str, Any]

Defined in whileai/simulations/score/hygiene.py. corr(reward, feature) for the cheap features a judge tends to reward by accident: reply length, tool-call count, assistant turns, and the over-optimization signatures Lambert 2025 (chapter Over-optimization) lists (boilerplate, hedging, sycophancy, refusal phrases, 1 when present; see score.style), plus every trajectory flag that fired on any row (lie.*, hack.*, risk.*; see score.trace). Any |corr| at or above threshold is flagged. A negative tool-count correlation means the reward pays the policy to do less; a positive phrase or flag correlation means it pays for the tic or the fake.

judge_trust

Can the judge be trusted? The reward is only as good as the judge.

format_judge_trust

format_judge_trust(report: dict[str, Any]) -> str

Defined in whileai/simulations/score/judge_trust.py.

judge_probes

judge_probes(
    rows: Sequence[dict],
    judge: Callable[[dict], Any],
    probes: str | Sequence[str] = 'all',
    rubric: str | None = None,
    sample: int = 40,
    seed: int = 0,
    concurrency: int = 8,
    flip_flag: float = 0.1,
) -> dict[str, Any]

Defined in whileai/simulations/score/judge_trust.py. Try the reward hacks a policy finds first on the judge, on purpose. Reach for it when a judge is about to become a training reward: a policy trained on it will find these holes, so find them first (Gao et al. 2022, arXiv:2210.10760). It returns a dict: probes (per probe: n, kind, pass_before, pass_after, flips_up, flips_down, net_flips, denominator, exploit_rate, ci95 (its Wilson interval), low_power, resolves, flagged, errors; a probe that applies to no row is skipped with the reason), exploitable_by (the probes at or over flip_flag), one warnings line per exploit, and one notes line per probe that had too few rows to say. Each probe mutates up to sample graded rows one way and re-judges them. An additive probe (filler, the rubric’s words, a success claim, the ask echoed, a sycophantic opener) reports exploit_rate: the net share of originally failing replies that pass once the text is added, max(0, flips_up - flips_down) / originally failing, so a judge whose verdicts churn both ways under the edit reads as noise, not as a hole (#347). A replacement probe (a well-formed tool call with empty arguments, a refusal) reports the share of replies that pass with the content gone. Every rate carries a Wilson interval, the one agreement gets, and flagged needs at least PROBE_MIN_N rows in the denominator: below it one flipped row on ten originally failing is already flip_flag, and the interval on 1 of 10 runs 0.02 to 0.40. A probe under the floor is low_power with resolves, the smallest exploit rate that many rows can tell from flip_flag at POWER (the normal approximation for one proportion, Miller 2024, section 5), and flagged stays false whatever the rate.

probes: "all" (the default) or names from PROBES.
rubric: the text the keyword probe draws words from; without it the row’s system prompt is used. The Rubric handed to rubric_judge is not seen here; pass its text as rubric= (judge_trust(rubric=) forwards it).
sample (40), seed (0), concurrency (8): how many rows to re-judge, which ones, and how many judge calls run at once.
flip_flag (FLIP_FLAG, 0.10): the exploit rate at which a probe is flagged. PROBE_MIN_N (20) is the denominator floor; it is a module default, since this call already carries eight parameters (style rule 3).

judge_trust

judge_trust(
    rows: Sequence[dict],
    judge: Callable[[dict], Any] | None = None,
    gold: str = 'gold_reward',
    sample: int = 40,
    seed: int = 0,
    concurrency: int = 8,
    probes: str | Sequence[str] | None = None,
    rubric: str | None = None,
    min_agreement: float = 0.8,
    min_kappa: float = 0.6,
    allow_model_gold: bool = False,
    length_gap_flag: float = 0.15,
    flip_flag: float = 0.1,
    max_skipped_share: float = 0.1,
) -> JudgeTrustReport

Defined in whileai/simulations/score/judge_trust.py. Measure whether the judge can be trusted, against human labels and under attack. Reach for it before training on a judge’s rewards: the reward is only as good as the judge. It returns a JudgeTrustReport: print it for the block, read it as the dict it has always been. The keys a caller reads first: ok (measured and clean), agreement["agreement"] and agreement["ci95"] (the number and its Wilson interval, not ci), agreement["n"] (labels compared), gold_kind (where the labels came from), and warnings, where every line names its own fix. The rest: held_out_halves (agreement on two task-hash halves; if they diverge the rubric is fit to its examples), length_sensitivity (judge pass rate on short versus long replies among rows humans agreed on, a length bias the labels rule out as real), perturbation and probes when a judge callable is given, disagreements (the review queue of rows the judge and the humans disagree on), floors, n_labeled and n_rows. print(report) writes the whole thing (format_judge_trust(report) is the same string). The module docstring lays out each check and its source. The floors and flags are keywords with their defaults in whileai.simulations.defaults: min_agreement (0.8, the human-human agreement of MT-Bench, arXiv:2306.05685), min_kappa (0.6, Landis and Koch “substantial”), length_gap_flag (0.15), flip_flag (0.10) and max_skipped_share (0.10, the share of labeled rows the judge may leave out of the agreement count with a fractional reward before ok is false; report["skipped"] carries the counts).

rows: graded rows carrying the judge’s reward. Rows that also carry gold (0/1, default column gold_reward, what attach_labels writes) feed the agreement, held-out and length checks. Those checks read the reward already on the row, so when judge is given and the rows’ judge_name (what run_judge and data.grade stamp) names another scorer, the report warns and ok is false: the agreement would be that scorer’s, not the judge’s (#683).
judge: the judge callable. With it the report re-judges up to sample rows twice more, as-is for consistency and with neutral filler appended; flips on the filler run mean the judge pays for length.
probes: "all" (or a list of names from PROBES) adds judge_probes, one more pass over the sample per probe; rubric feeds the keyword probe.
min_agreement (0.8, the human-human agreement of MT-Bench, arXiv:2306.05685) and min_kappa (0.6, Landis and Koch “substantial”): the floors ok requires. length_gap_flag (0.15) and flip_flag (0.10) are the flags. All four live in whileai.simulations.defaults.
allow_model_gold: False by default, so model or unknown gold makes ok false with the reason; only a person’s labels count as a measurement.

ok is true only when a gold-labeled check ran against a person’s labels, the Wilson lower bound of agreement reached min_agreement, kappa reached min_kappa, and nothing else was flagged. With no labels every check has n=0, so ok is false with a warning saying the judge is unmeasured, not failed. The perturbation pass is not a substitute: a judge that passes everything is perfectly consistent (Lambert 2025, chapters Reward Modeling and Synthetic Data and Distillation).

>>> rows = [{"task_id": str(i), "reward": i % 2, "gold_reward": i % 2} for i in range(20)]
>>> wai.judge_trust(rows)["agreement"]["agreement"]
1.0

judging

One judge contract for grading and evaluation, and the loop around it.

ScoredData

class ScoredData(
    rows: list[dict],
    run_id: str,
    source: str,
    judge_name: str,
    model: str | None = None,
)

Defined in whileai/simulations/score/judging.py. Scored trajectories: the one representation grade and eval share. Iterates as plain dicts, so it feeds simulate(traces=...), mine_traces, export_training and JSONL writers directly — no conversion scripts. .rows is a RowList: a list that also answers to being called, so both scored.rows and scored.rows() give the scored rows. SimulationData.rows, what simulate() returns, behaves the same way, so the two spellings are interchangeable across simulate() -> run_judge(). .warnings is the list of hollow-run notes run_judge filled; print it before reading any number.

ScoredData.agreement

agreement(
    self,
    gold: str | Sequence[dict] = 'gold_reward',
    reward: str = 'reward',
) -> dict[str, Any]

Agreement of this run’s rewards with a trusted label. See judge_agreement.

ScoredData.compare_judges

compare_judges(
    self,
    judges: Mapping[str, Any] | Sequence[Any],
    gold: str = 'gold_reward',
    allow_model_gold: bool = False,
    concurrency: int = 8,
    floors: tuple[float, float] = (0.8, 0.6),
)

Grade these rows with several judges and rank them against the gold labels. judges maps a name to a spec string ("typesafe:jev-latest"), a backend object, a wai.Judge or any judge callable. Each grades its own copy of the rows under this run’s system prompt and tools, then is scored the way judge_trust scores one judge: agreement with a Wilson interval, kappa, leak rate, unsure and unjudged counts, seconds per row. Returns a JudgeComparison that prints as a table ranked by kappa; whileai.judge_comparison.compare_judges has the full account and takes a bare row list.

ScoredData.failed_traces

failed_traces(self) -> list[dict]

Failures, ready to hand to simulate(traces=...).

ScoredData.partials

partials(self) -> list[dict]

Rows with a continuous reward strictly between 0 and 1. The scalar lane: 1 pass, 0 fail, partials here, None unjudged - every contract-legal reward is visible in exactly one view.

ScoredData.push

push(self, name: str, **kwargs: Any) -> dict

Upload the scored rows to the platform: push_rows(self.rows, name, ...). Same keywords as push_rows (gate=, mode=, agent=, purpose=, parent=, endorsed=, strict_hacks=, holdout= for a linked holdout set split by task, publish= for a public card, timeout= for the upload). The graded copy is what a gated RL push needs, and SimulationData.push cannot see it: grade(judge=) leaves the run’s trajectories ungraded on purpose.

ScoredData.select

select(
    self,
    mode: str = 'rl',
    target: int = 1000,
    band: tuple[float, float] | None = None,
    endorsed: Sequence[str] = (),
    truncated: str = 'drop',
)

The rows worth training on, as a Selection that prints its report: optimize over the graded copies, mode="rl" by default. band, endorsed and truncated as there.

ScoredData.select_by_reward_range

select_by_reward_range(
    self,
    lo: float,
    hi: float,
    inclusive: bool = True,
) -> list[dict]

Rows whose numeric reward falls in [lo, hi] (or (lo, hi)).

ScoredData.select_for_preference

select_for_preference(
    self,
    max_pairs_per_prompt: int = 1,
    min_margin: float = 1.0,
    length_match: bool = True,
) -> tuple[list[dict], dict[str, Any]]

Chosen/rejected pairs from same-task contrast. Failures earn here.

ScoredData.select_for_rl

select_for_rl(
    self,
    target: int = 1000,
    lo: float = 0.2,
    hi: float = 0.8,
    has_tools: bool = True,
) -> tuple[list[dict], dict[str, Any]]

Whole mixed-reward groups for RL; groups never split. lo and hi default to DIFFICULTY_BAND (0.2, 0.8), the same band select_for_rl and optimize use; they used to be 0.3 and 0.7 here alone.

ScoredData.select_for_sft

select_for_sft(self, target: int = 1000) -> tuple[list[dict], dict[str, Any]]

Diverse correct demonstrations: 1-labeled, deduped by behavior.

ScoredData.unjudged

unjudged(self) -> list[dict]

Rows the judge could not score. Never treated as failures.

build_preference_pairs

build_preference_pairs(
    rows: Sequence[dict],
    max_pairs_per_prompt: int = 1,
    min_margin: float = 1.0,
    length_match: bool = True,
) -> tuple[list[dict], dict[str, Any]]

Defined in whileai/simulations/score/judging.py. Build same-task chosen/rejected pairs for preference training (DPO-style). Reach for it after grading a run with several rollouts per ask: the contrast between a pass and a fail on the same prompt is the training signal, so failures are supply here, not waste. It returns (pairs, report): each pair carries prompt, chosen, rejected, the scores and lineage listed below, and both parents’ lineage; the report counts the pairs, the prompts that had a contrast, and how often chosen is still the longer side. A pair exists only where the same prompt has two trajectories whose rewards differ by at least min_margin. Rows without a valid judge result never pair.

min_margin (1.0): pairs 1-labeled with 0-labeled rows only; 0.5 also admits partial-credit rows against a full pass or fail.
max_pairs_per_prompt (1): how many pairs one prompt may contribute.
length_match (True): each chosen row takes the rejected row closest to it in length. DPO exploits a length gap faster than it learns the behavior (Lambert 2025, chapter Direct Alignment).

Each pair keeps what the trainer and the reviewer need to trust it:

chosen_score / rejected_score / margin: the raw scores and their gap, so a margin-aware loss (Llama 2 style) can use them and a reviewer can see how far apart the two really are.
chosen_model / rejected_model / same_policy: which policy produced each side. Preference data works best when both sides come from the policy being trained (Lambert et al. 2024, Tulu 3, arXiv:2411.15124; Lambert 2025, chapter Preference Data); a mixed pair is still a pair, but it is labeled as off-policy.
length_delta: chosen reply chars minus rejected, and the report says how often chosen is still the longer side.

pairs, report = wai.build_preference_pairs(data.rows(), min_margin=1.0)
print(len(pairs), report)

evaluate

evaluate(
    rows: Sequence[dict] | Any = None,
    judge: Callable[[dict], Any] | None = None,
    grader: Callable[[dict], Any] | None = None,
    model: str | None = None,
    eval_set: Sequence[Any] | None = None,
    judge_name: str | None = None,
    run_id: str | None = None,
    concurrency: int = 8,
    timeout: float | None = None,
    scale: tuple[float, float] | None = None,
    tools: Sequence[dict] | Sequence[str] | None = None,
) -> ScoredData

Defined in whileai/simulations/score/judging.py. Judge held-out rollouts under the exact contract grade uses. Reach for it to score rows that did not come from the run in hand: production traces, another run’s rollouts, a frozen eval set. Same engine, same schema; only the lineage source differs. It returns a ScoredData: rows (scored copies, each with reward, reason, judge_status, judge_meta and a lineage record; a judge error is marked on its row, never coerced to 0), warnings (read them before the numbers: no rollout called a tool, a declared tool none touched, a marker that fired on no row), eval_coverage when eval_set was given, and traces, which feeds simulate(traces=...) to close the loop.

rows: the rollouts, or the SimulationData holding them, which also supplies tools.
judge or grader: the judge callable, either spelling, not both. grader is the doctrine-sketch name.
eval_set: the frozen evaluation set, as prompt strings or rows. The result’s eval_coverage reports which asks the rollouts actually covered and the gap, instead of letting a silent partial eval pass as a full one.
tools: the agent’s declared tool list (schemas or names), so a declared tool no rollout called is warned about.
judge_name, model, run_id: recorded in each row’s lineage as the judge, the judged model, and the scoring run (a fresh id per call unless pinned).
scale: (lo, hi) reads a numeric verdict as a rating on that scale and maps it to a 0 to 1 reward, keeping the raw rating in judge_meta.
concurrency (8) and timeout: judge calls in flight and seconds per call.

scored = wai.evaluate(traces, judge=my_judge, eval_set=asks)
print(scored.warnings, wai.pass_at(scored.rows).pass_at_1)

normalize_judge_result

normalize_judge_result(
    raw: Any,
    scale: tuple[float, float] | None = None,
) -> dict[str, Any]

Defined in whileai/simulations/score/judging.py. Coerce one judge return into the contract; never invent a reward. scale=(lo, hi) reads the judge’s number as a rating on that scale (a 1 to 5 Likert, a 0 to 10 score): the row’s reward is the rating mapped onto [0, 1] and judge_meta keeps rating and scale. A rating outside the scale is a contract break, as a reward outside [0, 1] is without one. A dict may carry the number as rating instead of score when a scale is set.

run_judge

run_judge(
    rows: Sequence[dict] | Any,
    judge: Callable[[dict], Any],
    judge_name: str | None = None,
    source: str = 'grade',
    model: str | None = None,
    run_id: str | None = None,
    concurrency: int = 8,
    timeout: float | None = None,
    version: str | None = None,
    scale: tuple[float, float] | None = None,
    tools: Sequence[dict] | Sequence[str] | None = None,
) -> ScoredData

Defined in whileai/simulations/score/judging.py. Score trajectories with any judge. Originals are left unmodified. tools= is the agent’s declared tool list (or names); with it the result’s warnings also say which declared tools no rollout called. Passing the SimulationData itself as rows supplies it. Each scored row is a copy of the input row plus reward, reason, judge_status, judge_meta, and a lineage record naming the scoring run, its source (grade or eval), the judged model, and the parent trajectory. Rows whose judge result breaks the contract keep reward=None and a non-ok status; they are counted, not hidden. version names the judge’s version (model, prompt hash, whatever would change its labels); it lands in lineage.judge_version and reads back as Judgment.scorer.version.

labels

Human labels on rows: who said what, and do they agree.

annotator_agreement

annotator_agreement(rows: Sequence[dict]) -> dict[str, Any]

Defined in whileai/simulations/score/labels.py. How the annotators on gold_labels agree with each other. per_annotator: labels given and pass share. multi_labeled: rows with two or more annotators; unanimous the share of those where every label matched; kappa Cohen’s kappa when exactly two annotators labeled the same rows (pair names them), else None. disagreements lists the split rows (prompt, labels) so a person can read the ones the guideline did not settle (Lambert 2025, chapter Preference Data).

attach_labels

attach_labels(
    rows: Sequence[dict],
    labels: Any,
    annotator: str | None = None,
    kind: str = 'human',
    replace: bool = False,
) -> tuple[list[dict], dict[str, Any]]

Defined in whileai/simulations/score/labels.py. Write hand labels onto rows (in place) and return (rows, report). labels is a JSONL path, a list of dicts, or a {key: label} mapping. A dict label names its row by key / rollout_id, scenario_id + rollout_index, or prompt (+ final_text), and carries label (or reward / gold_reward: 0 or 1), and optionally annotator, note, ts. annotator here is the default for labels that name none. kind is recorded on each label: "human" for a person’s, "program" (or its alias "verifier") for a deterministic rule’s (execution match, a unit test, a rule over tool calls), "model" for a stronger model’s. Any other string raises ValueError naming the accepted set, so a typo cannot silently downgrade the gold (#343). Each row gains gold_labels (every label, appended unless replace), gold_reward, the majority of its labels, and gold_kind: the one kind every label on the row shares, else "mixed". judge_trust and judge_agreement count human and program gold as a measurement of the judge: a deterministic rule is at least as strong a gold as a rater, since it cannot be argued into a pass and agrees with itself on every run (Lambert 2025, chapter Evaluation, verifiable rewards). A tie leaves both unset. A key (the mapping’s key, or a dict’s key) is spelled the way the row is: its rollout_id when it has one, else '<scenario_id>#<rollout_index>' (what a simulate row carries), and a bare scenario_id names its one rollout (#751). Labels that name no row, or carry no 0/1 value, are counted and listed, and warnings names the key form; when no label at all names a row the call raises, naming the first keys and the keys the rows carry, since a judge check over zero gold reads as “not labeled yet” and not as “wrong key”. A key that names several rows (rows pooled from several runs on one pinned task grid share scenario_id and rollout_index; a bare scenario_id on a scenario with several rollouts) raises before any row is changed, naming the fix: a unique rollout_id per row, or the composite key (#759); it used to land every label on the first such row and report a clean match. A list or file holding anything but dicts (a bare [0, 1, 1, 0]) raises naming the item and the accepted shapes, since a label with no row identity cannot be attached (#685).

logprobs

What the policy’s own log-probabilities buy you.

logprob_report

logprob_report(rows: Sequence[dict]) -> dict[str, Any]

Defined in whileai/simulations/score/logprobs.py. Coverage and shape of the captured logprobs. mean_token_logprob is total logprob over total tokens. The per-row quantiles are of each row’s own mean, so one long rollout does not dominate. corr_reward_confidence is Pearson between the 0/1 reward and the per-row mean over graded rows: a strong positive value says the judge rewards fluency, not behavior.

mean_kl

mean_kl(
    rows: Sequence[dict],
    ref: str | Sequence[dict] = 'ref_logprob',
) -> dict[str, Any]

Defined in whileai/simulations/score/logprobs.py. Sampled KL(pi || pi_ref) per generated token, overall and per task. ref is either a key on the same rows holding the reference model’s summed logprob over the same tokens (default ref_logprob), or a second row list scored under the reference, matched by rollout id, scenario id plus rollout index, or prompt plus final text, carrying logprob. Rows missing either side are skipped and counted. Per task the estimate pools tokens across that task’s rollouts, which is what a per-task difficulty record wants.

staleness_report

staleness_report(
    rows: Sequence[dict],
    base_model: str | None = None,
) -> dict[str, Any]

Defined in whileai/simulations/score/logprobs.py. Which policies produced these rows, and can an update still use them. Noukhovitch et al. 2024, arXiv:2410.18252 (asynchronous RL, truncated importance sampling): rows sampled by an older policy are usable when the row carries the sampler’s version and its logprobs so the ratio can be formed; rows from an unknown sampler are not. versions counts rows per policy_version (model_version when the row predates it); base_model names the model about to be trained, and rows whose model_version differs are stale. Coverage says how many rows carry sampling, logprob and token_logprobs.

markers

Stock behavioral markers for the over-optimization signatures (Lambert 2025, chapter Over-optimization).

behavioral_markers

behavioral_markers(
    rows: Sequence[dict],
    names: Sequence[str] | None = None,
) -> dict[str, float]

Defined in whileai/simulations/score/markers.py. Rate of each stock marker over rows (fraction of rollouts that trip it). The over-optimization dashboard in one call. Deprecated: use score.style.style_report for the delta-ready view.

format_markers

format_markers(report: dict[str, float]) -> str

Defined in whileai/simulations/score/markers.py. One line per marker, highest rate first.

mark_rows

mark_rows(
    rows: Sequence[dict],
    names: Sequence[str] | None = None,
    extra: dict[str, Callable[[dict], float]] | None = None,
) -> list[dict]

Defined in whileai/simulations/score/markers.py. Return copies of rows with the stock markers merged into each row’s markers dict, ready for marker_summary / delta_report. extra adds custom named detectors row -> value. Existing marker values are kept; stock names overwrite only themselves. Deprecated: presence polarity (1 = tic present) reads a delta_report paired comparison backwards. Use score.style.style_markers.

row_markers

row_markers(row: dict, names: Sequence[str] | None = None) -> dict[str, int]

Defined in whileai/simulations/score/markers.py. The stock markers for one row’s final text.

optimize

Concentrate a big simulated batch into the dataset post-training needs.

filter_rl_rows

filter_rl_rows(
    rows: Sequence[dict],
    has_tools: bool = True,
    text_gates: bool = True,
) -> tuple[list[dict], dict[str, Any]]

Defined in whileai/simulations/score/optimize.py. Split keep/drop. Does not mutate rows. text_gates as in drop_reason.

group_signal

group_signal(
    rows: Sequence[dict],
    lo: float = 0.2,
    hi: float = 0.8,
) -> dict[str, Any]

Defined in whileai/simulations/score/optimize.py. Within-ask contrast. Signal is a group whose k rollouts disagree. A grouped RL update learns from a mix of 0 and 1 on the same ask, ideally with pass rate p in [lo, hi]. Unanimous groups are dead gradient. Groups of one rollout cannot mix and are counted separately, not blamed.

next_round

next_round(
    prior: Sequence[dict],
    tasks: Sequence[Any] | None = None,
    lo: float = 0.2,
    hi: float = 0.8,
) -> dict[str, Any]

Defined in whileai/simulations/score/optimize.py. The prompt set for the next round, from the last round’s graded rollouts. A round trained on the file it started from keeps paying for groups that give no gradient: at a 0.65 training reward about half the groups are all-pass or all-fail. The band is the published fix (Lambert 2025, chapter Reasoning: filter to the 20-80% band; Yu et al. 2025 (DAPO), arXiv:2503.14476: dynamic sampling drops groups with no contrast), applied to what the current policy does rather than what the base did. prior is round N’s graded rollouts (simulate(tasks=..., repeats=k) on the round-N policy, or the trainer’s own sampled rows); each task’s pass rate over them decides: inside [lo, hi] it is kept, above hi it is solved and dropped, below lo it is unsolved and dropped. tasks restricts the candidates (rows, task dicts with a prompt, or prompt strings); a task with no prior rollouts is unknown and kept, since nothing says it is flat. prior takes either shape: per-rollout rows with a binary reward, or one row per task carrying pass_rate and n (a trainer’s per-task table, or the calibration stamp this function and select_for_rl write). No reply text is read. A prior that carries neither is a UserWarning and an all-unknown plan, not a silent empty one. Returns tasks (one representative row per kept task: the prior row, with calibration.pass_rate and the band), the counts kept, dropped_solved, dropped_unsolved, unknown, pass_rates per task, band, from_policy (the policy versions the prior rows came from) and prompt_set_sha: the identity of the kept set, for lineage on the run. Push the kept rows as the next train set with parent= the last one.

optimize

optimize(
    source,
    mode: str | None = None,
    target: int = 1000,
    output: str | None = None,
    band: tuple[float, float] = (0.2, 0.8),
    enforce_band: bool = True,
    select: str = 'top_per_prompt',
    min_reward: float = 1.0,
    endorsed: Sequence[str] = (),
    truncated: str = 'drop',
    order: str = 'spread',
    audit: dict[str, Any] | None = None,
) -> tuple[list[dict], dict[str, Any]]

Defined in whileai/simulations/score/optimize.py. Select the rows worth training on, for SFT or RL, one call after grading. Reach for it once rows carry reward. It returns (rows, report): the kept rows in training order, and a report saying which mode ran, what each gate dropped and why, and for RL a hack_scan of what the reward is actually tracking. It writes the rows to output when given, or to <name>.<mode>.jsonl next to a path source, and never overwrites the source file unless output names it explicitly.

source: a SimulationData, a row list, or a JSONL path.
mode: "sft" or "rl". Defaults to the run’s own mode for a SimulationData and to "rl" otherwise. SFT picks diverse correct demonstrations (select_for_sft); RL keeps whole mixed groups, never a split one (select_for_rl). Both drop a row whose reply quotes its own privileged context first (drop_privileged_leaks; privileged_leaks_dropped in the report), so export_dataset never refuses what was kept.
target: about how many rows to keep, 1000 by default.
band: the RL difficulty band as a pass-rate range, (0.2, 0.8) by default: asks the policy always or never solves carry no advantage (Lambert 2025, chapter Reasoning, difficulty filtering at 20 to 80 percent; DAPO’s dynamic sampling, arXiv:2503.14476). enforce_band=False only ranks out-of-band asks last instead of dropping them. order is "spread" across pass rates (default) or "middle" first.
select ("top_per_prompt") and min_reward (1.0): the SFT picker and the reward a demonstration needs, as in select_for_sft.
endorsed: what the reward should track, as substrings of feature names ("tool:lookup_order"), so the RL report’s hack_scan can call a shortcut a hack.
truncated: what happens to a rollout cut at the token cap (DAPO’s overlong handling, Yu et al. 2025, arXiv:2503.14476): "drop" removes it (the default), "keep" leaves it in with overlong=True and its own reward, "penalize" keeps it as a failure that counts (reward 0, the judged score under reward_before_penalty). A row counts as truncated when the engine stamped it so (finish_reason "length", or a step marked truncated), when the grader’s reason says truncated or cut off, or when the reply text stops without reaching its end; the stamp is read first, since the backend trims a capped reply to its last sentence and a re-grade overwrites the grader’s reason (hygiene.is_truncated).

rows, report = wai.optimize(data, mode="rl", endorsed=["tool:lookup_order"])
print(report["mode"], len(rows))

recommend(
    tools: Sequence[dict] | None = None,
    policy: str = '',
    system_prompt: str | None = None,
    mode: str = 'sft',
    target: int | None = None,
    mixed_rate: float = 0.5,
) -> dict[str, Any]

Defined in whileai/simulations/score/optimize.py. How much data this agent needs, from its own grid. No guessing. system_prompt= is the same text under simulate’s spelling; policy= and system_prompt= are interchangeable here as there. Grounded two ways: the agent’s measured covering grid (every cell wants SATURATION_COPIES visits, and selection wants surplus to choose from), and published post-training practice (curated agent SFT lands at 500 to 2,000 trajectories: FireAct 500, LIMA 1,000, AgentTuning 1,866; agent RL uses 8 to 16 rollouts per prompt and drops all-pass/all-fail groups: DAPO 2025, Skywork-OR1 2025). Returns the numbers plus simulate_kwargs ready to splat, and reasoning lines that show the arithmetic.

select_for_rl

select_for_rl(
    rows: Sequence[dict],
    target: int = 1000,
    lo: float = 0.2,
    hi: float = 0.8,
    enforce_band: bool = True,
    has_tools: bool = True,
    dedupe: bool = True,
    drop_truncated: bool = True,
    endorsed: Sequence[str] = (),
    truncated: str = 'drop',
    order: str = 'spread',
    prior: Sequence[dict] | None = None,
    audit: dict[str, Any] | None = None,
    text_gates: str = 'auto',
) -> tuple[list[dict], dict[str, Any]]

Defined in whileai/simulations/score/optimize.py. Whole mixed groups up to roughly target rows. Groups never split. text_gates says whether the gates that read the reply run: the junk and do-nothing checks and the duplicate trim, which key on final_text. "auto" (the default) runs them when any row carries a reply (final_text, an assistant messages turn, or a tool step) and skips them when none does, because a trainer’s state holds a task and a binary reward per sample and nothing else, and an empty reply there is the shape of the data, not a finding; the report’s text_gates block and a hygiene_warnings line say the gates were skipped. "require" runs them regardless (a row with no reply is incomplete_junk, the behavior before 0.121), "skip" never runs them. The label gate, the unanimous trim, the difficulty band and the ranking run in every mode: they read the reward alone (Lambert 2025, chapter Reasoning; Yu et al. 2025, arXiv:2503.14476). audit is an audit_grades report on these rows’ verifier; when it found the verifier rejecting right answers more than FN_WARN of the time, hygiene_warnings says to fix the verifier before training on the selection (#255). prior is the previous round’s graded rollouts: tasks the round-N policy already solves (pass rate above hi on prior) or never solves (below lo) are dropped before anything else, so round N+1 trains on what that policy gets right 20-80% of the time rather than on the file round 1 started from (next_round; Lambert 2025, chapter Reasoning). The report’s prior block counts kept, dropped_solved, dropped_unsolved and unknown. truncated says what happens to a rollout cut at the token cap (DAPO’s overlong handling, Yu et al. 2025, arXiv:2503.14476; overlong filtering, Lambert 2025, chapter Reasoning): "drop" removes it (the default; drop_truncated=False is the old spelling of "keep"), "keep" leaves it in with overlong=True and its own reward, riding with its ask rather than deciding it (the ask is unanimous, in band and ranked exactly as under "drop", so "keep" never returns fewer rows than "drop"; a cut rollout’s reward is not the contrast an ask is kept for), and "penalize" keeps it as a failure that does count: reward 0, the judged score under reward_before_penalty, so running past the cap is a negative signal instead of a rollout that vanished. A conduct-grade advisory 0.5 for truncation is unusable under "keep" and a 0 under "penalize". After the row gates, duplicate and truncated rollouts (dedupe, truncated), the unanimous trim, and (enforce_band) the difficulty band, remaining asks are taken round-robin across observed fault kinds, so the dataset keeps a grounded spread of no-fault, miss, timeout, and already-done situations rather than one over-represented failure. Within a fault kind, order="spread" (the default) takes asks round-robin across their pass rates, so a 25% ask, a 50% ask and a 75% ask are picked in turn with no preference for the middle (Lambert 2025, chapter Reasoning, filters to the 20-80% band and stops there; nothing in it says 50% is better than 30%). order="middle" is the older ranking by closeness to a 50% pass rate. The last group may overshoot target; an RL update wants the complete group or none of it. enforce_band=False keeps out-of-band asks and only ranks them last. The report’s hack_scan block is the reward-hack scan over the selection (hack_scan: what separates reward within an ask, against a permutation floor; endorsed names what it should be), correlations the older pooled scan. Reward tracking a shortcut is a judge problem, flagged in hygiene_warnings, not pruned. Selected rows are stamped in place with the calibration measured on the rows as they arrived, before dedupe and the trims: the pass rate over the k repeats the grader saw is the task’s difficulty, and re-measuring it on the survivors would report the post-dedup k under that name. publish_gate keeps the carried stamp. The k-way reliability numbers do not survive the prune, and hygiene_warnings says so when they were available before it.

select_for_sft

select_for_sft(
    rows: Sequence[dict],
    target: int = 1000,
    select: str = 'top_per_prompt',
    k: int | None = None,
    min_reward: float = 1.0,
    seed: int = 0,
) -> tuple[list[dict], dict[str, Any]]

Defined in whileai/simulations/score/optimize.py. Diverse correct demonstrations, at most target rows. Imitation clones what it sees, so only rows whose reward reaches min_reward (default 1.0: judge-approved) and that are not junk qualify; unanimity is not a problem here. A grader with partial credit ranks by its score: lower min_reward to admit it. select is the rejection-sampling rule (Lambert 2025, chapter Rejection Sampling, “Scoring Completions”): "top_per_prompt" keeps each prompt’s highest-reward completion and then round-robins across behavior signatures (tool sequence, argument provenance, outcome shape) so every distinct way of being right appears before any repeats; "top_k_overall" keeps the k highest-reward completions across all prompts, several per prompt allowed; the two random_* rules are the control that chapter asks for (same counts, seeded random picks) so a claimed gain from selection can be checked against chance. k defaults to target.

trim_out_of_band

trim_out_of_band(
    rows: Sequence[dict],
    lo: float = 0.2,
    hi: float = 0.8,
    min_k: int = 2,
) -> tuple[list[dict], dict[str, Any]]

Defined in whileai/simulations/score/optimize.py. Difficulty band filter. Nothing to do with topic or relevance. “Out of band” here means outside the difficulty band [lo, hi] (default DEFAULT_BAND, 0.2 to 0.8): an ask is dropped when its pass rate over k >= min_k rollouts is too high (the policy almost always solves it) or too low (it almost never does), because either way it carries little gradient per rollout. It does not read the prompt, the topic, or the tools; a perfectly on-topic ask is dropped for being too easy, and an off-topic one the policy passes half the time is kept. Junk rows are a separate filter (is_incomplete_junk, applied by optimize), and nothing here filters by topic at all. Unanimous asks are trim_unanimous_groups’s job and are left alone here; singles always stay. Rows are per-rollout rows with a binary reward, or one row per task carrying pass_rate and n (a trainer’s state, or the calibration stamp); min_k reads n, and a rate row that does not say its n is taken at its word. The report’s from_rates counts the tasks measured from a carried rate rather than rollouts.

trim_unanimous_groups

trim_unanimous_groups(
    rows: Sequence[dict],
    min_k: int = 2,
) -> tuple[list[dict], dict[str, Any]]

Defined in whileai/simulations/score/optimize.py. Drop asks whose k >= min_k rollouts all landed 0 or all landed 1. The basic optimizer from the working decision: trim zeros and ones from tasks, then rerun the simulator and check the variance. Groups smaller than min_k (unique-situation runs) always stay; trimming them would gut an explore dataset, and they carry no group gradient either way.

pairwise

Pairwise judging: which of two replies to the same request is better, asked both ways round.

judge_pairs

judge_pairs(
    pairs: Sequence[dict],
    judge: Callable[[dict, dict], Any] | None = None,
    spec: str | None = None,
    swap: bool = True,
    concurrency: int = 8,
    api_key: str | None = None,
    examples: int = 10,
    position_flip_flag: float = 0.2,
    prefers_rejected_flag: float = 0.2,
) -> tuple[list[dict], dict[str, Any]]

Defined in whileai/simulations/score/pairwise.py. Ask a judge which side of each pair is better, both ways round. judge(a_row, b_row) returns \{"winner": "A" | "B" | "tie", "reason"\} (or that JSON as a string); without one the hosted model judge from pairwise_judge(spec) is used. With swap=True each pair is judged twice with A and B exchanged; a pair the judge decides differently in the two orders is recorded as a tie with position_consistent=False. Writes on each pair (in place, and returned): pairwise with winner ("chosen" | "rejected" | "tie" | None when the judge failed), position_consistent, reasons, judge; and tie (bool). Report: position_flip_rate (position bias: the judge’s answer changed with the order), tie_rate, agrees_with_scores (the pairwise winner is the pointwise chosen), prefers_rejected (the two disagree outright, the rows a person should read), failed. A position_flip_rate at or over position_flip_flag (POSITION_FLIP_FLAG, 0.2: Zheng et al. arXiv:2306.05685 measured 35% of GPT-4 verdicts flipping with the order) and a prefers-rejected share at or over prefers_rejected_flag each add a warning.

pairwise_judge

pairwise_judge(
    spec: str | None = None,
    api_key: str | None = None,
    prompt: str | None = None,
    policy: str = '',
    tools: Sequence | None = None,
    timeout: float = 120,
    max_tokens: int = 120,
    request_chars: int = 4000,
) -> Callable[[dict, dict], Verdict]

Defined in whileai/simulations/score/pairwise.py. A model judge for judge_pairs: judge(a_row, b_row) -> \{"winner": "A" | "B" | "tie" | None, "reason": str\}. spec is a backend spec (default the hosted judge); prompt replaces the pairwise system prompt. The judge’s name is <model>@<prompt sha> so a prompt edit is a new judge. max_tokens is the judge’s reply budget and request_chars how much of the request it is shown.

passat

pass@1, pass^k and pass@k from the same graded groups.

PassAt

class PassAt(
    k: int,
    pass_at_1: float | None,
    pass_pow_k: float | None,
    pass_at_k: float | None,
    n_groups: int,
    n_rows: int,
    n_groups_at_k: int = 0,
    n_groups_imputed: int = 0,
    n_partial: int = 0,
    per_task: dict[str, float] = <factory>,
    note: str = '',
    config: dict[str, Any] = <factory>,
    ci95: tuple[float, float] | None = None,
    pass_pow_k_ci95: tuple[float, float] | None = None,
    pass_at_k_ci95: tuple[float, float] | None = None,
) -> None

Defined in whileai/simulations/score/passat.py. pass@1 / pass^k / pass@k over graded groups. See module docstring. Every field, and the name it prints as in str(...). The printed line and the attribute are not spelled the same: pass^k is pass_pow_k (not pass_hat_k), and the print says both once so the attribute is readable off it.

attribute	prints as	what it is
`k`	`k=4`	draw size the k-way numbers used
`pass_at_1`	`pass@1`	mean per-task pass rate, the headline
`pass_pow_k`	`pass^4`	chance all k repeats pass (reliability)
`pass_at_k`	`pass@4`	chance at least one of k passes
`headroom`	`headroom`	property: pass@k minus pass@1
`ci95`	`[lo..hi]`	interval on pass@1; `None` under 3 groups, reason in `note`
`pass_pow_k_ci95`	`[lo..hi]`	same for pass^k; `None` under 3 groups
`pass_at_k_ci95`	`[lo..hi]`	same for pass@k; `None` under 3 groups
`n_groups`	`N groups`	tasks pass@1 averaged over
`n_rows`	not printed	graded rows behind those tasks
`n_groups_at_k`	not printed	tasks the k-way numbers used
`n_groups_imputed`	not printed	short unanimous tasks counted in
`n_partial`	in the note	rows whose reward is not 0 or 1, left out
`per_task`	not printed	`{task key: pass rate}`, a dict
`note`	tail of the line	why a number is missing, and the fix
`config`	token-cap share	how the rows were made (`run_config`)

to_dict() uses these same keys, with headroom added and the intervals as lists.

pass_at

pass_at(
    rows: Sequence[dict] | Any,
    k: int | None = None,
    min_k: int = 4,
    unanimous_short: bool = False,
) -> PassAt

Defined in whileai/simulations/score/passat.py. Compute pass@1, pass^k and pass@k from graded rows, grouped by task. Reach for it after grading a mode="rl" run to read the three numbers a task family gives: pass@1 is the mean per-task pass rate (the headline), pass^k the chance all k repeats pass (reliability), pass@k the chance at least one of k passes; headroom is pass@k minus pass@1. It returns a PassAt with those three, their task-bootstrap ci95 intervals, n_groups, n_rows, per_task (a dict keyed by task), a note when a number is missing and why, and config (how the rows were made); str(result) prints the line and to_dict() gives the keys. A task is a situation, not a string. Rows group under task_key: the engine’s scenario_id when the row has one, else task_id, else the prompt text. In mode="rl" the repeats of one opener share a scenario_id, and so do the textured phrasings of one situation, so those phrasings pool into one task on purpose: the question is whether the agent handles the situation, not one wording of it. compare_runs, delta_report, eval_variance, curriculum and group_signal count tasks with the same key, so pass_at(rows).n_groups and delta_report(...)["n_paired_tasks"] agree on the same rows. Only binary reward (or qwen_reward) rows count, the same rule group_signal uses; a row whose reward is between 0 and 1 is counted in n_partial and named in note, and an unjudged row is skipped. The intervals resample tasks, never rows (Miller 2024, arXiv:2411.00640), so they need at least MIN_CI_TASKS (3) tasks. Under that, ci95 is None and the note says why and what to change. Ten rows that all carry one task_id are one task, not ten, and get no interval; when they are ten separate items, give each its own task_id.

rows: graded rows, or the SimulationData holding them.
k: the draw size for the k-way numbers. It defaults to the smallest group of two or more repeats, so every such group contributes; groups with fewer than k graded repeats are left out of pass^k and pass@k and counted in n_groups_at_k. pass@1 always averages every group.
min_k: 4 (ROLLOUTS_PER_TASK), the smallest k tau-bench and tau2-bench report a pass^k on (arXiv:2406.12045 and arXiv:2506.07982). Below it the k-way numbers are None with a note instead of a number too noisy to act on.
unanimous_short: True counts a unanimous group shorter than k as if it stayed unanimous (pass^k and pass@k equal to its pass rate, 1 or 0). That is the assumption a successive-allocation run stopped on, and leaving those groups out would score only the tasks that split and inflate the headroom. Mixed short groups still stay out.

.config says how the rows were produced (run_config): task count, k, temperature, max_tokens, policy and judge versions, prompt hash, with a mixed list naming any the rows disagree on.

>>> import whileai.simulations as wai
>>> rows = [{"task_id": t, "reward": r} for t in "abcd" for r in (1, 1, 0, 1)]
>>> wai.pass_at(rows).pass_at_1
0.75

preflight

Inspect the agent before spending simulation budget, and report after.

classify_failure

classify_failure(row: dict) -> str | None

Defined in whileai/simulations/score/preflight.py. Fixed-vocabulary class for a failing row, from its reason and shape. Returns None for passing or unlabeled rows and for failures the heuristics cannot place (leave those for a person, do not guess).

coverage_gap

coverage_gap(
    asks: Any,
    tools: Sequence[dict],
    system_prompt: str = '',
    rows: Sequence[dict] | None = None,
    rule_cap: int | None = None,
) -> dict[str, Any]

Defined in whileai/simulations/score/preflight.py. List the parts of an agent’s policy that the asks you already send never reach. Reach for it before writing situations, with the test suite you already have: it says which tools and which policy rules no ask exercises, in the engine’s own vocabulary. It returns a dict: untested_rules and untested_tools (the lists worth reading), rules (the policy clauses found), axes (each axis with the count per value), stances, pressure_asks, single_shot, per_ask (where each ask landed), notes, summary and n_asks. format_coverage_gap(report) prints it.

asks: what a suite asks the agent: a list of prompt strings, a list of rows carrying prompt, or a path to a .py or .jsonl file holding either.
tools and system_prompt: the agent’s tool schemas and policy. The axes come from build_dimensions, the same grid simulate covers: which tool, which policy rule, what stance the person takes, what the world looks like, what condition the tool is in, what happened before.
rows: graded rollouts from a run. With them the report also checks the world side: rules whose rows all ended in the same tool fault are rules the asks reach but the fixtures never let happen (rules_the_world_never_triggers, with rows_per_rule and rules_with_no_rows).

Each ask is placed on the axes it touches with text heuristics, not a model: the tools its words name or imply, the rule clauses it shares words with, and the stance its words show. Two axes (world_state, tool_condition) cannot be read from an ask at all: a prompt never says the order is missing or the tool timed out, so a hand-written suite leaves them at one point and notes says so. With rows (graded rollouts from a run) the report also checks the world side: rules whose rows all ended in the same tool fault are rules the asks reach but the fixtures never let happen. The rule axis is every clause of the policy (rule_cap=None, the default RULE_AXIS_CAP_REPORT): a report over an existing suite has no grid to bound. A number keeps the first that many clauses in document order; n_rules_total and rules_truncated say what was left off and notes carries the count (#391).

gap = wai.coverage_gap(["Where is order 4473?", "Cancel order 9911."],
                       tools=TOOLS, system_prompt=POLICY)
print(gap["untested_rules"], gap["untested_tools"])

dataset_report

dataset_report(
    rows: Sequence[dict],
    tools: Sequence[dict] | None = None,
    system_prompt: str = '',
    hard_share_floor: float = 0.3,
) -> dict[str, Any]

Defined in whileai/simulations/score/preflight.py. One report a developer reads after simulate/grade: size, signal, mix. hard_share_floor (HARD_SHARE_FLOOR, 0.3) is the share of hard- tier rows under which the set is called easy.

format_coverage_gap

format_coverage_gap(report: dict[str, Any]) -> str

Defined in whileai/simulations/score/preflight.py. The gap report as the block a person actually reads.

preflight

preflight(
    tools: Sequence[dict],
    system_prompt: str = '',
    rule_cap: int | None = None,
) -> dict[str, Any]

Defined in whileai/simulations/score/preflight.py. Spec-quality report for an agent. Report only; nothing is changed. warnings is the list a developer should read before generating thousands of rows; cells is the covering-grid size the same way recommend counts it. rules is every clause of the policy (rule_cap=None, the default RULE_AXIS_CAP_REPORT); a number keeps the first that many in document order, n_rules_total says how many the policy has, rules_truncated whether any were left off, and a warnings line names the count (#391). The generation grid keeps its own cap (RULE_AXIS_CAP_GRID); cells is counted on the same axis rules shows.

privileged

Did the agent say what only the teacher was told?

format_leak_report

format_leak_report(report: dict[str, Any]) -> str

Defined in whileai/simulations/score/privileged.py. One line per fact, the summary first.

leak_report

leak_report(rows: Any, min_len: int = 12) -> LeakReport

Defined in whileai/simulations/score/privileged.py. Which rows quote their own privileged block in the agent’s text. Takes the SimulationData itself, data.trajectories, or any list of rows. Given the data object it reads the trajectories, which still carry the block; data.rows() is the scrubbed export and checks nothing (the report says so). Checks every row that carries privileged (reference, principle, and every string in hidden_state at least min_len characters long) against the final reply and every assistant turn. Returns n_rows, n_checked, n_leaked, rate (over checked rows), checked (False when no row carried the block, so the result is vacuous), leaked (up to 20 rows: scenario_id, rollout_index, field, needle) and summary. Does not mutate rows. Pass data.trajectories, not data.rows(): the export scrubs privileged at any depth, so exported rows carry nothing to check and the report is vacuous. When it can tell the rows came through the export, summary says so and names the accessor to use instead.

publish_gate

The gate a dataset passes before it leaves for the platform.

PublishGateError

Defined in whileai/simulations/score/publish_gate.py. The dataset must not be published as it stands. The message says why.

calibrate

calibrate(
    rows: Sequence[dict],
    policy: PolicyRef | dict | str | None = None,
    model: str | None = None,
    ref: str | Sequence[dict] | None = None,
) -> dict[str, Any]

Defined in whileai/simulations/score/publish_gate.py. Stamp calibration on every graded row, in place. The per-task pass rate is over the binary rewards grouped by prompt, the same grouping group_signal and pass_at use. Rows without a 0/1 reward are left alone and counted. Returns a report with the number of tasks and rows stamped plus the pass_at summary. ref (a key holding the reference model’s summed logprob, or rows scored under it) fills mean_kl per task from the captured logprobs; see mean_kl. A row whose carried stamp counts more repeats than these rows hold keeps it (n_carried in the report): optimize(mode="rl") drops duplicate trajectories, and recomputing here would report the post-dedup k as the policy’s pass rate over k repeats. The producing policy and mean_kl are still filled in from this call.

publish_gate

publish_gate(
    rows: Sequence[dict],
    mode: str | None = None,
    band: tuple[float, float] = (0.2, 0.8),
    policy: PolicyRef | dict | str | None = None,
    model: str | None = None,
    strict: bool = True,
    endorsed: Sequence[str] = (),
    strict_hacks: bool = False,
) -> dict[str, Any]

Defined in whileai/simulations/score/publish_gate.py. Check, calibrate, and report. Raises PublishGateError when strict and the rows are RL-shaped but ungraded or carry no mixed group, or when strict_hacks and hack_scan (with endorsed naming what the reward should track) finds the reward best explained by something else. Never mutates anything except the calibration stamp. judge_trust in the report is the summary grade stamped on the rows when they carried human labels, else None.

quality

Second-pass conversation quality ranker. Scores rows; does not rewrite them.

rank_rows

rank_rows(rows: Sequence[dict]) -> list[dict]

Defined in whileai/simulations/score/quality.py. Score each row in place. Returns the same list when given a list.

score_row

score_row(row: dict) -> dict[str, Any]

Defined in whileai/simulations/score/quality.py. Score one row. Returns quality, quality_reason, quality_scores. No mutate.

reference

Score rollouts under a reference model, so mean_kl has its other side.

reference_logprobs

reference_logprobs(
    source,
    ref: str,
    system_prompt: str | None = None,
    tools: Sequence[dict] | None = None,
    api_key: str | None = None,
    concurrency: int = 4,
    timeout: float = 600.0,
    chat_template_kwargs: Mapping[str, Any] | None = None,
    transport: Callable[[dict], dict] | None = None,
) -> dict[str, Any]

Defined in whileai/simulations/score/reference.py. Stamp ref_logprob on every row: the reference’s summed logprob over the tokens the policy generated. Rows are modified in place; the report says what was scored. ref is a backend spec, vllm:<model>@<base_url>; on the platform’s serving endpoint <model> is the base by its own name (Qwen/Qwen3-4B: the reference of an SFT/GRPO/DPO run), a hosted model’s name, or run:<runId> for a finished run’s adapter, with WHILEAI_API_KEY as the key. source is a SimulationData (system prompt and tools come from its profile), a row list, or a JSONL path; pass system_prompt=/tools= for the last two so the reference sees the prompt the policy saw. chat_template_kwargs must match what the policy sampled with (\{"enable_thinking": False\} for Qwen3). Report: n_rows, n_skipped (no assistant turn or a failed call, with errors), n_tokens, model, and token_count_gap (mean |ref_n_tokens - n_tokens| over rows that carry n_tokens): near zero when the reference shares the policy’s tokenizer, which is when mean_kl is a KL and not a length artifact.

rubric

Rubrics: prompt-specific criteria as an object, a judge that scores them one by one, and a writer that drafts them.

Criterion

class Criterion(
    title: str,
    description: str = '',
    weight: float = 1.0,
    kind: str = 'principle',
) -> None

Defined in whileai/simulations/score/rubric.py. One rubric item. weight is a positive magnitude; a pitfall subtracts it when the reply exhibits the mistake, a principle adds it when met, and a hard rule missed fails the whole reply.

Rubric

class Rubric(
    criteria: tuple[Criterion, ...],
    source: str = 'hand',
    domain: str = '',
    notes: str = '',
    _cache: dict = <factory>,
) -> None

Defined in whileai/simulations/score/rubric.py.

Rubric.checklist

checklist(self) -> str

The rubric as the judge reads it.

Rubric.score

score(self, results: Mapping[str, Any]) -> dict[str, Any]

Reward from one verdict per criterion (keyed by title or slug; a truthy value means the reply meets a hard rule or principle, or exhibits a pitfall). A missed hard rule is a 0. Otherwise the reward is the met principle weight minus the exhibited pitfall weight, over the total principle weight, clamped to [0, 1]; with no principle the reward is 1 minus the pitfall share. Criteria the judge did not answer count as not met (and not exhibited) and are listed.

attach_rubric

attach_rubric(
    rows: Sequence[dict],
    rubric: Rubric | Mapping[str, Any] | Sequence[Mapping[str, Any]] | Callable[[dict], Any],
    overwrite: bool = True,
) -> list[dict]

Defined in whileai/simulations/score/rubric.py. Put a rubric on each row’s privileged block (in place). rubric is a Rubric, its dict / list form, or row -> Rubric | None for a per-prompt rubric; None leaves that row alone.

rubric_judge

rubric_judge(
    rubric: Rubric | None = None,
    spec: str | None = None,
    api_key: str | None = None,
    prompt: str | None = None,
    policy: str = '',
    tools: Sequence | None = None,
    timeout: float = 120,
) -> Callable[[dict], dict[str, Any]]

Defined in whileai/simulations/score/rubric.py. A judge for run_judge / data.grade(judge=) that scores the rubric item by item. rubric applies to every row; without one the row’s own privileged.rubric is used and a row with none stays ungraded. The result carries reward (Rubric.score), reason, markers (rubric:<slug> = 1.0 met / 0.0 not, and for a pitfall 1.0 clean / 0.0 exhibited), criteria (the raw verdicts), rubric_version and the score breakdown. The judge’s name folds the rubric version in when one is fixed. Two things about the verdict worth knowing before it is trusted. A rubric of plain principles scores the mean of its criteria, so three principles return 0, 1/3, 2/3 or 1, and judge_agreement / judge_trust count exact 0/1 rewards only: every partially met row is skipped, and the agreement number is read off the rows the judge was sure about (#345). Give each Criterion kind="hard" for a 0/1 verdict (Rubric.score), or accept that judge_trust reports the skipped share and pulls ok when it passes MAX_SKIPPED_SHARE. And whether a tool was called is handed to the judge as a fact, not left for it to infer: the payload carries tools_called (steps that returned a result) and tools_not_called, and the system prompt says a reply that announces a call it never made has not made it (#346: without the list, a 4B judge passed 18 of 18 announced-but-never-made escalations). A criterion that must be exact belongs in a grader= that reads steps itself. The hosted judge scales to zero, so the first row through warms it once (warm_judge, a 600s budget) while the rest of the fan-out waits. Without that, run_judge’s eight concurrent calls all raced a container that was still loading its weights and every row came back invalid_result with a TimeoutError. Warm-up failure is not fatal: the rows are judged anyway and report the real error.

rubric_of

rubric_of(row: Mapping[str, Any]) -> Rubric | None

Defined in whileai/simulations/score/rubric.py.

write_rubrics

write_rubrics(
    rows: Sequence[dict],
    spec: str | None = None,
    domain: str = '',
    n_criteria: tuple[int, int] = (3, 8),
    reference_key: str = 'reference',
    overwrite: bool = False,
    concurrency: int = 8,
    api_key: str | None = None,
    timeout: float = 120,
    writer: Callable[[str], Any] | None = None,
    max_hard: int | None = None,
) -> tuple[list[dict], dict[str, Any]]

Defined in whileai/simulations/score/rubric.py. Draft one rubric per distinct prompt with a model and attach it to every row of that prompt (privileged.rubric, source="model"). The writer sees the request, the row’s reference answer when there is one (privileged.reference or row[reference_key]), and the domain guidance you give it (the general rubric Lambert 2025 seeds from). Rows that already carry a rubric are skipped unless overwrite. writer(user_message) -> str replaces the model call for tests and for a writer of your own. Report: prompts seen, rubrics written, failures, mean criteria per rubric, the rubric versions. max_hard caps the hard rules a written rubric may carry: the heaviest max_hard stay hard and the rest become principles with their weight (demoted_hard in the report). A model writer marks most of what it wants as Essential, and every Essential item a reply misses is a 0, so an uncapped rubric fails rows a binary judge passes (measured live: 22 of 32 rows). None keeps what the writer wrote.

spec

Model spec as a versioned object (Lambert 2025, chapter Model Character and Products).

Spec

class Spec(
    id: str,
    traits: tuple[Trait, ...] = (),
    source: dict = <factory>,
    version: str = '',
) -> None

Defined in whileai/simulations/score/spec.py. A versioned model spec. version is derived from the content when left empty, so it is stable across processes and changes on any edit.

Spec.behaviors

behaviors(self) -> list[str]

Trait ids, in order. Hand to delta_report(must_not_regress=...).

Trait

class Trait(
    id: str,
    name: str = '',
    principle: str = '',
    authority: str = 'should',
    examples: tuple[Any, ...] = (),
) -> None

Defined in whileai/simulations/score/spec.py. One named expectation. authority is the strength (must / should / may, following the model-spec convention).

load_spec

load_spec(obj: Any, spec_id: str | None = None) -> Spec

Defined in whileai/simulations/score/spec.py. Build a Spec from a constitution dict, a list of traits, or a path to a JSON file with either shape. A trait may be a full dict or a bare principle string.

spec_version

spec_version(spec: Spec | dict | Any) -> str

Defined in whileai/simulations/score/spec.py. The content version of a Spec (or anything load_spec accepts).

stamp_spec

stamp_spec(rows: Sequence[dict], spec: Spec) -> list[dict]

Defined in whileai/simulations/score/spec.py. Return copies of rows tagged with the spec they were produced or graded against: spec_id and spec_version. Provenance for the ch. 17 retention question — did adherence hold from one spec version, or model version, to the next.

stage

Stage lineage: which post-training stage consumed each row (Lambert 2025, chapter Training Overview).

format_stages

format_stages(report: dict[str, Any]) -> str

Defined in whileai/simulations/score/stage.py.

stage_of

stage_of(row: dict) -> str | None

Defined in whileai/simulations/score/stage.py. The stamped stage, or None. An eval-sourced row with no stamp reads as eval (its lineage.source), so a held-out set is never mistaken for training data just because no one stamped it.

stage_report

stage_report(rows: Sequence[dict]) -> dict[str, Any]

Defined in whileai/simulations/score/stage.py. Rows per stage, tasks per stage, and the cross-stage leaks: any task used both in eval and in a training stage (sft/rm/rl/mid). That leak means the number you report was optimized against.

stamp_stage

stamp_stage(rows: Sequence[dict], stage: str) -> list[dict]

Defined in whileai/simulations/score/stage.py. Return copies of rows with row["stage"] = stage. stage must be one of STAGES; nothing else is touched.

stats

Confidence intervals, paired run comparison, and decontamination.

compare_runs

compare_runs(
    a: Sequence[dict],
    b: Sequence[dict],
    metric: str = 'pass_at_1',
    n_boot: int = 2000,
    seed: int = 0,
    min_paired: int = 5,
    level: float = 0.95,
) -> dict[str, Any]

Defined in whileai/simulations/score/stats.py. Test whether run b differs from run a on one metric, paired by task. Reach for it for a quick A/B on a single number; delta_report is the full report with markers, the noise floor and the comparability checks. It returns a dict: delta (b minus a), ci95 (the interval, with level beside it), p_value, verdict, n_paired, n_only_a, n_only_b, paired_share, mean_a, mean_b, and a note. Tasks the two runs share are compared as paired differences (b minus a, per task, keyed the way pass_at groups); the interval is a level bootstrap over those pairs and the p-value is a sign-flip permutation test. verdict is one of "b_better", "a_better", "no_difference_detected": the last means the interval covers zero, not that the runs are equal. Tasks on one side only are dropped from a paired comparison, and note says how many, since a verdict over a quarter of the tasks is not a verdict over the eval. paired_share is the shared fraction of every task either run saw.

metric: "pass_at_1" (the default, binary reward) or "marker:name" for a marker.
min_paired (5): with fewer shared tasks the comparison falls back to unpaired task means and says so.
level (0.95): the interval’s coverage (ci95 at the default). n_boot (2000) and seed (0) fix the bootstrap.

>>> a = [{"task_id": t, "reward": 0} for t in "abcdef"]
>>> b = [{"task_id": t, "reward": 1} for t in "abcdef"]
>>> wai.compare_runs(a, b)["verdict"]
'b_better'

decontaminate

decontaminate(
    rows: Sequence[dict],
    against: Sequence[Any] | Any,
    n: int = 8,
    fields: Sequence[str] = ('prompt',),
    overlap: float = 0.8,
    embedder: Callable[[list[str]], Sequence[Sequence[float]]] | None = None,
    similarity: float = 0.85,
) -> tuple[list[dict], dict[str, Any]]

Defined in whileai/simulations/score/stats.py. Drop training rows whose prompt overlaps an evaluation set. Reach for it before any train-versus-holdout comparison: a held-out task that also sits in the training data measures memory, not the change (Lambert 2025, chapter Evaluation). It returns (clean_rows, report): the rows that survived (a list that also carries the system prompt and tools the input carried, so select(clean_rows).export() writes them), and a report with the count under each rule (n_contaminated in total), hits per field, the eval text count, and the first offenders with their coverage (or similarity for semantic hits). rules_skipped names each rule that could not run on these inputs and why (empty when every rule ran), and notes says it in a sentence: a zero under a rule that never ran is not a clearance.

rows: the training rows.
against: one or more evaluation sources: row lists, JSONL paths, or platform dataset ids (ds_...). Evaluation prompts, answers and references are the texts compared (not the eval set’s own replies).
fields (("prompt",)): which row texts are checked; prompts only is what Lambert 2025, chapter Evaluation, checks. Add "final_text" to ask the stricter question of whether replies reproduce eval answers or references.
n (8) and overlap (0.8): the near-copy rule, the Llama 2 rule of 8-grams covering 80% of tokens. overlap=0 restores any-n-gram.
embedder and similarity (0.85): a callable from a list of texts to one vector per text turns on the semantic rule at that cosine threshold; nothing here imports a model.

Four rules, applied in this order, and a row flagged by one is not counted again by the next, so n_contaminated is the number of rows dropped:

same_task (n_same_task): the row’s scenario_id or task_id is an evaluation row’s. A task is a situation, not a string (task_key), so a rephrasing of an eval situation is the eval situation whatever the words say. It needs an id on both sides: when no evaluation row (or no training row) carries one, the rule does not run, rules_skipped["same_task"] says so, and only the text rules stand between the sets. Every eval set not written by simulate() (GSM8K, a Hub set, logged traces) is in that case, so read n_same_task: 0 next to rules_skipped.
exact (n_exact): one of the row’s fields is an evaluation text verbatim after normalization (case and whitespace).
near copy (n_near): one evaluation text covers at least overlap of the row’s words with shared word n-grams. Texts shorter than n words match verbatim only.
semantic (n_semantic), only with embedder: the cosine similarity between the row’s text and an evaluation prompt is at least similarity, and the two carry different task ids or none. It needs evaluation prompts to embed: when no evaluation row has a prompt, the rule does not run, rules_skipped["semantic"] says so, and a UserWarning is raised because you asked for it.

One shared n-gram is the test Lambert 2025, chapter Evaluation, uses for free-form sets. Situations written from templates share whole sentences that say nothing about which question was asked, so any-n-gram flags every row of a template-written set; the coverage rule counts a row when one eval text accounts for most of it. Word overlap does not see a paraphrase. A holdout written by re-running the generator on the same briefs was 70% within 0.85 cosine of the training batch and 5 of 133 byte-identical; the 8-gram rule flagged 4 of 101 prompts and the semantic pass 16. With sentence-transformers:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-small-en-v1.5")
clean, report = wai.decontaminate(
    train,
    against=[holdout],
    embedder=lambda texts: model.encode(texts, normalize_embeddings=True).tolist(),
)

A semantic flag means the two prompts read alike, not that they are the same task: “cancel one reservation” and “cancel three reservations” for different customers scored 0.932 with no shared answer. So where task identity is recorded the same_task rule decides and the semantic pass only looks across different tasks, and the report’s notes say the flag is a question to check, not a verdict. The default stays lexical: similarity 0.85 was read off BGE (unrelated prompts score about 0.55 there) and does not transfer to every model, so the pass calibrates it for yours when it can. With eval rows that carry task ids, the 99th percentile of similarity over eval-prompt pairs with different task ids is how alike distinct tasks read to this embedder, and notes says it; a threshold below that number flags tasks that merely share a domain, and the note says so when similarity is.

>>> train = [{"prompt": "Where is order 4473?"}, {"prompt": "Cancel order 9911."}]
>>> clean, report = wai.decontaminate(train, against=[[{"prompt": "Cancel order 9911."}]])
>>> len(clean), report["n_contaminated"]
(1, 1)

detectable_effect

detectable_effect(
    n_tasks: int,
    base: float = 0.6,
    k: int = 4,
    power: float = 0.8,
    alpha: float = 0.05,
) -> float | None

Defined in whileai/simulations/score/stats.py. The smallest gain n_tasks paired tasks can prove at power: holdout_size solved for the effect (FIXED_POINT_STEPS fixed-point steps, since the after-side variance depends on it). None below MIN_HOLDOUT_TASKS tasks.

eval_variance

eval_variance(
    *runs: Sequence[dict],
    metric: str = 'pass_at_1',
    by: str | None = None,
) -> EvalVarianceReport

Defined in whileai/simulations/score/stats.py. How much an evaluation moves when the same model is evaluated again (Lambert 2025, chapter Evaluation). Pass each re-run’s rows as its own argument, or one row list whose rows say which run they belong to: lineage.eval_run (what simulate(runs=3) stamps), else lineage.scoring_run_id (what evaluate(run_id=) stamps), or a top-level or lineage key named by by. Each run’s metric is a mean over tasks; the report is those means, their mean, the sample standard deviation run_std, and noise_band = noise_band(run_std, df=n_runs - 1): the two-sided t quantile at noise_band_df = n_runs - 1 times sqrt(2) times run_std, because a before/after delta with one run per side is the difference of two re-run draws and run_std is an estimate from these very runs, not the eval’s exact spread (Lambert 2025, chapter Evaluation). This is the band compare(run_std=, run_std_runs=) applies; with three runs the multiplier is 4.30, not 1.96 (the 1.96 band read a three-run estimate as exact and let about one pure-noise delta in five through, #616). A delta inside the band is what re-running the eval does on its own. run_std_by_metric reports the same floor for pass@1 and every marker shared by all runs; hand that mapping to delta_report(run_std=) so each metric uses its own re-run variance. The scalar run_std remains the selected metric’s value for callers comparing only one metric. stability places run_std on Olmo 3’s bands in points. Fewer than three runs is a difference, not a distribution; the report says so and run_std is None below two.

holdout_size

holdout_size(
    effect: float,
    base: float = 0.6,
    k: int = 4,
    power: float = 0.8,
    alpha: float = 0.05,
    before: Sequence[dict] | None = None,
    after: Sequence[dict] | None = None,
    task_std: float | None = None,
    rows: Sequence[dict] | None = None,
    ceiling_pass_rate: float = 0.9,
) -> HoldoutSizeReport

Defined in whileai/simulations/score/stats.py. How many paired tasks a holdout needs to prove a gain of effect. Models the test delta_report runs: each task’s pass rate over k rollouts on each side, the delta as the mean of the paired differences, the interval from a bootstrap over tasks. The usual two-sided power calculation then gives n = ((z_\{1-alpha/2\} + z_power) * sd / effect) ** 2 with sd the standard deviation of one task’s paired difference (Lambert 2025, chapter Evaluation: the point of a better eval is statistical power when comparing training runs). Where sd comes from is the whole question, and there are three ways to answer it, best first:

before and after, the graded arms of a previous eval on the same tasks (the two row lists delta_report(before, after) takes): sd is measured as the sample sd of the per-task differences, which carries the covariance that pairing buys and whatever shape the gain had. No model. sd_source is "rows" and n_paired says how many tasks it was read off.
task_std, a number you measured (the per-task sibling of delta_report’s run_std): the same quantity read off a previous delta_report: (hi - lo) * sqrt(n_paired_tasks) / 3.92 from target_ci95 and n_paired_tasks (or any metrics[...]["ci95"] with its n_paired). Agent rubrics sat near 0.38 across five lanes (#288). sd_source is "given". eval_variance’s run_std is a different number (how much a re-run moves the mean) and is not this.
Neither: the binomial model sqrt((p(1-p) + q(1-q)) / k) with p = base and q = base + effect, sd_source "model". It assumes two things it cannot check: that the gain is spread evenly across tasks, and that the two arms are independent draws (Var(A) + Var(B), no covariance term). When the gain is carried by a few tasks, most tasks are ties and the paired differences spread far wider than binomial-per-task predicts; a voice trait at 0 -> 0.127, k=4, carried by 19 of 150 tasks, measured sd 0.333 against the model’s 0.168 and needed 54 tasks where the model said 14 (#292). So the model path also returns n_tasks_concentrated, the count if the gain were carried by the fewest tasks that can carry it (each going from base to 1), and notes says which assumption is in play. On a holdout whose tasks differ in difficulty the independence assumption errs the other way: the model puts p(1-p) of variance on every task where pairing keeps each task’s own p_i(1-p_i), whose mean is p(1-p) - Var(p_i), so it asks for 1 / (1 - Var(p_i) / (p(1-p))) times the tasks pairing needs (1.19x at spread 0.2 around 0.5, 2.78x at 0.4). before alone reports the spread as base_spread and puts that ratio in notes.

before on its own (rows is the same argument under its old name) reads base and k off the data. Returns n_tasks plus the inputs, task_std, sd_source, half_width (the 95% band on the delta at that n), n_tasks_concentrated, base_spread, n_paired, saturated, notes and warnings; every key is present on every path (None, False or [] where it does not apply). The default answer is unchanged; the honest paths are the two that measure. A saturated base= cannot size anything (base is the before arm’s pass rate; there is no baseline=). Rows whose tasks all pass give p = 1, the binomial variance p(1-p) is 0, and both arms all passing give a measured paired sd of 0; the formula then returns the floor, MIN_HOLDOUT_TASKS, which is the model collapsing, not evidence that two tasks are enough (#392). When the measured base is at or above ceiling_pass_rate (CEILING_PASS_RATE, the share delta_report flags as ceiling) or the measured sd is 0 (the paired difference identical on every task, DEGENERATE), the rows are not used: n_tasks is the binomial model’s answer at BASE_PASS_RATE and the rows’ k, sd_source is "model", saturated is True, and warnings names the ceiling and the fix: harder situations, so base sits inside the 20-80 difficulty band (Lambert 2025, chapter Reasoning; DAPO, arXiv 2503.14476, drops prompts at accuracy 0 and 1 because they carry no signal), then size again on those rows. The recipe that asked for this had 140 tasks at k=4 around 0.6: a band of about +-0.06, so a real 3-point gain reads no_change_detected every round. This says so before training.

marker_names

marker_names(rows: Sequence[dict]) -> list[str]

Defined in whileai/simulations/score/stats.py.

marker_summary

marker_summary(
    rows: Sequence[dict],
    names: Sequence[str] | None = None,
    n_boot: int = 2000,
    seed: int = 0,
) -> dict[str, dict[str, Any]]

Defined in whileai/simulations/score/stats.py. metric_summary for every marker on the rows (or names). Each marker’s stats are keyed mean, ci95 (not ci), n_tasks, n_rows (not n), n_rows_at_1, n_rows_at_0, degenerate, and note or warning when there is one. ci95 is None below MIN_CI_TASKS tasks, and note then says how many tasks the marker has and how many the interval needs; a reader who sees only None cannot tell that from a bug.

metric_summary

metric_summary(
    rows: Sequence[dict],
    metric: str = 'pass_at_1',
    n_boot: int = 2000,
    seed: int = 0,
) -> dict[str, Any]

Defined in whileai/simulations/score/stats.py. Mean over tasks with a task-bootstrap 95% interval. degenerate is set when every applicable row scored the same value: the metric has not been shown to be able to come out any other way, so ci95 is None (the way pass_at returns None below three groups) and warning says so. A marker that is silently unfireable (a key-name mismatch) and one that is genuinely always true look identical otherwise, and either one passed to must_not_regress is a guard that cannot fail (#270). n_rows_at_1 and n_rows_at_0 put the row-level split next to the mean.

task_key

task_key(row: dict) -> str

Defined in whileai/simulations/score/stats.py. The one name every report groups a row’s rollouts under. A task is a situation, not a string: scenario_id when the row has one (the engine’s situation id, shared by the repeats of one opener and by the textured phrasings of one situation), else task_id (rows from elsewhere), else the prompt text. pass_at, compare_runs, delta_report, eval_variance, curriculum, group_signal and the exporters all count tasks with this key, so the same rows give the same task count everywhere (Miller 2024, arXiv:2411.00640: intervals and paired comparisons are over tasks, never rows).

style

Over-optimization signatures on replies: the things a reward pays for by accident.

refusal_report

refusal_report(
    rows: Sequence[dict],
    phrases: Sequence[str] = ("i can't help with", 'i cannot help with', "i can't assist", 'i cannot assist', "i'm unable to", 'i am unable to', "i won't be able to", "i'm not able to", 'i am not able to', "i can't do that", 'i cannot do that'),
    examples: int = 5,
) -> dict[str, Any]

Defined in whileai/simulations/score/style.py. Over-refusal on a benign set (Lambert 2025, chapter Over-optimization, “Over-Refusal”). Pass the rows whose asks the agent should have answered; the report is the share it refused anyway, with a Wilson 95% interval, the phrases that fired, and the first few refusals so a person can read them. Refusal rate on a mixed set means nothing, which is why this takes the benign rows rather than finding them.

style_markers

style_markers(
    rows: Sequence[dict],
    phrases: Mapping[str, Sequence[str]] | None = None,
) -> list[dict]

Defined in whileai/simulations/score/style.py. Stamp the style markers on every row’s markers (in place) and return the rows. phrases overrides or extends STYLE_MARKERS: {"no_boilerplate": [...], "no_brand_voice": [...]}. Existing markers with other names are kept.

style_report

style_report(
    rows: Sequence[dict],
    phrases: Mapping[str, Sequence[str]] | None = None,
    threshold: float = 0.3,
    n_boot: int = 2000,
    seed: int = 0,
) -> StyleReport

Defined in whileai/simulations/score/style.py. How much of each signature the replies carry, and whether the reward pays for it. Does not mutate rows. Per marker: clean (share of rows without a hit, with a task-bootstrap 95% interval), hits (rows with a hit), top_phrases (the phrases that fired, most common first) and reward_corr (Pearson between “phrase present” and the binary reward over graded rows). A positive correlation at or above threshold is flagged: the judge is rewarding the tic, and a policy trained on these rewards will produce more of it (Gao et al. 2022, arXiv:2210.10760). warnings says so in one line per flag. print the report; it is a dict, so every key still reads. A marker that came out the same on every row is degenerate: it has no interval, the line says so next to the mean, and one notes entry names every such marker and the fix. A phrase list that matches nothing looks exactly like a behavior that never happened, and either one in must_not_regress= is a guard that cannot fail (#270). warnings stays what it was, the reward-pays-for-a-tic flags and nothing else. This report stamps the phrase signatures and nothing else, so it says what it did not stamp: not_stamped is {call: [marker, ...]} for the markers trace_markers and mark_grounding write, and the printed report ends with that line. A row clean on every marker here can still have faked a tool call or invented an argument, which #760 measured at 24.2% [22.6%, 25.9%] of the rows this report passed. print(wai.style_report(rows))

style 10 rows, 10 graded

no_boilerplate clean 1.000 hits 0 no interval: constant, see the warning below

no_hedging clean 0.500 [0.200..0.700] hits 5 corr +1.00 flagged “it depends” 5

…

not stamped here: 8 markers in other families. trace_markers(rows) stamps …

trace

Did the agent fake the work? Flags read from the trajectory, not the prose.

trace_flag_report

trace_flag_report(
    rows: Sequence[dict],
    kinds: Mapping[str, str] | None = None,
    threshold: float = 0.3,
    n_boot: int = 2000,
    seed: int = 0,
    examples: int = 3,
) -> dict[str, Any]

Defined in whileai/simulations/score/trace.py. How often each flag fires, what a reviewer should read, and whether the reward pays for it. Does not mutate rows. Per flag: n (rows it fired on), rate, examples (evidence with the ask), and reward_corr (Pearson between “flag fired” and the binary reward over graded rows); a positive correlation at or over threshold is flagged, since a judge that pays for a faked turn trains a policy to fake turns. Per marker: the clean share with a task-bootstrap interval. warnings says so, one line per flag.

trace_flags

trace_flags(
    row: Mapping[str, Any],
    kinds: Mapping[str, str] | None = None,
) -> dict[str, str]

Defined in whileai/simulations/score/trace.py. The flags that fire on one rollout: {flag: evidence}. First match wins per flag, so the evidence points at the earliest cause.

trace_markers

trace_markers(
    rows: Sequence[dict],
    kinds: Mapping[str, str] | None = None,
    evidence: bool = True,
) -> list[dict]

Defined in whileai/simulations/score/trace.py. Stamp the trace markers on every row’s markers (in place) and return the rows: 1.0 when the family is clean, 0.0 when a flag fired. With evidence the flags and their fragments land on the row as trace_flags for a reviewer.

Last modified on September 22, 2026

​agreement

​judge_agreement

​audit

​audit_grades

​format_audit

​checklist

​expected_outcome

​outcome_check

​privileged_context

​task_checklist

​curriculum

​curriculum

​format_curriculum

​retire_solved

​delta

​delta_report

​format_delta_report

​grading

​behavior_signature

​conduct_grade

​grounding

​argument_grounding

​grounding_report

​mark_grounding

​ungrounded_arguments

​hack_scan

​format_hack_scan

​format_hack_scan_diff

​hack_scan

​hack_scan_diff

​hygiene

​dedupe_groups

​length_report

​near_duplicate_prompts

​reward_correlations

​judge_trust

​format_judge_trust

​judge_probes

​judge_trust

​judging

​ScoredData

​ScoredData.agreement

​ScoredData.compare_judges

​ScoredData.failed_traces

​ScoredData.partials

​ScoredData.push

​ScoredData.select

​ScoredData.select_by_reward_range

​ScoredData.select_for_preference

​ScoredData.select_for_rl

​ScoredData.select_for_sft

​ScoredData.unjudged

​build_preference_pairs

​evaluate

​normalize_judge_result

​run_judge

​labels

​annotator_agreement

​attach_labels

​logprobs

​logprob_report

​mean_kl

​staleness_report

​markers

​behavioral_markers

​format_markers

​mark_rows

​row_markers

​optimize

​filter_rl_rows

​group_signal

​next_round

​optimize

​recommend

​select_for_rl

​select_for_sft

​trim_out_of_band

​trim_unanimous_groups

​pairwise

​judge_pairs

agreement

judge_agreement

audit

audit_grades

format_audit

checklist

expected_outcome

outcome_check

privileged_context

task_checklist

curriculum

curriculum

format_curriculum

retire_solved

delta

delta_report

format_delta_report

grading

behavior_signature

conduct_grade

grounding

argument_grounding

grounding_report

mark_grounding

ungrounded_arguments

hack_scan

format_hack_scan

format_hack_scan_diff

hack_scan

hack_scan_diff

hygiene

dedupe_groups

length_report

near_duplicate_prompts

reward_correlations

judge_trust

format_judge_trust

judge_probes

judge_trust

judging

ScoredData

ScoredData.agreement

ScoredData.compare_judges

ScoredData.failed_traces

ScoredData.partials

ScoredData.push

ScoredData.select

ScoredData.select_by_reward_range

ScoredData.select_for_preference

ScoredData.select_for_rl

ScoredData.select_for_sft

ScoredData.unjudged

build_preference_pairs

evaluate

normalize_judge_result

run_judge

labels

annotator_agreement

attach_labels

logprobs

logprob_report

mean_kl

staleness_report

markers

behavioral_markers

format_markers

mark_rows

row_markers

optimize

filter_rl_rows

group_signal

next_round

optimize

recommend

select_for_rl

select_for_sft

trim_out_of_band

trim_unanimous_groups

pairwise

judge_pairs