6 public names. import whileai.simulations as wai, then wai.name.
SimulationData
Defined in whileai/simulations/data.py.
SimulationData.compare_judges
Grade these rows with several judges and rank them against the gold labels.
judges maps a name to a spec string ("typesafe:jev-latest"),
a backend object, a wai.Judge or any judge callable. Each grades
its own copy of the rows under this run’s system prompt and tools,
then is scored the way judge_trust scores one judge: agreement
with a Wilson interval, kappa, leak rate, unsure and unjudged
counts, seconds per row. Returns a JudgeComparison that prints
as a table ranked by kappa; whileai.judge_comparison.compare_judges
has the full account and takes a bare row list.
SimulationData.grade
Grade this run’s rows in place with the hosted judge or your own callable.
Reach for it right after simulate to score the rows without
leaving the object. Simulation never calls it on its own. Three paths,
chosen by what you pass:
- No callable (or
llm=True): grade_llm, the hosted LLM judge
(Phi-4 unless WHILEAI_JUDGE is set, a different family from the
hosted Qwen policy), read from VLLM_API_KEY. It writes
reward (0 or 1) and reason onto the rows in place and
returns the judge report, a dict with graded, n0, n1,
backend, judge_version and warnings.
grader=, a plain callable returning a number or
{"reward": ..., "reason": ...} per row: scores every row in
place over concurrency threads and returns the run itself, so
data.grade(my_grader).pass_at reads through.judge=, the contract path: any callable honoring the judge
contract, which returns
{"reward": 0 or 1, "reason": str, "markers": {name: value}}
per row (a bare number works too). The contract
and its failure modes are written out in full in
whileai.simulations.score.judging (note the score.; there is
no whileai.simulations.judging). It returns a ScoredData of
copies: the trajectories here stay unmodified, judge errors are
marked per row instead of coerced to 0, and its output feeds
export_dataset and simulate(traces=...) directly.
Arguments that matter:
version: names the judge’s version (model, rubric hash) and is
recorded on every scored row; the hosted grader stamps its own.rubric: what doing the job means, as text, for the hosted judge;
without it the judge grades the conduct floor only, and says so.use_privileged: True shows the hosted judge each row’s
privileged block (principle, reference, hidden state) the agent
never saw.trust: the judge check against the rows’ human labels
(attach_labels(kind="human")), run on every path, with the
summary stamped on each graded row’s judge_meta["trust"].
"warn" (the default) logs one line when the check failed or no
labels exist, "require" raises instead, "off" skips it.path: write the graded run’s JSONL there afterwards.spec: which model judges on the hosted path, as a backend spec
("typesafe:jev-latest", "openai:gpt-4.1-mini"); the same
keyword grade_llm, pairwise_judge and rubric_judge
take. llm_spec is its older name and still works.
SimulationData.grade_llm
Binary 0/1 situation grade. Default brain is the hosted judge
(Phi-4 unless WHILEAI_JUDGE is set), never the policy model.
use_privileged shows the judge each row’s privileged block
(principle, reference, hidden state) the agent never saw. trust
is the judge check against human labels: see grade.
payload_chars caps the evidence the judge reads per row and
max_tokens its reply (defaults JUDGE_PAYLOAD_CHARS and
JUDGE_MAX_TOKENS in defaults.py); both land in
judge_meta.
SimulationData.leak_report
Did any reply quote its own privileged block? Reads the
trajectories, which still carry the block; rows() is scrubbed
and would check nothing. Same report as leak_report.
SimulationData.llm_grade
Advisory LLM pass. Leaves deterministic reward untouched.
SimulationData.push
Upload this run to your While account as a dataset.
purpose is the section it lands in on the Datasets page
("train" by default; "holdout" or "eval").
holdout=0.2 keeps a fifth of the tasks (by scenario_id) out
of the training set and pushes them as a second, linked dataset
with purpose "holdout"; the entry carries it as ["holdout"].
The simulation mode is recorded on both.
api_key defaults to the WHILEAI_API_KEY env var, then the
key saved by wai login. Pass parent (a ds_...
id) when this run iterates on an existing dataset, so lineage shows
on the platform. publish=True with an agent name also puts it
on the public catalog at huggingface.co/while-ai as a card. Returns
the registry entry with datasetId.
gate=True runs publish_gate first: every graded row gets a
calibration stamp (per-task pass rate, k, producing policy),
and an RL-shaped run that is ungraded or has no mixed group is
refused with PublishGateError. The gate report is returned as
entry["gate"]. gate=False uploads rows as they are.
endorsed names what the reward should track (feature-name
substrings, e.g. "tool:lookup_order") for the gate’s
hack_scan; strict_hacks=True refuses a set whose reward
is best explained by something else.
SimulationData.rank
Second-pass quality scores. Leaves conduct reward untouched.
Writes quality, quality_reason, quality_scores on each
trajectory and rewrites the saved JSONL, or path if you pass one.
SimulationData.report
Run-level coverage summary (same as data.coverage).
SimulationData.select
The rows worth training on, as a Selection that prints its report.
With no mode: diverse pass-labeled demonstrations via
select_for_sft, one of each distinct way of being right before
any repeats, junk and duplicate prompts dropped. With
mode="rl" or "sft": optimize, the full gate sequence
(privileged leaks, difficulty band, unanimous groups, duplicates,
truncation, hack scan), with band, endorsed and
truncated as there.
Requires graded rows — grade in-loop (grade=True, grader=)
or afterwards with grade(). The report lands in
search["selection"] and on the result’s .report.
SimulationData.training_set
Select the recommended rows and export them trainer-ready.
select() picks diverse pass-labeled rows, export_training
writes them as chat JSONL with this run’s system prompt and tools
and the tool-call round-trip gate. Returns the export report with
the selection report attached; pass output to write the file.
Raw simulation rows are not the training artifact — this is.
conversation
Defined in whileai/simulations/data.py.
User/agent turns from prompt + steps. Tool calls stay on the assistant turn.
grade
Defined in whileai/simulations/data.py.
Grade rows 0 or 1 with the hosted LLM judge and write a reason beside each.
Reach for it when the rows are a JSONL path or a row list rather than
a SimulationData you hold (that object has the same call as
data.grade()). It writes reward (0 or 1) and a one-sentence
reason on each row, keeps a previous score as qwen_reward
when present, and returns the judge report, a dict with graded,
n0, n1, backend, judge_version, warnings and
path. It does not run during simulate(), and search never
reads reward. The default judge is the hosted Phi-4 unless
WHILEAI_JUDGE is set; it reads VLLM_API_KEY.
source: a SimulationData, a JSONL path, or a row list. A path
source is rewritten graded, unless output names another file;
a row list is updated in place.policy and tools: for a path or row list, pass the agent’s
system prompt and tool schemas so the judge sees the rules the agent
was under; a SimulationData supplies its own.limit: grade that many rows, then stop.spec, base_url, model, api_key: point the judge at
another OpenAI-compatible server instead of the hosted one.use_privileged: True shows the judge each row’s privileged
block (principle, reference, hidden state) the agent never saw.trust: the judge check against human labels ("warn",
"require", "off"), the same as SimulationData.grade.payload_chars (8000) caps the evidence the judge reads per row
and max_tokens (120) its reply; both land in judge_meta.
grade_llm
Defined in whileai/simulations/data.py.
Grade rows 0 or 1 with the hosted LLM judge and write a reason beside each.
Reach for it when the rows are a JSONL path or a row list rather than
a SimulationData you hold (that object has the same call as
data.grade()). It writes reward (0 or 1) and a one-sentence
reason on each row, keeps a previous score as qwen_reward
when present, and returns the judge report, a dict with graded,
n0, n1, backend, judge_version, warnings and
path. It does not run during simulate(), and search never
reads reward. The default judge is the hosted Phi-4 unless
WHILEAI_JUDGE is set; it reads VLLM_API_KEY.
source: a SimulationData, a JSONL path, or a row list. A path
source is rewritten graded, unless output names another file;
a row list is updated in place.policy and tools: for a path or row list, pass the agent’s
system prompt and tool schemas so the judge sees the rules the agent
was under; a SimulationData supplies its own.limit: grade that many rows, then stop.spec, base_url, model, api_key: point the judge at
another OpenAI-compatible server instead of the hosted one.use_privileged: True shows the judge each row’s privileged
block (principle, reference, hidden state) the agent never saw.trust: the judge check against human labels ("warn",
"require", "off"), the same as SimulationData.grade.payload_chars (8000) caps the evidence the judge reads per row
and max_tokens (120) its reply; both land in judge_meta.
llm_grade
Defined in whileai/simulations/data.py.
Module helper: advisory LLM scores on an existing SimulationData.
rank
Defined in whileai/simulations/data.py.
Score already-generated rows. source is a JSONL path, a row list,
or a SimulationData. Does not change simulate() or reward. Last modified on September 22, 2026
