whileai.simulations.data

6 public names. import whileai.simulations as wai, then wai.name.

Name	What it does
`SimulationData`	no docstring
`conversation`	User/agent turns from prompt + steps.
`grade`	Grade rows 0 or 1 with the hosted LLM judge and write a reason beside each.
`grade_llm`	Grade rows 0 or 1 with the hosted LLM judge and write a reason beside each.
`llm_grade`	Module helper: advisory LLM scores on an existing SimulationData.
`rank`	Score already-generated rows.

SimulationData

class SimulationData(
    trajectories: list[dict] = <factory>,
    arm_yield: dict = <factory>,
    stopped_because: str = 'budget',
    declared_tools: set = <factory>,
    stages: list[str] = <factory>,
    scaffold_chars: int = 0,
    degraded: list[str] = <factory>,
    warnings: list[str] = <factory>,
    semantic: bool = False,
    profile: AgentProfile | None = None,
    embedder_name: str = '',
    elapsed_seconds: float = 0.0,
    rows_per_second: float = 0.0,
    arm_weights: dict = <factory>,
    scenario_generation_seconds: float = 0.0,
    embedding_selection_seconds: float = 0.0,
    rollout_seconds: float = 0.0,
    row_seconds: list = <factory>,
    unique_prompts: int = 0,
    scene_brief: str = '',
    scene_brief_seconds: float = 0.0,
    first_row_seconds: float = 0.0,
    semantic_duplicate_rate: float | None = None,
    unique_behavior_signatures: int = 0,
    coverage_curve: list[dict] = <factory>,
    coverage: dict = <factory>,
    search: dict = <factory>,
    budget: int = 0,
    path: str = '',
    mode: str = 'explore',
    repeat_policy: str = 'none',
    n_situations: int | None = None,
    requests_per_situation: int = 1,
    rollouts_per_request: int = 1,
    unique_situations: bool = False,
    allocator: dict = <factory>,
    writer_model: str = '',
    user_model: str | None = None,
    system_prompts: dict[str, str] = <factory>,
) -> None

Defined in whileai/simulations/data.py.

SimulationData.compare_judges

compare_judges(
    self,
    judges: Mapping[str, Any] | Sequence[Any],
    gold: str = 'gold_reward',
    allow_model_gold: bool = False,
    concurrency: int = 8,
    floors: tuple[float, float] = (0.8, 0.6),
)

Grade these rows with several judges and rank them against the gold labels. judges maps a name to a spec string ("typesafe:jev-latest"), a backend object, a wai.Judge or any judge callable. Each grades its own copy of the rows under this run’s system prompt and tools, then is scored the way judge_trust scores one judge: agreement with a Wilson interval, kappa, leak rate, unsure and unjudged counts, seconds per row. Returns a JudgeComparison that prints as a table ranked by kappa; whileai.judge_comparison.compare_judges has the full account and takes a bare row list.

SimulationData.grade

grade(
    self,
    grader=None,
    judge=None,
    llm: bool = False,
    llm_spec: str | None = None,
    spec: str | None = None,
    api_key: str | None = None,
    path: str | None = None,
    concurrency: int = 32,
    llm_concurrency: int = 16,
    version: str | None = None,
    use_privileged: bool = False,
    scale: tuple[float, float] | None = None,
    rubric: str | None = None,
    trust: str = 'warn',
    payload_chars: int = 8000,
    max_tokens: int = 120,
)

Grade this run’s rows in place with the hosted judge or your own callable. Reach for it right after simulate to score the rows without leaving the object. Simulation never calls it on its own. Three paths, chosen by what you pass:

No callable (or llm=True): grade_llm, the hosted LLM judge (Phi-4 unless WHILEAI_JUDGE is set, a different family from the hosted Qwen policy), read from VLLM_API_KEY. It writes reward (0 or 1) and reason onto the rows in place and returns the judge report, a dict with graded, n0, n1, backend, judge_version and warnings.
grader=, a plain callable returning a number or {"reward": ..., "reason": ...} per row: scores every row in place over concurrency threads and returns the run itself, so data.grade(my_grader).pass_at reads through.
judge=, the contract path: any callable honoring the judge contract, which returns {"reward": 0 or 1, "reason": str, "markers": {name: value}} per row (a bare number works too). The contract and its failure modes are written out in full in whileai.simulations.score.judging (note the score.; there is no whileai.simulations.judging). It returns a ScoredData of copies: the trajectories here stay unmodified, judge errors are marked per row instead of coerced to 0, and its output feeds export_dataset and simulate(traces=...) directly.

Arguments that matter:

version: names the judge’s version (model, rubric hash) and is recorded on every scored row; the hosted grader stamps its own.
rubric: what doing the job means, as text, for the hosted judge; without it the judge grades the conduct floor only, and says so.
use_privileged: True shows the hosted judge each row’s privileged block (principle, reference, hidden state) the agent never saw.
trust: the judge check against the rows’ human labels (attach_labels(kind="human")), run on every path, with the summary stamped on each graded row’s judge_meta["trust"]. "warn" (the default) logs one line when the check failed or no labels exist, "require" raises instead, "off" skips it.
path: write the graded run’s JSONL there afterwards.
spec: which model judges on the hosted path, as a backend spec ("typesafe:jev-latest", "openai:gpt-4.1-mini"); the same keyword grade_llm, pairwise_judge and rubric_judge take. llm_spec is its older name and still works.

data = wai.simulate(agent, tools=TOOLS, simulator=False, budget=16)
data.grade(lambda row: 1.0 if row.get("final_text") else 0.0)
print(data.pass_at)

SimulationData.grade_llm

grade_llm(
    self,
    spec: str | None = None,
    base_url: str | None = None,
    model: str | None = None,
    concurrency: int = 16,
    api_key: str | None = None,
    path: str | None = None,
    limit: int | None = None,
    prompt: str | None = None,
    use_privileged: bool = False,
    rubric: str | None = None,
    trust: str = 'warn',
    payload_chars: int = 8000,
    max_tokens: int = 120,
)

Binary 0/1 situation grade. Default brain is the hosted judge (Phi-4 unless WHILEAI_JUDGE is set), never the policy model. use_privileged shows the judge each row’s privileged block (principle, reference, hidden state) the agent never saw. trust is the judge check against human labels: see grade. payload_chars caps the evidence the judge reads per row and max_tokens its reply (defaults JUDGE_PAYLOAD_CHARS and JUDGE_MAX_TOKENS in defaults.py); both land in judge_meta.

SimulationData.leak_report

leak_report(self, min_len: int = 12) -> dict[str, Any]

Did any reply quote its own privileged block? Reads the trajectories, which still carry the block; rows() is scrubbed and would check nothing. Same report as leak_report.

SimulationData.llm_grade

llm_grade(
    self,
    spec: str | None = None,
    concurrency: int = 16,
    api_key: str | None = None,
    path: str | None = None,
)

Advisory LLM pass. Leaves deterministic reward untouched.

SimulationData.push

push(
    self,
    name: str,
    api_key: str | None = None,
    parent: str | None = None,
    agent: str | None = None,
    publish: bool = False,
    description: str | None = None,
    gate: bool = True,
    purpose: str = 'train',
    holdout: float | None = None,
    endorsed: Sequence[str] = (),
    strict_hacks: bool = False,
) -> dict

Upload this run to your While account as a dataset. purpose is the section it lands in on the Datasets page ("train" by default; "holdout" or "eval"). holdout=0.2 keeps a fifth of the tasks (by scenario_id) out of the training set and pushes them as a second, linked dataset with purpose "holdout"; the entry carries it as ["holdout"]. The simulation mode is recorded on both. api_key defaults to the WHILEAI_API_KEY env var, then the key saved by wai login. Pass parent (a ds_... id) when this run iterates on an existing dataset, so lineage shows on the platform. publish=True with an agent name also puts it on the public catalog at huggingface.co/while-ai as a card. Returns the registry entry with datasetId. gate=True runs publish_gate first: every graded row gets a calibration stamp (per-task pass rate, k, producing policy), and an RL-shaped run that is ungraded or has no mixed group is refused with PublishGateError. The gate report is returned as entry["gate"]. gate=False uploads rows as they are. endorsed names what the reward should track (feature-name substrings, e.g. "tool:lookup_order") for the gate’s hack_scan; strict_hacks=True refuses a set whose reward is best explained by something else.

SimulationData.rank

rank(self, path: str | None = None) -> dict

Second-pass quality scores. Leaves conduct reward untouched. Writes quality, quality_reason, quality_scores on each trajectory and rewrites the saved JSONL, or path if you pass one.

SimulationData.report

report(self) -> dict

Run-level coverage summary (same as data.coverage).

SimulationData.select

select(
    self,
    mode: str | None = None,
    target: int = 1000,
    band: tuple[float, float] | None = None,
    endorsed: Sequence[str] = (),
    truncated: str = 'drop',
)

The rows worth training on, as a Selection that prints its report. With no mode: diverse pass-labeled demonstrations via select_for_sft, one of each distinct way of being right before any repeats, junk and duplicate prompts dropped. With mode="rl" or "sft": optimize, the full gate sequence (privileged leaks, difficulty band, unanimous groups, duplicates, truncation, hack scan), with band, endorsed and truncated as there. Requires graded rows — grade in-loop (grade=True, grader=) or afterwards with grade(). The report lands in search["selection"] and on the result’s .report.

rows = data.select(mode="rl")
print(rows)              # what each gate dropped and why
rows.export("train.jsonl")

SimulationData.training_set

training_set(
    self,
    output: str | None = None,
    target: int = 1000,
    validate: bool = True,
) -> dict

Select the recommended rows and export them trainer-ready. select() picks diverse pass-labeled rows, export_training writes them as chat JSONL with this run’s system prompt and tools and the tool-call round-trip gate. Returns the export report with the selection report attached; pass output to write the file. Raw simulation rows are not the training artifact — this is.

conversation

conversation(row: dict) -> list[dict]

Defined in whileai/simulations/data.py. User/agent turns from prompt + steps. Tool calls stay on the assistant turn.

grade

grade(
    source,
    spec: str | None = None,
    base_url: str | None = None,
    model: str | None = None,
    concurrency: int = 16,
    api_key: str | None = None,
    path: str | None = None,
    limit: int | None = None,
    output: str | None = None,
    prompt: str | None = None,
    policy: str = '',
    tools: list | None = None,
    use_privileged: bool = False,
    trust: str = 'warn',
    payload_chars: int = 8000,
    max_tokens: int = 120,
)

Defined in whileai/simulations/data.py. Grade rows 0 or 1 with the hosted LLM judge and write a reason beside each. Reach for it when the rows are a JSONL path or a row list rather than a SimulationData you hold (that object has the same call as data.grade()). It writes reward (0 or 1) and a one-sentence reason on each row, keeps a previous score as qwen_reward when present, and returns the judge report, a dict with graded, n0, n1, backend, judge_version, warnings and path. It does not run during simulate(), and search never reads reward. The default judge is the hosted Phi-4 unless WHILEAI_JUDGE is set; it reads VLLM_API_KEY.

source: a SimulationData, a JSONL path, or a row list. A path source is rewritten graded, unless output names another file; a row list is updated in place.
policy and tools: for a path or row list, pass the agent’s system prompt and tool schemas so the judge sees the rules the agent was under; a SimulationData supplies its own.
limit: grade that many rows, then stop.
spec, base_url, model, api_key: point the judge at another OpenAI-compatible server instead of the hosted one.
use_privileged: True shows the judge each row’s privileged block (principle, reference, hidden state) the agent never saw.
trust: the judge check against human labels ("warn", "require", "off"), the same as SimulationData.grade.
payload_chars (8000) caps the evidence the judge reads per row and max_tokens (120) its reply; both land in judge_meta.

grade_llm

grade_llm(
    source,
    spec: str | None = None,
    base_url: str | None = None,
    model: str | None = None,
    concurrency: int = 16,
    api_key: str | None = None,
    path: str | None = None,
    limit: int | None = None,
    output: str | None = None,
    prompt: str | None = None,
    policy: str = '',
    tools: list | None = None,
    use_privileged: bool = False,
    trust: str = 'warn',
    payload_chars: int = 8000,
    max_tokens: int = 120,
)

source: a SimulationData, a JSONL path, or a row list. A path source is rewritten graded, unless output names another file; a row list is updated in place.
policy and tools: for a path or row list, pass the agent’s system prompt and tool schemas so the judge sees the rules the agent was under; a SimulationData supplies its own.
limit: grade that many rows, then stop.
spec, base_url, model, api_key: point the judge at another OpenAI-compatible server instead of the hosted one.
use_privileged: True shows the judge each row’s privileged block (principle, reference, hidden state) the agent never saw.
trust: the judge check against human labels ("warn", "require", "off"), the same as SimulationData.grade.
payload_chars (8000) caps the evidence the judge reads per row and max_tokens (120) its reply; both land in judge_meta.

llm_grade

llm_grade(
    data: SimulationData,
    spec: str | None = None,
    concurrency: int = 16,
    api_key: str | None = None,
    path: str | None = None,
) -> SimulationData

Defined in whileai/simulations/data.py. Module helper: advisory LLM scores on an existing SimulationData.

rank

rank(
    source,
    output: str | None = None,
    min_quality: float | None = None,
) -> dict

Defined in whileai/simulations/data.py. Score already-generated rows. source is a JSONL path, a row list, or a SimulationData. Does not change simulate() or reward.

​SimulationData

​SimulationData.compare_judges

​SimulationData.grade

​SimulationData.grade_llm

​SimulationData.leak_report

​SimulationData.llm_grade

​SimulationData.push

​SimulationData.rank

​SimulationData.report

​SimulationData.select

​SimulationData.training_set

​conversation

​grade

​grade_llm

​llm_grade

​rank