> ## Documentation Index > Fetch the complete documentation index at: https://docs.while.ai/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > Install with `uv add whileai`; import as `import whileai as wai`. > Run the offline path first (`simulator=False`, `wai.seeded_agent`, a callable judge); no key is needed for it. > Report every pass rate with its interval and n, as `scored.pass_at` prints it. # whileai.simulations.data > SimulationData: the object simulate() returns, and what it can do next. 6 public names. `import whileai.simulations as wai`, then `wai.name`. | Name | What it does | | ----------------------------------- | --------------------------------------------------------------------------- | | [`SimulationData`](#simulationdata) | *no docstring* | | [`conversation`](#conversation) | User/agent turns from prompt + steps. | | [`grade`](#grade) | Grade rows 0 or 1 with the hosted LLM judge and write a reason beside each. | | [`grade_llm`](#grade_llm) | Grade rows 0 or 1 with the hosted LLM judge and write a reason beside each. | | [`llm_grade`](#llm_grade) | Module helper: advisory LLM scores on an existing SimulationData. | | [`rank`](#rank) | Score already-generated rows. | ### SimulationData ```python theme={"theme":"vitesse-dark"} class SimulationData( trajectories: list[dict] = , arm_yield: dict = , stopped_because: str = 'budget', declared_tools: set = , stages: list[str] = , scaffold_chars: int = 0, degraded: list[str] = , warnings: list[str] = , semantic: bool = False, profile: AgentProfile | None = None, embedder_name: str = '', elapsed_seconds: float = 0.0, rows_per_second: float = 0.0, arm_weights: dict = , scenario_generation_seconds: float = 0.0, embedding_selection_seconds: float = 0.0, rollout_seconds: float = 0.0, row_seconds: list = , unique_prompts: int = 0, scene_brief: str = '', scene_brief_seconds: float = 0.0, first_row_seconds: float = 0.0, semantic_duplicate_rate: float | None = None, unique_behavior_signatures: int = 0, coverage_curve: list[dict] = , coverage: dict = , search: dict = , budget: int = 0, path: str = '', mode: str = 'explore', repeat_policy: str = 'none', n_situations: int | None = None, requests_per_situation: int = 1, rollouts_per_request: int = 1, unique_situations: bool = False, allocator: dict = , writer_model: str = '', user_model: str | None = None, system_prompts: dict[str, str] = , ) -> None ``` Defined in [`whileai/simulations/data.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/data.py). #### SimulationData.compare\_judges ```python theme={"theme":"vitesse-dark"} compare_judges( self, judges: Mapping[str, Any] | Sequence[Any], gold: str = 'gold_reward', allow_model_gold: bool = False, concurrency: int = 8, floors: tuple[float, float] = (0.8, 0.6), ) ``` Grade these rows with several judges and rank them against the gold labels. `judges` maps a name to a spec string (`"typesafe:jev-latest"`), a backend object, a `wai.Judge` or any judge callable. Each grades its own copy of the rows under this run's system prompt and tools, then is scored the way `judge_trust` scores one judge: agreement with a Wilson interval, kappa, leak rate, unsure and unjudged counts, seconds per row. Returns a `JudgeComparison` that prints as a table ranked by kappa; `whileai.judge_comparison.compare_judges` has the full account and takes a bare row list. #### SimulationData.grade ```python theme={"theme":"vitesse-dark"} grade( self, grader=None, judge=None, llm: bool = False, llm_spec: str | None = None, spec: str | None = None, api_key: str | None = None, path: str | None = None, concurrency: int = 32, llm_concurrency: int = 16, version: str | None = None, use_privileged: bool = False, scale: tuple[float, float] | None = None, rubric: str | None = None, trust: str = 'warn', payload_chars: int = 8000, max_tokens: int = 120, ) ``` Grade this run's rows in place with the hosted judge or your own callable. Reach for it right after `simulate` to score the rows without leaving the object. Simulation never calls it on its own. Three paths, chosen by what you pass: * No callable (or `llm=True`): `grade_llm`, the hosted LLM judge (Phi-4 unless `WHILEAI_JUDGE` is set, a different family from the hosted Qwen policy), read from `VLLM_API_KEY`. It writes `reward` (0 or 1) and `reason` onto the rows in place and returns the judge report, a dict with `graded`, `n0`, `n1`, `backend`, `judge_version` and `warnings`. * `grader=`, a plain callable returning a number or `{"reward": ..., "reason": ...}` per row: scores every row in place over `concurrency` threads and returns the run itself, so `data.grade(my_grader).pass_at` reads through. * `judge=`, the contract path: any callable honoring the judge contract, which returns `{"reward": 0 or 1, "reason": str, "markers": {name: value}}` per row (a bare number works too). The contract and its failure modes are written out in full in `whileai.simulations.score.judging` (note the `score.`; there is no `whileai.simulations.judging`). It returns a `ScoredData` of copies: the trajectories here stay unmodified, judge errors are marked per row instead of coerced to 0, and its output feeds `export_dataset` and `simulate(traces=...)` directly. Arguments that matter: * `version`: names the judge's version (model, rubric hash) and is recorded on every scored row; the hosted grader stamps its own. * `rubric`: what doing the job means, as text, for the hosted judge; without it the judge grades the conduct floor only, and says so. * `use_privileged`: `True` shows the hosted judge each row's `privileged` block (principle, reference, hidden state) the agent never saw. * `trust`: the judge check against the rows' human labels (`attach_labels(kind="human")`), run on every path, with the summary stamped on each graded row's `judge_meta["trust"]`. `"warn"` (the default) logs one line when the check failed or no labels exist, `"require"` raises instead, `"off"` skips it. * `path`: write the graded run's JSONL there afterwards. * `spec`: which model judges on the hosted path, as a backend spec (`"typesafe:jev-latest"`, `"openai:gpt-4.1-mini"`); the same keyword `grade_llm`, `pairwise_judge` and `rubric_judge` take. `llm_spec` is its older name and still works. ```python theme={"theme":"vitesse-dark"} data = wai.simulate(agent, tools=TOOLS, simulator=False, budget=16) data.grade(lambda row: 1.0 if row.get("final_text") else 0.0) print(data.pass_at) ``` #### SimulationData.grade\_llm ```python theme={"theme":"vitesse-dark"} grade_llm( self, spec: str | None = None, base_url: str | None = None, model: str | None = None, concurrency: int = 16, api_key: str | None = None, path: str | None = None, limit: int | None = None, prompt: str | None = None, use_privileged: bool = False, rubric: str | None = None, trust: str = 'warn', payload_chars: int = 8000, max_tokens: int = 120, ) ``` Binary 0/1 situation grade. Default brain is the hosted judge (Phi-4 unless `WHILEAI_JUDGE` is set), never the policy model. `use_privileged` shows the judge each row's `privileged` block (principle, reference, hidden state) the agent never saw. `trust` is the judge check against human labels: see `grade`. `payload_chars` caps the evidence the judge reads per row and `max_tokens` its reply (defaults `JUDGE_PAYLOAD_CHARS` and `JUDGE_MAX_TOKENS` in `defaults.py`); both land in `judge_meta`. #### SimulationData.leak\_report ```python theme={"theme":"vitesse-dark"} leak_report(self, min_len: int = 12) -> dict[str, Any] ``` Did any reply quote its own `privileged` block? Reads the trajectories, which still carry the block; `rows()` is scrubbed and would check nothing. Same report as `leak_report`. #### SimulationData.llm\_grade ```python theme={"theme":"vitesse-dark"} llm_grade( self, spec: str | None = None, concurrency: int = 16, api_key: str | None = None, path: str | None = None, ) ``` Advisory LLM pass. Leaves deterministic reward untouched. #### SimulationData.push ```python theme={"theme":"vitesse-dark"} push( self, name: str, api_key: str | None = None, parent: str | None = None, agent: str | None = None, publish: bool = False, description: str | None = None, gate: bool = True, purpose: str = 'train', holdout: float | None = None, endorsed: Sequence[str] = (), strict_hacks: bool = False, ) -> dict ``` Upload this run to your While account as a dataset. `purpose` is the section it lands in on the Datasets page (`"train"` by default; `"holdout"` or `"eval"`). `holdout=0.2` keeps a fifth of the tasks (by `scenario_id`) out of the training set and pushes them as a second, linked dataset with purpose `"holdout"`; the entry carries it as `["holdout"]`. The simulation mode is recorded on both. `api_key` defaults to the `WHILEAI_API_KEY` env var, then the key saved by `wai login`. Pass `parent` (a `ds_...` id) when this run iterates on an existing dataset, so lineage shows on the platform. `publish=True` with an `agent` name also puts it on the public catalog at huggingface.co/while-ai as a card. Returns the registry entry with `datasetId`. `gate=True` runs `publish_gate` first: every graded row gets a `calibration` stamp (per-task pass rate, k, producing policy), and an RL-shaped run that is ungraded or has no mixed group is refused with `PublishGateError`. The gate report is returned as `entry["gate"]`. `gate=False` uploads rows as they are. `endorsed` names what the reward should track (feature-name substrings, e.g. `"tool:lookup_order"`) for the gate's `hack_scan`; `strict_hacks=True` refuses a set whose reward is best explained by something else. #### SimulationData.rank ```python theme={"theme":"vitesse-dark"} rank(self, path: str | None = None) -> dict ``` Second-pass quality scores. Leaves conduct `reward` untouched. Writes `quality`, `quality_reason`, `quality_scores` on each trajectory and rewrites the saved JSONL, or `path` if you pass one. #### SimulationData.report ```python theme={"theme":"vitesse-dark"} report(self) -> dict ``` Run-level coverage summary (same as `data.coverage`). #### SimulationData.select ```python theme={"theme":"vitesse-dark"} select( self, mode: str | None = None, target: int = 1000, band: tuple[float, float] | None = None, endorsed: Sequence[str] = (), truncated: str = 'drop', ) ``` The rows worth training on, as a `Selection` that prints its report. With no `mode`: diverse pass-labeled demonstrations via `select_for_sft`, one of each distinct way of being right before any repeats, junk and duplicate prompts dropped. With `mode="rl"` or `"sft"`: `optimize`, the full gate sequence (privileged leaks, difficulty band, unanimous groups, duplicates, truncation, hack scan), with `band`, `endorsed` and `truncated` as there. Requires graded rows — grade in-loop (`grade=True`, `grader=`) or afterwards with `grade()`. The report lands in `search["selection"]` and on the result's `.report`. ```python theme={"theme":"vitesse-dark"} rows = data.select(mode="rl") print(rows) # what each gate dropped and why rows.export("train.jsonl") ``` #### SimulationData.training\_set ```python theme={"theme":"vitesse-dark"} training_set( self, output: str | None = None, target: int = 1000, validate: bool = True, ) -> dict ``` Select the recommended rows and export them trainer-ready. `select()` picks diverse pass-labeled rows, `export_training` writes them as chat JSONL with this run's system prompt and tools and the tool-call round-trip gate. Returns the export report with the selection report attached; pass `output` to write the file. Raw simulation rows are not the training artifact — this is. ### conversation ```python theme={"theme":"vitesse-dark"} conversation(row: dict) -> list[dict] ``` Defined in [`whileai/simulations/data.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/data.py). User/agent turns from prompt + steps. Tool calls stay on the assistant turn. ### grade ```python theme={"theme":"vitesse-dark"} grade( source, spec: str | None = None, base_url: str | None = None, model: str | None = None, concurrency: int = 16, api_key: str | None = None, path: str | None = None, limit: int | None = None, output: str | None = None, prompt: str | None = None, policy: str = '', tools: list | None = None, use_privileged: bool = False, trust: str = 'warn', payload_chars: int = 8000, max_tokens: int = 120, ) ``` Defined in [`whileai/simulations/data.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/data.py). Grade rows 0 or 1 with the hosted LLM judge and write a reason beside each. Reach for it when the rows are a JSONL path or a row list rather than a `SimulationData` you hold (that object has the same call as `data.grade()`). It writes `reward` (0 or 1) and a one-sentence `reason` on each row, keeps a previous score as `qwen_reward` when present, and returns the judge report, a dict with `graded`, `n0`, `n1`, `backend`, `judge_version`, `warnings` and `path`. It does not run during `simulate()`, and search never reads `reward`. The default judge is the hosted Phi-4 unless `WHILEAI_JUDGE` is set; it reads `VLLM_API_KEY`. * `source`: a `SimulationData`, a JSONL path, or a row list. A path source is rewritten graded, unless `output` names another file; a row list is updated in place. * `policy` and `tools`: for a path or row list, pass the agent's system prompt and tool schemas so the judge sees the rules the agent was under; a `SimulationData` supplies its own. * `limit`: grade that many rows, then stop. * `spec`, `base_url`, `model`, `api_key`: point the judge at another OpenAI-compatible server instead of the hosted one. * `use_privileged`: `True` shows the judge each row's `privileged` block (principle, reference, hidden state) the agent never saw. * `trust`: the judge check against human labels (`"warn"`, `"require"`, `"off"`), the same as `SimulationData.grade`. * `payload_chars` (8000) caps the evidence the judge reads per row and `max_tokens` (120) its reply; both land in `judge_meta`. ### grade\_llm ```python theme={"theme":"vitesse-dark"} grade_llm( source, spec: str | None = None, base_url: str | None = None, model: str | None = None, concurrency: int = 16, api_key: str | None = None, path: str | None = None, limit: int | None = None, output: str | None = None, prompt: str | None = None, policy: str = '', tools: list | None = None, use_privileged: bool = False, trust: str = 'warn', payload_chars: int = 8000, max_tokens: int = 120, ) ``` Defined in [`whileai/simulations/data.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/data.py). Grade rows 0 or 1 with the hosted LLM judge and write a reason beside each. Reach for it when the rows are a JSONL path or a row list rather than a `SimulationData` you hold (that object has the same call as `data.grade()`). It writes `reward` (0 or 1) and a one-sentence `reason` on each row, keeps a previous score as `qwen_reward` when present, and returns the judge report, a dict with `graded`, `n0`, `n1`, `backend`, `judge_version`, `warnings` and `path`. It does not run during `simulate()`, and search never reads `reward`. The default judge is the hosted Phi-4 unless `WHILEAI_JUDGE` is set; it reads `VLLM_API_KEY`. * `source`: a `SimulationData`, a JSONL path, or a row list. A path source is rewritten graded, unless `output` names another file; a row list is updated in place. * `policy` and `tools`: for a path or row list, pass the agent's system prompt and tool schemas so the judge sees the rules the agent was under; a `SimulationData` supplies its own. * `limit`: grade that many rows, then stop. * `spec`, `base_url`, `model`, `api_key`: point the judge at another OpenAI-compatible server instead of the hosted one. * `use_privileged`: `True` shows the judge each row's `privileged` block (principle, reference, hidden state) the agent never saw. * `trust`: the judge check against human labels (`"warn"`, `"require"`, `"off"`), the same as `SimulationData.grade`. * `payload_chars` (8000) caps the evidence the judge reads per row and `max_tokens` (120) its reply; both land in `judge_meta`. ### llm\_grade ```python theme={"theme":"vitesse-dark"} llm_grade( data: SimulationData, spec: str | None = None, concurrency: int = 16, api_key: str | None = None, path: str | None = None, ) -> SimulationData ``` Defined in [`whileai/simulations/data.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/data.py). Module helper: advisory LLM scores on an existing SimulationData. ### rank ```python theme={"theme":"vitesse-dark"} rank( source, output: str | None = None, min_quality: float | None = None, ) -> dict ``` Defined in [`whileai/simulations/data.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/data.py). Score already-generated rows. `source` is a JSONL path, a row list, or a `SimulationData`. Does not change `simulate()` or `reward`.