> ## Documentation Index > Fetch the complete documentation index at: https://docs.while.ai/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > Install with `uv add whileai`; import as `import whileai as wai`. > Run the offline path first (`simulator=False`, `wai.seeded_agent`, a callable judge); no key is needed for it. > Report every pass rate with its interval and n, as `scored.pass_at` prints it. # whileai.simulations.simulation > simulate(): the entry point, and every argument it takes. 1 public names. `import whileai.simulations as wai`, then `wai.name`. | Name | What it does | | ----------------------- | --------------------------------------------------------------------- | | [`simulate`](#simulate) | Generate situations for an agent, roll them out, and return the rows. | ### simulate ```python theme={"theme":"vitesse-dark"} simulate( agent: Any = None, spec: Any = None, tools: list[dict] | None = None, system_prompt: str | None = None, budget: int | None = 1000, time_budget: float | None = None, until: str = 'compute', mode: str = 'explore', situations: int | None = None, requests_per_situation: int | None = None, rollouts_per_request: int | None = None, unique_situations: bool = False, reproducible: bool | None = None, grade: bool | str = False, llm_grade: bool = False, traces: Any = None, grader: Any = None, rubric: str | None = None, strategy: str = 'auto', seeds: list | None = None, scaffold: str | None = None, execute: Callable | None = None, output: str | None = None, tasks: Any = None, runs: int = 1, checkpoint: str | None = None, on_progress: Callable[[dict], None] | None = None, advanced: dict | None = None, repeats: int | None = None, phrasings: int | None = None, repeat_policy: str | None = None, concurrency: int | None = None, simulator: Any = None, user_model: Any = None, backend: Any = None, seed: int | None = None, sampling: dict | None = None, max_turns: int | None = None, avg_turns: float | None = None, fault_rate: float | None = None, temperature: float | None = None, timeout: float | None = None, logprobs: bool | None = None, hard_share: float | None = None, patience: str | None = None, **passed: Any, ) -> SimulationData ``` Defined in [`whileai/simulations/simulation.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/simulation.py). Generate situations for an agent, roll them out, and return the rows. Reach for it first: it is the run everything else reads. Give it the agent and it writes a grid of human asks (ordinary, vague, complex, adversarial), plays each one against the agent, and returns a `SimulationData`: `rows()` (one row per rollout, with the prompt, the tool calls, the final reply and its lineage), `warnings` (plain-words notes, each naming the call that changes it), `report()`, `search` (how the budget was spent), `pass_at` once graded, and `save(path)`. Nothing is graded unless you ask: pass `grade=True` to grade against the rubric with the judge, the same `wai.Judge(rubric=...)` that `data.grade` runs, so rows carry `reward`, `judge_status` and `judge_name` (no key stops before any budget is spent, nothing is substituted), `grade="conduct"` for the deterministic conduct check by name (what the agent did, not whether it did the job; rows carry `label_source="conduct"`), a callable `grader=` to score inside the loop, or grade later with `data.grade(...)` or `grade()`. The agent and the budget: * `agent`: a callable `message -> trajectory`, played single-turn (one message in, one trajectory out); or a backend object (`wai.OpenAI("gpt-4.1-mini")`) or spec string (`"openai:gpt-4.1-mini"`), which the SDK plays multi-turn from `tools` and `system_prompt`. Leave it `None` to play the model `configure(agent=)` set, else the model While hosts, from `tools`, `system_prompt` (alias `policy`) and `backend` (its model); `spec=` is the third way in. * `budget`: rows the run may produce, 1000 by default, a per-run cap when `runs` is above 1 (`runs=3, budget=100` returns up to 300 rows, and `report()["budget_per_run"]` carries the cap). `time_budget` is the same cap in seconds. * `mode`: `"explore"` (default) spends the budget on new coverage. `"rl"` gives every ask several repeats, so pass rates and RL groups exist, and defaults `repeat_policy` to `"successive"`, which stops early on unanimous asks; `"fixed"` gives every ask all k repeats. * `situations` (N, distinct worlds), `phrasings` (n, wordings of one world; alias `requests_per_situation`) and `repeats` (k, independent runs of one wording; alias `rollouts_per_request`) are three independent counts. Do not collapse them. Under `mode="rl"` `repeats` is a floor, not a count: dynamic sampling (Yu et al. 2025 (DAPO), arXiv:2503.14476) re-rolls uneven groups, so some asks end with more than k rollouts and `pass_at` reports the smallest k. Follow-ups branch on the run. `unique_situations=True` (alias `unique`) keeps picking new worlds (n=1, k=1 unless you set them). With `situations=N` the run stops once all N have their rollouts (`stopped_because="situations_exhausted"`), whatever `budget` still allows; a budget above `situations x phrasings x repeats` is not spent. * `concurrency` (32 parallel rollouts), `seed`, `sampling`, `temperature`, `timeout`, `logprobs`, `fault_rate`, `max_turns` / `avg_turns` (model-backed agents only; `avg_turns=1` is one user line and one reply, the follow-up branch never runs): each is `None` unless you set it, and a misspelled keyword is a `TypeError`, never silently ignored. Writer completions are `advanced["completions_per_request"]`; seed openers are `advanced["seed_prompts"]`. * `timeout`: seconds one agent call may take. Unset, it is 300 s or the reply budget at 4 tokens a second, whichever is longer (`max(300, agent_max_tokens / 4)`: 1,024 s at `agent_max_tokens=4096`), so a long reply is not re-rolled for taking the time it was allowed; a call that runs past it is an agent error and is re-rolled up to `repeats` times. Set it when you know the server's rate: `timeout >= agent_max_tokens / tokens-per-second-per-request`. What the situations come from: * `simulator`: the situation writer. `"hosted"` is the default written out, the same as leaving it unset; `False` is the offline template writer, no key needed. `user_model` plays the simulated person (a backend spec; `None` means the writer's model, the agent's own by default). * `seeds`: opening asks the writer keeps and varies. Every seed is run and becomes at least one situation: `situations` is sized up to `len(seeds)` when you pass a smaller number, and the search never spends a seed's slot on an ask it wrote itself. The one thing that can still drop a seed is `budget`, which pays for `len(seeds) * repeats` rows before anything else; when it cannot, `warnings` says which seeds were dropped and `search["seeds_dropped"]` lists them before rolling out. With a callable agent whose world has real ids (order numbers, account names), put those ids in the seeds or the tool descriptions, or the writer invents ids and every rollout is "not found". Seeds are asks to build a run around, not the eval set: to check that a fixed list of asks all ran and how each scored, use `evaluate(eval_set=asks)`. * `traces`: rows or a JSONL path of production traces. The grid then aims at the tools, faults and world states those traces show instead of the whole space (without it the grid comes from the agent's tools and policy alone, a cold start), and any generated row that near-copies a source trace is dropped, so held-out traces stay out of training. Traces reproduce world-visible situations. A failure that lives in how a reply is worded (an unsupported claim, an estimate not labelled as one, two questions where one was asked for) has no world-visible trigger, so traces alone cannot aim at it; put a grader in the loop for those. * `hard_share`: the difficulty dial, the share of situations drawn from the ambiguous, boundary and adversarial tiers, 0.40 by default, where a base fails most often. `search["tier_mix"]` reports the share asked for and the share drawn; `dimensions={"stance": [...]}` pins one axis and keeps the other axes of the grid. * `tasks`: a previous run (a `SimulationData`, its rows, or its JSONL path) whose task set is replayed instead of drawing a new one. Every distinct prompt is rolled out again, on its own `scenario_id` and `scenario_dimensions`, under the same faults and world state, and nothing else is generated. An unpinned re-run draws by seed and, above `concurrency: 1`, by completion order, so it shares only part of its tasks with the first and `compare_runs` drops the rest; pinning is how an A/B (a prompt edit, a model swap, another seed) keeps every pair. k comes from this call's `repeats` when given, otherwise from the pinned run (the most rollouts any of its prompts has), never from this call's `mode`, so `pass_at` reports the same k on both sides. The run stops when every pinned prompt has its rollouts (`stopped_because="tasks_done"`) or the budget is spent. * `runs`: replay the same task set that many times in one call and stamp `lineage.eval_run` (0, 1, 2, ...) on every row, which is what `delta_report` needs before it will call a change real (Lambert 2025, chapter Evaluation and its evaluation-variance appendix: one evaluation is a draw, three give a standard deviation). `simulate(tasks=base, runs=3)` is the usual form; without `tasks=` the first run draws the task set (from `seeds=` when given) and the rest replay it. Between runs nothing changes but the agent's own sampling (same tasks, faults, world state and seed), so a deterministic agent gives identical runs and a zero re-run band. All rows come back in one `SimulationData` (`output=` holds them all); `search["eval_runs"]` lists the rows and stop reason per run, and `eval_variance(data.rows())` splits by `eval_run` on its own. Replayed rows keep the writer of the run they replay on `writer_model` and say `lineage.replayed_from_run`, so `delta_report` on two runs of one call sees one writer. Watching and resuming a long run: * Progress goes to the `whileai.simulations` logger at INFO (and to stderr when nothing listens) every 10 finished rollouts or 10 s, re-rolls and losses counted as events too, so a run that only re-rolls still speaks: `120/2404 rollouts, 601 situations written, 1h2m elapsed, ~19h left, 96 re-rolled, 3 lost (3 agent error)`. `on_progress=` is a callable that receives the same numbers as a dict on every line, whatever the budget: `rows` (in `data.rows()` so far, resumed ones included), `cap`, `landed` (this call), `resumed`, `rerolled` and `rerolled_by` (`agent_error`, `empty_reply`, `tool_markup`), `timed_out` (the agent errors that were call timeouts), `lost` and `lost_by`, `inflight`, `situations`, `elapsed_s`. The run's final counts are `search["rollouts"]`, and `warnings` (plus a `UserWarning`) says so when more rollouts were re-rolled than landed, since the run then spent most of its time on calls that never became rows. * `checkpoint`: a JSONL path every row is appended to the moment it lands, so a killed run keeps its rows. Call again with the same `checkpoint=` and `tasks=` to resume: the rows on disk are loaded, a task with its `repeats` rows is skipped, one with fewer gets only the missing rollouts, and the returned `SimulationData` is the union (`search["rollouts"]["resumed"]` counts the loaded rows; `lineage.resumed` marks each). Without `tasks=` the rows on disk are loaded and count toward `budget`, and the run draws new situations for the rest. `output=` still writes the whole run at the end; `checkpoint=` is the file that survives a kill. What steers the search and answers the tools: * `grader`: a callable judge. Its verdict steers the search the way a tool fault already does: a row the grader failed is re-rolled and its ask is mutated into new ones, so the budget moves toward what the grader catches, not only toward broken tools. A grader that fails reply-form rules is exactly the signal traces cannot give (measured with a 12-rule grader: every rule with a tool-result trigger was reproduced and every rule about the reply's wording was not), and without a grader there is no verdict to steer by, so there is no switch to set: the grader is the switch. A graded failure is a reward under 0.5 (a 0 from a 0/1 judge, a failed verifier, a rubric below half); markers ride along on the row but do not aim on their own, since their direction differs per marker. `search["mutation_aims"]` counts the parents and the mutated rows per aim, `world_fault` and `graded_failure`. To grade beside the loop and still steer by tool faults alone, pass `advanced={"mutate_graded_failures": False}`. * `execute`: your world, a function `(tool, arguments) -> result` that answers every tool call for real, against your repo, database or service. Without it the mock world answers, which fits record-shaped tools and not code. Scheduled faults still apply first. `whileai.simulations.generate.agents.current_rollout` is a thread-local set before each rollout with `prompt`, `rollout_index` and `seed`, so `execute` can tell which run it is answering. * `patience`: how long the simulated person keeps answering the agent's questions. `"normal"` (the default) always tries to answer the first question, and from the second on may walk away (35% on the second, 60% on each after that, drawn per thread so a seeded run reproduces); at any question the person may also leave when it asks for something they could not or would not know. `"short"` walks away sooner (60% then 90%); `"endless"` never walks away, so the person answers every question until the depth cap and no rubric criterion about asking can fail. The odds are a default, not a measurement: to ground them, fit a Kaplan-Meier hazard per question index on source traces and set the levels from it. A row the person left carries `ended_by="user_left"` and ends on the agent's question; `search["ended_on_question"]` is `\{"share", "n", "user_left"\}`: of `n` rows, the share that ended on a question and how many of those the person left. * `scaffold`: generation-only guidance appended to the system prompt of the model-backed teacher during rollout (and to the scene writer). It never enters `profile.policy`, so exports and evals stay on the plain policy; it is ignored for callable agents. Measured to help some agents and hurt others, so there is no default: configure it per agent. * `reproducible`: the default, `None`, is `True` unless `time_budget` is set. `True` makes a seeded run bit-for-bit at any concurrency, apart from timing fields and per-invocation identity (with `grader=` every row's `lineage.scoring_run_id` names that one scoring pass, a fresh id per call; pass `run_id=` to `run_judge` to pin it): each batch of rollouts finishes, and every verdict of the batch lands, before the next is chosen, so every round sees the same state. A slow rollout holds its batch, so uneven latency costs throughput; pass `False` to trade the same task set on every machine for that throughput. A clock turns it off on its own, since a clock stop lands wherever the run happens to be. It was `False` before 0.111: three runs of one seed at the default concurrency then drew three different task sets (59, 58 and 58 tasks from one `budget=160`), because which rows land before the cap depended on thread timing, and a lesson that wanted the same file on every machine had to pin `concurrency=1`. Either way the draw is a function of the seed and the inputs alone, not of the interpreter: every float sum on the row-selection path is correctly rounded (`math.fsum`), so one seed picks the same rows on CPython 3.10 through 3.13, and a golden-value test pins that draw. Releases before this fix used the builtin `sum`, whose algorithm changed in CPython 3.12, and could swap one row per run between minor versions (issue #410). What every row records. `sampling` (`temperature`, `max_tokens` and `model` as the model backend resolved them), because a result is only comparable with its sampling settings on record (Lambert 2025, chapter Evaluation); a callable agent samples however it samples, so its rows carry `sampling: None` unless you pass `sampling={...}`, which is recorded as given. Three models can take part, the agent (`agent=` or `backend=`), the situation writer (`simulator=`) and the simulated user (`user_model=`), and every row names all three next to `model_version`: `writer_model`, `user_model`, and `judge_meta.model` once graded. When the agent model also wrote the situations or played the user, the run's `degraded` list carries `same_model` and `warnings` says which call separates them, since training on a model's own unfiltered output teaches it its own habits (Lambert 2025, chapter Synthetic Data and Distillation). ```python theme={"theme":"vitesse-dark"} import whileai.simulations as wai TOOLS = [{"type": "function", "function": { "name": "lookup_order", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}}, "required": ["order_id"]}}}] agent = wai.seeded_agent(TOOLS) # an offline demo agent, no key base = wai.simulate(agent, tools=TOOLS, simulator=False, seed=0, mode="rl", repeats=4, budget=32) rerun = wai.simulate(agent, tools=TOOLS, simulator=False, seed=0, tasks=base, mode="rl", # same tasks, k=4 inherited checkpoint="rerun.jsonl", # rows land here as they finish on_progress=print) # or watch the whileai.simulations logger print(base.stopped_because, len(base.rows()), base.warnings) print(rerun.search["rollouts"]) # landed, resumed, rerolled_by, lost_by ```