Skip to main content
1 public names. import whileai.simulations as wai, then wai.name.

​
simulate

Defined in whileai/simulations/simulation.py. Generate situations for an agent, roll them out, and return the rows. Reach for it first: it is the run everything else reads. Give it the agent and it writes a grid of human asks (ordinary, vague, complex, adversarial), plays each one against the agent, and returns a SimulationData: rows() (one row per rollout, with the prompt, the tool calls, the final reply and its lineage), warnings (plain-words notes, each naming the call that changes it), report(), search (how the budget was spent), pass_at once graded, and save(path). Nothing is graded unless you ask: pass grade=True to grade against the rubric with the judge, the same wai.Judge(rubric=...) that data.grade runs, so rows carry reward, judge_status and judge_name (no key stops before any budget is spent, nothing is substituted), grade="conduct" for the deterministic conduct check by name (what the agent did, not whether it did the job; rows carry label_source="conduct"), a callable grader= to score inside the loop, or grade later with data.grade(...) or grade(). The agent and the budget:
  • agent: a callable message -> trajectory, played single-turn (one message in, one trajectory out); or a backend object (wai.OpenAI("gpt-4.1-mini")) or spec string ("openai:gpt-4.1-mini"), which the SDK plays multi-turn from tools and system_prompt. Leave it None to play the model configure(agent=) set, else the model While hosts, from tools, system_prompt (alias policy) and backend (its model); spec= is the third way in.
  • budget: rows the run may produce, 1000 by default, a per-run cap when runs is above 1 (runs=3, budget=100 returns up to 300 rows, and report()["budget_per_run"] carries the cap). time_budget is the same cap in seconds.
  • mode: "explore" (default) spends the budget on new coverage. "rl" gives every ask several repeats, so pass rates and RL groups exist, and defaults repeat_policy to "successive", which stops early on unanimous asks; "fixed" gives every ask all k repeats.
  • situations (N, distinct worlds), phrasings (n, wordings of one world; alias requests_per_situation) and repeats (k, independent runs of one wording; alias rollouts_per_request) are three independent counts. Do not collapse them. Under mode="rl" repeats is a floor, not a count: dynamic sampling (Yu et al. 2025 (DAPO), arXiv:2503.14476) re-rolls uneven groups, so some asks end with more than k rollouts and pass_at reports the smallest k. Follow-ups branch on the run. unique_situations=True (alias unique) keeps picking new worlds (n=1, k=1 unless you set them). With situations=N the run stops once all N have their rollouts (stopped_because="situations_exhausted"), whatever budget still allows; a budget above situations x phrasings x repeats is not spent.
  • concurrency (32 parallel rollouts), seed, sampling, temperature, timeout, logprobs, fault_rate, max_turns / avg_turns (model-backed agents only; avg_turns=1 is one user line and one reply, the follow-up branch never runs): each is None unless you set it, and a misspelled keyword is a TypeError, never silently ignored. Writer completions are advanced["completions_per_request"]; seed openers are advanced["seed_prompts"].
  • timeout: seconds one agent call may take. Unset, it is 300 s or the reply budget at 4 tokens a second, whichever is longer (max(300, agent_max_tokens / 4): 1,024 s at agent_max_tokens=4096), so a long reply is not re-rolled for taking the time it was allowed; a call that runs past it is an agent error and is re-rolled up to repeats times. Set it when you know the server’s rate: timeout >= agent_max_tokens / tokens-per-second-per-request.
What the situations come from:
  • simulator: the situation writer. "hosted" is the default written out, the same as leaving it unset; False is the offline template writer, no key needed. user_model plays the simulated person (a backend spec; None means the writer’s model, the agent’s own by default).
  • seeds: opening asks the writer keeps and varies. Every seed is run and becomes at least one situation: situations is sized up to len(seeds) when you pass a smaller number, and the search never spends a seed’s slot on an ask it wrote itself. The one thing that can still drop a seed is budget, which pays for len(seeds) * repeats rows before anything else; when it cannot, warnings says which seeds were dropped and search["seeds_dropped"] lists them before rolling out. With a callable agent whose world has real ids (order numbers, account names), put those ids in the seeds or the tool descriptions, or the writer invents ids and every rollout is “not found”. Seeds are asks to build a run around, not the eval set: to check that a fixed list of asks all ran and how each scored, use evaluate(eval_set=asks).
  • traces: rows or a JSONL path of production traces. The grid then aims at the tools, faults and world states those traces show instead of the whole space (without it the grid comes from the agent’s tools and policy alone, a cold start), and any generated row that near-copies a source trace is dropped, so held-out traces stay out of training. Traces reproduce world-visible situations. A failure that lives in how a reply is worded (an unsupported claim, an estimate not labelled as one, two questions where one was asked for) has no world-visible trigger, so traces alone cannot aim at it; put a grader in the loop for those.
  • hard_share: the difficulty dial, the share of situations drawn from the ambiguous, boundary and adversarial tiers, 0.40 by default, where a base fails most often. search["tier_mix"] reports the share asked for and the share drawn; dimensions={"stance": [...]} pins one axis and keeps the other axes of the grid.
  • tasks: a previous run (a SimulationData, its rows, or its JSONL path) whose task set is replayed instead of drawing a new one. Every distinct prompt is rolled out again, on its own scenario_id and scenario_dimensions, under the same faults and world state, and nothing else is generated. An unpinned re-run draws by seed and, above concurrency: 1, by completion order, so it shares only part of its tasks with the first and compare_runs drops the rest; pinning is how an A/B (a prompt edit, a model swap, another seed) keeps every pair. k comes from this call’s repeats when given, otherwise from the pinned run (the most rollouts any of its prompts has), never from this call’s mode, so pass_at reports the same k on both sides. The run stops when every pinned prompt has its rollouts (stopped_because="tasks_done") or the budget is spent.
  • runs: replay the same task set that many times in one call and stamp lineage.eval_run (0, 1, 2, …) on every row, which is what delta_report needs before it will call a change real (Lambert 2025, chapter Evaluation and its evaluation-variance appendix: one evaluation is a draw, three give a standard deviation). simulate(tasks=base, runs=3) is the usual form; without tasks= the first run draws the task set (from seeds= when given) and the rest replay it. Between runs nothing changes but the agent’s own sampling (same tasks, faults, world state and seed), so a deterministic agent gives identical runs and a zero re-run band. All rows come back in one SimulationData (output= holds them all); search["eval_runs"] lists the rows and stop reason per run, and eval_variance(data.rows()) splits by eval_run on its own. Replayed rows keep the writer of the run they replay on writer_model and say lineage.replayed_from_run, so delta_report on two runs of one call sees one writer.
Watching and resuming a long run:
  • Progress goes to the whileai.simulations logger at INFO (and to stderr when nothing listens) every 10 finished rollouts or 10 s, re-rolls and losses counted as events too, so a run that only re-rolls still speaks: 120/2404 rollouts, 601 situations written, 1h2m elapsed, ~19h left, 96 re-rolled, 3 lost (3 agent error). on_progress= is a callable that receives the same numbers as a dict on every line, whatever the budget: rows (in data.rows() so far, resumed ones included), cap, landed (this call), resumed, rerolled and rerolled_by (agent_error, empty_reply, tool_markup), timed_out (the agent errors that were call timeouts), lost and lost_by, inflight, situations, elapsed_s. The run’s final counts are search["rollouts"], and warnings (plus a UserWarning) says so when more rollouts were re-rolled than landed, since the run then spent most of its time on calls that never became rows.
  • checkpoint: a JSONL path every row is appended to the moment it lands, so a killed run keeps its rows. Call again with the same checkpoint= and tasks= to resume: the rows on disk are loaded, a task with its repeats rows is skipped, one with fewer gets only the missing rollouts, and the returned SimulationData is the union (search["rollouts"]["resumed"] counts the loaded rows; lineage.resumed marks each). Without tasks= the rows on disk are loaded and count toward budget, and the run draws new situations for the rest. output= still writes the whole run at the end; checkpoint= is the file that survives a kill.
What steers the search and answers the tools:
  • grader: a callable judge. Its verdict steers the search the way a tool fault already does: a row the grader failed is re-rolled and its ask is mutated into new ones, so the budget moves toward what the grader catches, not only toward broken tools. A grader that fails reply-form rules is exactly the signal traces cannot give (measured with a 12-rule grader: every rule with a tool-result trigger was reproduced and every rule about the reply’s wording was not), and without a grader there is no verdict to steer by, so there is no switch to set: the grader is the switch. A graded failure is a reward under 0.5 (a 0 from a 0/1 judge, a failed verifier, a rubric below half); markers ride along on the row but do not aim on their own, since their direction differs per marker. search["mutation_aims"] counts the parents and the mutated rows per aim, world_fault and graded_failure. To grade beside the loop and still steer by tool faults alone, pass advanced={"mutate_graded_failures": False}.
  • execute: your world, a function (tool, arguments) -> result that answers every tool call for real, against your repo, database or service. Without it the mock world answers, which fits record-shaped tools and not code. Scheduled faults still apply first. whileai.simulations.generate.agents.current_rollout is a thread-local set before each rollout with prompt, rollout_index and seed, so execute can tell which run it is answering.
  • patience: how long the simulated person keeps answering the agent’s questions. "normal" (the default) always tries to answer the first question, and from the second on may walk away (35% on the second, 60% on each after that, drawn per thread so a seeded run reproduces); at any question the person may also leave when it asks for something they could not or would not know. "short" walks away sooner (60% then 90%); "endless" never walks away, so the person answers every question until the depth cap and no rubric criterion about asking can fail. The odds are a default, not a measurement: to ground them, fit a Kaplan-Meier hazard per question index on source traces and set the levels from it. A row the person left carries ended_by="user_left" and ends on the agent’s question; search["ended_on_question"] is \{"share", "n", "user_left"\}: of n rows, the share that ended on a question and how many of those the person left.
  • scaffold: generation-only guidance appended to the system prompt of the model-backed teacher during rollout (and to the scene writer). It never enters profile.policy, so exports and evals stay on the plain policy; it is ignored for callable agents. Measured to help some agents and hurt others, so there is no default: configure it per agent.
  • reproducible: the default, None, is True unless time_budget is set. True makes a seeded run bit-for-bit at any concurrency, apart from timing fields and per-invocation identity (with grader= every row’s lineage.scoring_run_id names that one scoring pass, a fresh id per call; pass run_id= to run_judge to pin it): each batch of rollouts finishes, and every verdict of the batch lands, before the next is chosen, so every round sees the same state. A slow rollout holds its batch, so uneven latency costs throughput; pass False to trade the same task set on every machine for that throughput. A clock turns it off on its own, since a clock stop lands wherever the run happens to be. It was False before 0.111: three runs of one seed at the default concurrency then drew three different task sets (59, 58 and 58 tasks from one budget=160), because which rows land before the cap depended on thread timing, and a lesson that wanted the same file on every machine had to pin concurrency=1. Either way the draw is a function of the seed and the inputs alone, not of the interpreter: every float sum on the row-selection path is correctly rounded (math.fsum), so one seed picks the same rows on CPython 3.10 through 3.13, and a golden-value test pins that draw. Releases before this fix used the builtin sum, whose algorithm changed in CPython 3.12, and could swap one row per run between minor versions (issue #410).
What every row records. sampling (temperature, max_tokens and model as the model backend resolved them), because a result is only comparable with its sampling settings on record (Lambert 2025, chapter Evaluation); a callable agent samples however it samples, so its rows carry sampling: None unless you pass sampling={...}, which is recorded as given. Three models can take part, the agent (agent= or backend=), the situation writer (simulator=) and the simulated user (user_model=), and every row names all three next to model_version: writer_model, user_model, and judge_meta.model once graded. When the agent model also wrote the situations or played the user, the run’s degraded list carries same_model and warnings says which call separates them, since training on a model’s own unfiltered output teaches it its own habits (Lambert 2025, chapter Synthetic Data and Distillation).
Last modified on September 22, 2026