The test is not training data. It is the only piece the model never sees,
which is why a score on it means something [1].
The problem it solves
An agent’s failures are specific: it hands off too early on one kind of request, invents an order number under one kind of pressure. Fixing that in the weights needs enough varied examples of the situation done right that the model learns the behavior, not the example. Writing those by hand is the expensive part. The simulator replaces the writing, not the judgment.Two ways in
Describe the behavior. One sentence starts it: “a personal finance assistant that confirms before it moves money.” The SDK drafts the tools, builds a world, writes the people, runs the conversations. Point at the agent’s traces. Graded traces, OpenTelemetry spans included, become a picture of which situations fail, which are new since the last model version, and which stopped failing. That picture sets the generation budget. Traces reproduce tools, faults and world states; a failure that lives in the wording of a reply has no trigger in the world, so passgrader= and the search mutates on graded failures too.
Both paths use the same engine. Nothing in the agent changes; the SDK reads
what it does.
The loop
1
Watch
See where the agent fails at work. Live traffic is read into a map of
what the agent handles well and what it does not.
2
Simulate
Run it through realistic scenarios. The weak spots become thousands of
rollouts, with tools that break on purpose and customers who do not
cooperate.
3
Train
Every rollout gets a reward, and the ones that pass become SFT, DPO or
GRPO data for the next version of the model.
4
Prove
Evaluate on a held-out set. A paired before and after per task
category, with a 95% confidence interval [1].
How the simulator thinks
Situations are coordinates, not prompts. A thousand user requests from a model are a thousand variations of the same polite ask. The SDK declares six axes (tool, policy rule, user stance, world state, tool condition, history) and renders points in that space as a pairwise covering array: every pair of axis values appears together at least once, because most real failures are two things interacting [2]. On a cold start nine in ten fault cells flip to success, so the tool-condition axis is sampled, not covered, unless you raisefault_rate or pass prefer_success=False. data.coverage["pairwise"]
holds pairs_planned, pairs_covered and fraction: a 64-row offline run
plans 381 pairs and covers 139, and that 0.36 is arithmetic, not a failed
eval. Policy coverage is a different question; coverage_gap(asks, tools=..., system_prompt=...) answers it.
The next situation comes from what worked. Five writers fill the grid.
Each batch, the writers that found new behavior get more of the budget, and
generation stops when the squares stop filling [3].
People are sampled, not described. A coordinate says the customer is in
a hurry and the order was already cancelled. A second layer decides how they
write: lowercase, clipped, sarcastic. The writer sees an aside in prose, not
the labels, because a model told to be terse writes an essay about being
terse. The same person shows up on turn five that showed up on turn one.
The world answers honestly. Tool calls go to a simulated world that is
deterministic per seed, returns records shaped like the tool’s own schema,
remembers what it created, and says no. An unknown identifier is not found.
An argument that echoes the schema (“first name”, user@example.com) is
refused with a hint. A world that never says no teaches an agent that never
expects it.
Grading is the customer’s authority. Rows come back ungraded. The
deterministic conduct checks catch structural failures (an action claimed
without a tool call, an identifier the person never gave, success declared
after a failed call); then your grader decides what good means. Two things
are insisted on, because a judge is a reward model. The hosted grader (Phi-4)
is a different family from the hosted policy (Qwen), since a judge grading
its own writing prefers it [4]. And every label says who made it: the hosted
grader stamps model, rubric hash and settings; a custom judge passes
data.grade(judge=..., version=...). The judge is measured, not trusted:
hand-label a sample, attach_labels(rows, labels, kind="human"), and
judge_agreement reports agreement, kappa and pass_when_gold_fail, the
rate at which the judge passed a row you failed [4]. A gold_reward column
with no author is reported as unmeasured.
Failure is loud. When the hosted writer fails, the offline template
writer takes over and data.degraded carries generator_fallback; a run
with no rows keeps the writer’s last error in data.search["writer_errors"].
A dataset that looks real and is not is worse than none.
Which model runs it
Four roles take their own model: the agent (agent=), the situation writer
(simulator=), the simulated person (user_model=) and the judge (spec=
on grade()). All take the same spec.
claude-sonnet-5, claude-opus-5) that no longer take a sampling
temperature: the backend drops the field and retries when the API asks it to.
Omitting agent= runs the While-hosted model on your account key. One model
in two roles is the regime to avoid: data.degraded then carries
same_model and warnings names the call that separates them [4].
spec="typesafe:jev-latest" grades with a decision model: the verdict is a
probability, failure_class is the judge’s own choice over the failure
vocabulary, and judge_meta.confidence within DECISION_UNSURE_BAND (0.1)
of even marks the row unsure. It cannot play the agent, writer or user.
What you get
A JSONL file of chat-format conversations with tool schemas. Each row carries its situation (axes, world state, scheduled faults), persona tags, and once graded, its reward and reason. From there:
scored.rows, a slice of it, passes(), what
decontaminate kept) carry the run’s system prompt and tool schemas, so
select(rows).export(path) writes both; a plain list carries neither,
so export takes system_prompt= and tools= and warns when a
tool-calling file would go out without its schema.
With logprobs=True every agent turn also carries the summed log-probability
of its tokens, their count, the per-token list when the backend returns one,
policy_version and the sampling settings: what an off-policy correction
and a KL to a reference model need.
Hugging Face, both directions
Two routes to the Hub. The local one uses your own token and never calls the platform:export(..., push_to=) uploads the file it just wrote, and
wai.hub.push uploads a file, an adapter directory or rows you already
hold. Repos are private until you say otherwise. Needs HF_TOKEN (or
hf auth login) and pip install 'whileai[hf]'.
WHILEAI_API_KEY or
wai login.
train, holdout, eval). A push
replaces the old parts, the commit message carries the delta, whileai.json
keeps the history, and load_dataset(repo, split, revision="zp-ds_...")
loads exactly one push.
Return shapes
simulate returns SimulationData; evaluate and grade return
ScoredData. .rows and .rows() both work; print scored.warnings
before any number. Field by field: Evals.
Proving it
After training, the new model answers the held-out set. So does the old one. The report is a pass@1 per task category, before and after, so a gain on refunds cannot hide a loss on cancellations. Nothing counts as better until the held-out interval excludes zero [1]. The first public run of this method is written up in We trained a support agent to hand customers off at the right time: a 15-dollar simulated set took a telecom support agent, one that runs a phone company’s support chat with real account tools, from 5% to 30% on a public benchmark.What it is not
Not ground truth. Every row is a simulation kept by a grader; review it as you would a contractor’s work. The world is not your database and the people are not your customers. The value is coverage, variety, and honesty about both.Where it goes next
The same simulator serves as a live environment for on-policy RL: the trainer drives the policy, the SDK supplies situations, world, person and reward.export_environment(data, out, reward=...) writes an installable
verifiers environment with train and holdout splits, the world dials in
spec.json, and the 20 to 80 percent difficulty band [5, 6].
References
- Lambert, N. Reinforcement Learning from Human Feedback, chapter Evaluation. 2025.
- Kuhn, D. R., Wallace, D. R., Gallo, A. M. Software Fault Interactions and Implications for Software Testing. IEEE TSE 30(6), 2004.
- Lehman, J., Stanley, K. O. Abandoning Objectives: Evolution Through the Search for Novelty Alone. Evolutionary Computation 19(2), 2011.
- Zheng, L. et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023.
- Shao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. 2024.
- Yu, Q. et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. 2025. Dynamic sampling; the 20 to 80 band is the same rule with a margin for k=8 noise, and Lambert 2025, chapter Reasoning, measures it from sixteen rollouts.
- Touvron, H. et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. 2023. The 8-gram contamination rule.
What to run next
recipes/01-simulate/bring-your-own-agent
is the shortest version of the loop above, offline and in seconds;
recipes/03-select/prime-intellect-rl
is the verifiers export, and needs a key.