> ## Documentation Index
> Fetch the complete documentation index at: https://docs.while.ai/llms.txt
> Use this file to discover all available pages before exploring further.
> ## Agent Instructions
> Install with `uv add whileai`; import as `import whileai as wai`.
> Run the offline path first (`simulator=False`, `wai.seeded_agent`, a callable judge); no key is needed for it.
> Report every pass rate with its interval and n, as `scored.pass_at` prints it.
# The engine
> How simulate() makes evals and training data in eight steps: a pairwise covering array over situation axes, a novelty-weighted search, a sandbox world with failure modes, and a judge validated before training. With references.
Cover the situations. Run them in a sandbox that can fail. Grade with a
checked judge. Eight steps from an agent definition to a proven delta; how
the simulator thinks and why is [How it works](/concepts/how-it-works). The
one-page PDF is at
[while.ai/while-simulation-engine.pdf](https://while.ai/while-simulation-engine.pdf).
## Eight steps
| # | Step | What happens | Code |
| -- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------- |
| 01 | Axes | What varies: tool, policy rule, user stance, world state, tool condition, history. A situation is a point in that space, not a prompt. | `generate/scenarios.py` |
| 02 | Cover | A pairwise covering array: every pair of axis values co-occurs at least once, because most failures are two-factor interactions \[6]. `data.coverage["pairwise"]` reports `pairs_planned`, `pairs_covered`, `fraction`: grid coverage, not policy coverage (`coverage_gap` is that). | `generate/coverage.py` |
| 03 | Search | Five arms fill the grid: `structured` 42%, `llm_guided` 42%, `open_ended` 10%, `behavior_targeted` 3%, `failure_mutation` 3%. Each batch, `w *= 1 + 0.5 * yield` of new behavior signatures and cells, renormalized, floored and capped. Novelty search \[7]. | `generate/scenarios.py`, `generate/generator.py` |
| 04 | World | Tools answer from schema-shaped state, deterministic per seed. Unknown id: not found. Schema-echo argument: refused. Every dial is a `WorldOptions` field with its reason in `defaults.py`. | `world/sandbox.py`, `defaults.py` |
| 05 | Rollout | The agent on N situations x n phrasings x k samples. `logprobs=True` keeps each row's log-probabilities, policy version and sampling settings. | `run/engine.py` |
| 06 | Grade | Deterministic conduct rules, then your judge, scored against gold labels before its grades count. | `score/grading.py`, `score/judge_trust.py` |
| 07 | Cut | SFT rows (reward=1, loss mask on agent turns), DPO pairs with margin, GRPO groups in the 20 to 80 percent band \[4, 5], or a reward-model set. | `score/optimize.py`, `score/publish_gate.py`, `export.py` |
| 08 | Delta | Re-run held-out tasks after training: a paired difference per task, bootstrap interval, sign-flip permutation p \[1]. | `score/delta.py`, `score/stats.py` |
Steps 02, 03, 04 and 07 are ours. The rest is the literature. Code paths
are relative to
[whileai/simulations/](https://github.com/whilehq/whileai-sdk/tree/main/whileai/simulations).
## Measurement
| Number | What it is | Code | | |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------- | -------------------------------------------------------------------------------------------------------------------- | -------------------- |
| **pass\@1 / pass^k / pass\@k** | Headline, reliability, RL headroom. Unbiased estimators over k samples per task \[2, 3]. | `score/passat.py` | | |
| **Intervals** | Bootstrap over tasks, not rollouts. Before and after is a paired difference with a sign-flip permutation p \[1]. Ship when the interval excludes zero. | `score/stats.py` | | |
| **Judge** | Agreement and kappa against gold labels, Wilson interval, held-out halves, length perturbation. A different model family than the policy \[8]. | `score/judge_trust.py` | | |
| **Hack scan** | \`Var(r) = E\[Var(r | task)] + Var(E\[r | task])\`; only the first term is GRPO gradient. The top within-task feature is compared to a permutation floor \[9]. | `score/hack_scan.py` |
The held-out tasks are the only number that counts. Everything else on
this page exists to make that number mean something.
## Questions
**Importance sampling?** No. We cover the failure space rather than
estimate production. A row carries the policy version, and with
`logprobs=True` the log-probabilities an off-policy trainer needs to form
the truncated ratio `exp(log pi_new - log pi_old)` itself \[10, 11].
**SFT or RL?** Both, from the same graded rows. Reward=1 rows for SFT,
pairs for DPO, groups for GRPO, everything for a reward model.
**The judge is another LLM.** Yes. So it is measured against gold labels,
probed with known hacks, versioned by rubric hash, and drawn from a
different model family than the policy \[8].
**How do you know training helped?** `delta_report`: paired before and
after on held-out tasks with a bootstrap interval. A `must_not_regress`
marker whose interval sits below zero fails the run; markers such as
argument grounding catch what pass\@1 hides \[1].
## References
1. Lambert, N. [Reinforcement Learning from Human Feedback](https://rlhfbook.com), chapter [Evaluation](https://rlhfbook.com/c/16-evaluation). 2025.
2. Chen, M. et al. [Evaluating Large Language Models Trained on Code](https://arxiv.org/abs/2107.03374). 2021.
3. Yao, S. et al. [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://arxiv.org/abs/2406.12045). 2024.
4. Shao, Z. et al. [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://arxiv.org/abs/2402.03300). 2024.
5. Yu, Q. et al. [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://arxiv.org/abs/2503.14476). 2025.
6. Kuhn, D. R., Wallace, D. R., Gallo, A. M. [Software Fault Interactions and Implications for Software Testing](https://doi.org/10.1109/TSE.2004.24). IEEE TSE 30(6), 2004.
7. Lehman, J., Stanley, K. O. [Abandoning Objectives: Evolution Through the Search for Novelty Alone](https://doi.org/10.1162/EVCO_a_00025). Evolutionary Computation 19(2), 2011.
8. Zheng, L. et al. [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685). NeurIPS 2023.
9. Gao, L., Schulman, J., Hilton, J. [Scaling Laws for Reward Model Overoptimization](https://arxiv.org/abs/2210.10760). ICML 2023.
10. Schulman, J. et al. [Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347). 2017.
11. Noukhovitch, M. et al. [Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models](https://arxiv.org/abs/2410.18252). ICLR 2025.
## What to run next
[`recipes/01-simulate/bring-your-own-agent`](https://github.com/whilehq/whileai-sdk/tree/main/recipes/01-simulate/bring-your-own-agent)
runs these eight steps on a callable of your own, offline and in seconds.