> ## Documentation Index
> Fetch the complete documentation index at: https://docs.while.ai/llms.txt
> Use this file to discover all available pages before exploring further.
> ## Agent Instructions
> Install with `uv add whileai`; import as `import whileai as wai`.
> Run the offline path first (`simulator=False`, `wai.seeded_agent`, a callable judge); no key is needed for it.
> Report every pass rate with its interval and n, as `scored.pass_at` prints it.
# whileai
> A scientific post-training library for language models: SFT and RL on open models, with the measurement that says whether training helped. Your models, your keys, your compute.
**What you learn:** the loop, simulate to export, as one program. **Needs:** nothing. **Takes:** about twenty seconds on a laptop.
`whileai` is a post-training library for language models: SFT and RL on
open models, with the measurement that says whether training helped. It is
for AI researchers, ML engineers and applied-AI developers. The loop is
simulate, grade, measure with intervals, select, train, prove on a
held-out set, serve, and feed the new traces back in. Nothing in that loop
needs an account, and every default names the paper it came from.
wai is While's whale and the alias of the whileai SDK: `import whileai as wai`.
The data end of the loop is built for agents that call tools. Give it
an agent, or only the agent's tools and system prompt. It writes the
situations the agent might meet, runs the agent through them against a
world that fails on purpose, and hands back every conversation as a row
you grade with your own judge or a verifier.
```bash theme={"theme":"vitesse-dark"}
uv add whileai
```
The loop, offline, no key, about twenty seconds on a laptop:
```python theme={"theme":"vitesse-dark"}
import whileai as wai
@wai.tool # the function is the tool; its signature is the schema
def get_order(order_id: str) -> dict:
"""Look up an order by id."""
...
data = wai.simulate(
# a stand-in agent that misbehaves on a labeled share of rows
wai.seeded_agent([get_order]),
tools=[get_order],
system_prompt="Help customers with orders.",
simulator=False, # template situation writer: no model, no network
mode="rl",
repeats=4,
repeat_policy="fixed",
budget=64,
)
# any judge or verifier
scored = data.grade(judge=lambda row: {"reward": int(not row["seeded"])})
print(scored.pass_at)
# leaked answer keys dropped, the 20 to 80% band (DIFFICULTY_BAND),
# unanimous groups dropped, reward-hack scan
rows = scored.select(mode="rl")
rows.export("train.jsonl")
```
```text theme={"theme":"vitesse-dark"}
pass@1 0.67 [0.55..0.78] | pass^4 (pass_pow_k) 0.19 [0.00..0.38] | pass@4 1.00 [1.00..1.00] | headroom 0.33 (16 groups, k=4)
```
| Metric | Value | 95% interval | n | What it says |
| -------- | ----- | ------------ | ------------- | ----------------------------------------------------------------- |
| pass\@1 | 0.67 | 0.55 to 0.78 | 16 tasks, k=4 | how often the agent does the job on one try; bootstrap over tasks |
| pass^4 | 0.19 | 0.00 to 0.38 | 16 tasks, k=4 | how often all four tries pass; the reliability line |
| pass\@4 | 1.00 | 1.00 to 1.00 | 16 tasks, k=4 | at least one of four passes |
| headroom | 0.33 | | | pass\@4 minus pass\@1, the gap an RL update could close |
Swap the stand-in for your agent and the lambda for your judge and the
program is the same. The package then does the bookkeeping that is
tempting to skip and expensive to get wrong: pass rates with intervals,
difficulty bands for RL, a check that your judge agrees with people,
decontamination against your eval set, and a scan for rewards the policy
can game.
Two domains, kept apart. `import whileai as wai` is the library: simulate,
grade, measure, select, export, on your machine against your models, no
account needed. `whileai.platform` is the While platform: sign in, push
datasets, train and serve on hosted GPUs, track versions. Everything that
talks to while.ai lives there and nowhere else.
The program above, one step at a time, with what each line prints.
Name the model as a string, set the provider's own key, and see
exactly what stays on your machine.
A pass rate with an interval, a table of where the agent fails, and a
CI check that goes red when it gets worse.
GRPO or DPO on Modal, a GPU box or a notebook, with the paired delta
on a holdout at the end. One A10G, under fifteen minutes.
Never trained a model? [Learn](/learn) is seven lessons from zero, each
offline in under a minute, ending at the quickstart above.
## The defaults this run used
The program above sets no thresholds, so four defaults produced its
numbers. Each one is a named constant in
[`defaults.py`](https://github.com/whilehq/whileai-sdk/blob/main/whileai/simulations/defaults.py),
carries its source in a comment beside it, and takes an argument on the
call that uses it.
| Constant | Value | What it did here | Source |
| ----------------- | ---------- | ----------------------------------------------- | --------------------------------------------------------------------------------- |
| `CI_LEVEL` | 0.95 | the interval on every number in the table above | Miller 2024 \[1] |
| `BOOTSTRAP_DRAWS` | 2000 | resamples behind each interval | Efron and Tibshirani put the floor at 1000; above it, convention, untested |
| `DIFFICULTY_BAND` | (0.2, 0.8) | which tasks `select(mode="rl")` kept | Lambert 2025, chapter [Reasoning](https://rlhfbook.com/c/07-reasoning); DAPO \[2] |
| `HACK_THRESHOLD` | 0.3 | the reward-hack scan `select` ran | Gao et al. \[3] |
### References
1. Miller, E. [Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations](https://arxiv.org/abs/2411.00640). 2024.
2. Yu, Q. et al. [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://arxiv.org/abs/2503.14476). 2025.
3. Gao, L. et al. [Scaling Laws for Reward Model Overoptimization](https://arxiv.org/abs/2210.10760). 2022.
## The loop
| Step | Call | What it computes |
| ------------------ | ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| Simulate | `simulate(agent, tools=, system_prompt=, mode="rl", repeats=k)` | A covering array over tools, world state and user stance. k rollouts per prompt. Scheduled tool faults. |
| Grade | `data.grade(judge=)`, `verify.MathEqual`, `verify.CodeExec` | A reward per rollout under one contract. Verifiable rewards where a program can check. |
| Validate the judge | `judge_trust`, `judge_probes` | Agreement and Cohen's kappa against human gold. Length bias. Exploit probes. |
| Measure | `pass_at`, `delta_report`, `eval_variance`, `holdout_size` | pass\@1, pass^k, pass\@k with bootstrap intervals over tasks. A paired delta with a permutation p-value. Noise band. Power. |
| Select | `scored.select(mode="rl"\|"sft")`, `build_preference_pairs`, `curriculum` | The 20 to 80% difficulty band. Unanimous groups dropped. Rejection sampling. Length-matched pairs. |
| Guard | `decontaminate`, `hack_scan`, `trace_markers`, `HackMonitor` | Overlap with the eval set. Reward-feature correlation within task against a shuffle floor. Trajectory lies. |
| Train and export | `rows.export`, `export_environment`, `platform.train`, `platform.serve` | Loss masks. A `verifiers` environment for GRPO. Hosted LoRA SFT, GRPO, DPO, RM. |
Where each method comes from, with numbered references, is on
[the engine](/concepts/engine). The calls in the order a run happens are
on [the five calls](/reference/five-calls). Every public call, with its
signature and docstring, is under API in the sidebar; those pages are
generated from the package on each release, so they cannot drift.
## Guides
How the simulator thinks and why.
A stable way of talking, in the weights.
A coding agent improves the prompt, tools and loop on your own traffic, gated on days it never saw.
The gap between the training reward and the eval you care about.
Private data, actions, sends: can they be turned against the owner?
What the hosted service adds on top of the library, and where the line is.
Runnable, one folder each, offline where the method allows.
One recent post-training paper per recipe, under an hour on one GPU,
with the number it moved and the number it did not.
## Next
Run `uv add whileai`, then open
[the quickstart](/get-started/quickstart): it is the program above one
line at a time, with what each line prints.