whileai

What you learn: the loop, simulate to export, as one program. Needs: nothing. Takes: about twenty seconds on a laptop. whileai is a post-training library for language models: SFT and RL on open models, with the measurement that says whether training helped. It is for AI researchers, ML engineers and applied-AI developers. The loop is simulate, grade, measure with intervals, select, train, prove on a held-out set, serve, and feed the new traces back in. Nothing in that loop needs an account, and every default names the paper it came from. wai is While’s whale and the alias of the whileai SDK: import whileai as wai. The data end of the loop is built for agents that call tools. Give it an agent, or only the agent’s tools and system prompt. It writes the situations the agent might meet, runs the agent through them against a world that fails on purpose, and hands back every conversation as a row you grade with your own judge or a verifier.

uv add whileai

The loop, offline, no key, about twenty seconds on a laptop:

import whileai as wai


@wai.tool  # the function is the tool; its signature is the schema
def get_order(order_id: str) -> dict:
    """Look up an order by id."""
    ...


data = wai.simulate(
    # a stand-in agent that misbehaves on a labeled share of rows
    wai.seeded_agent([get_order]),
    tools=[get_order],
    system_prompt="Help customers with orders.",
    simulator=False,  # template situation writer: no model, no network
    mode="rl",
    repeats=4,
    repeat_policy="fixed",
    budget=64,
)
# any judge or verifier
scored = data.grade(judge=lambda row: {"reward": int(not row["seeded"])})
print(scored.pass_at)
# leaked answer keys dropped, the 20 to 80% band (DIFFICULTY_BAND),
# unanimous groups dropped, reward-hack scan
rows = scored.select(mode="rl")
rows.export("train.jsonl")

pass@1 0.67 [0.55..0.78] | pass^4 (pass_pow_k) 0.19 [0.00..0.38] | pass@4 1.00 [1.00..1.00] | headroom 0.33 (16 groups, k=4)

Metric	Value	95% interval	n	What it says
pass@1	0.67	0.55 to 0.78	16 tasks, k=4	how often the agent does the job on one try; bootstrap over tasks
pass^4	0.19	0.00 to 0.38	16 tasks, k=4	how often all four tries pass; the reliability line
pass@4	1.00	1.00 to 1.00	16 tasks, k=4	at least one of four passes
headroom	0.33			pass@4 minus pass@1, the gap an RL update could close

Swap the stand-in for your agent and the lambda for your judge and the program is the same. The package then does the bookkeeping that is tempting to skip and expensive to get wrong: pass rates with intervals, difficulty bands for RL, a check that your judge agrees with people, decontamination against your eval set, and a scan for rewards the policy can game. Two domains, kept apart. import whileai as wai is the library: simulate, grade, measure, select, export, on your machine against your models, no account needed. whileai.platform is the While platform: sign in, push datasets, train and serve on hosted GPUs, track versions. Everything that talks to while.ai lives there and nowhere else.

Quickstart

The program above, one step at a time, with what each line prints.

Your model and your key

Name the model as a string, set the provider’s own key, and see exactly what stays on your machine.

Evals

A pass rate with an interval, a table of where the agent fails, and a CI check that goes red when it gets worse.

Train on your own GPU

GRPO or DPO on Modal, a GPU box or a notebook, with the paired delta on a holdout at the end. One A10G, under fifteen minutes.

Never trained a model? Learn is seven lessons from zero, each offline in under a minute, ending at the quickstart above.

The defaults this run used

The program above sets no thresholds, so four defaults produced its numbers. Each one is a named constant in defaults.py, carries its source in a comment beside it, and takes an argument on the call that uses it.

Constant	Value	What it did here	Source
`CI_LEVEL`	0.95	the interval on every number in the table above	Miller 2024 [1]
`BOOTSTRAP_DRAWS`	2000	resamples behind each interval	Efron and Tibshirani put the floor at 1000; above it, convention, untested
`DIFFICULTY_BAND`	(0.2, 0.8)	which tasks `select(mode="rl")` kept	Lambert 2025, chapter Reasoning; DAPO [2]
`HACK_THRESHOLD`	0.3	the reward-hack scan `select` ran	Gao et al. [3]

References

Miller, E. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. 2024.
Yu, Q. et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. 2025.
Gao, L. et al. Scaling Laws for Reward Model Overoptimization. 2022.

The loop

Step	Call	What it computes
Simulate	`simulate(agent, tools=, system_prompt=, mode="rl", repeats=k)`	A covering array over tools, world state and user stance. k rollouts per prompt. Scheduled tool faults.
Grade	`data.grade(judge=)`, `verify.MathEqual`, `verify.CodeExec`	A reward per rollout under one contract. Verifiable rewards where a program can check.
Validate the judge	`judge_trust`, `judge_probes`	Agreement and Cohen’s kappa against human gold. Length bias. Exploit probes.
Measure	`pass_at`, `delta_report`, `eval_variance`, `holdout_size`	pass@1, pass^k, pass@k with bootstrap intervals over tasks. A paired delta with a permutation p-value. Noise band. Power.
Select	`scored.select(mode="rl"\|"sft")`, `build_preference_pairs`, `curriculum`	The 20 to 80% difficulty band. Unanimous groups dropped. Rejection sampling. Length-matched pairs.
Guard	`decontaminate`, `hack_scan`, `trace_markers`, `HackMonitor`	Overlap with the eval set. Reward-feature correlation within task against a shuffle floor. Trajectory lies.
Train and export	`rows.export`, `export_environment`, `platform.train`, `platform.serve`	Loss masks. A `verifiers` environment for GRPO. Hosted LoRA SFT, GRPO, DPO, RM.

Where each method comes from, with numbered references, is on the engine. The calls in the order a run happens are on the five calls. Every public call, with its signature and docstring, is under API in the sidebar; those pages are generated from the package on each release, so they cannot drift.

Guides

How it works

How the simulator thinks and why.

Character training

A stable way of talking, in the weights.

Harness optimization

A coding agent improves the prompt, tools and loop on your own traffic, gated on days it never saw.

Reward hacking

The gap between the training reward and the eval you care about.

Safety evals

Private data, actions, sends: can they be turned against the owner?

Platform

What the hosted service adds on top of the library, and where the line is.

Recipes

Runnable, one folder each, offline where the method allows.

Replicated papers

One recent post-training paper per recipe, under an hour on one GPU, with the number it moved and the number it did not.

Run uv add whileai, then open the quickstart: it is the program above one line at a time, with what each line prints.

Get started

Guides

Concepts

Recipes

Reference

Quickstart

Your model and your key

Evals

Train on your own GPU

The defaults this run used

References

The loop

Guides

How it works

Character training

Harness optimization

Reward hacking

Safety evals

Platform

Recipes

Replicated papers

Next

Quickstart

Your model and your key

Evals

Train on your own GPU

​The defaults this run used

​References

​The loop

​Guides

How it works

Character training

Harness optimization

Reward hacking

Safety evals

Platform

Recipes

Replicated papers

​Next

The defaults this run used

References

The loop

Guides

Next