Quickstart

What you learn: simulate a stand-in agent, grade it with a one-line judge, read pass@1 with an interval, keep the rows worth training on. Needs: nothing. Takes: sixty seconds.

Running your own model? Your model and your key is the two-line version: the model as a string, the key in the provider’s usual environment variable.

No key, no network. seeded_agent is a stand-in agent. It answers honestly most of the time and, on a labeled fraction of rollouts, does one thing wrong on purpose: hedges, flatters, or claims success after a tool failed. Each row records what it did in seeded, so you can check that your judge catches exactly those rows before you trust it on real ones.

Write a tool

A tool is a typed function. The signature is the schema, the docstring is the description. One is enough.

import whileai as wai


@wai.tool
def get_order(order_id: str) -> dict:
    """Look up an order by id."""
    ...

Simulate

simulator=False writes the customers from templates, so no model is called. mode="rl" with repeats=4 plays every ask four times, which is what pass@k needs.

data = wai.simulate(
    wai.seeded_agent([get_order]),
    tools=[get_order],
    system_prompt="Help customers with orders.",
    simulator=False,  # no model
    mode="rl",
    repeats=4,
    repeat_policy="fixed",
    budget=64,
)

Grade and read the number

Any callable that takes a row and returns a reward is a judge. This one uses the label the stand-in agent left behind.

scored = data.grade(judge=lambda row: {"reward": int(not row["seeded"])})
print(scored.pass_at)

pass@1 0.67 [0.55..0.78] | pass^4 (pass_pow_k) 0.19 [0.00..0.38] | pass@4 1.00 [1.00..1.00] | headroom 0.33 (16 groups, k=4)

Keep the rows worth training on

select runs the gates: replies that quote the answer key the grader was given, the 20 to 80% difficulty band, unanimous groups, duplicates, truncation, a scan for what the reward is really tracking. Print it and it says what each gate dropped and why. Four of the stand-in’s planted mistakes recite the answer key, and they never reach the training file.

rows = scored.select(mode="rl")
print(rows)
rows.export("train.jsonl")

rl selection: kept 27 of 64 rows
  band 20%..80% pass rate: 0 asks dropped (0 too easy, 0 too hard)
  unanimous groups dropped: 6; duplicates dropped: 27; truncated drop: 0
  privileged leaks dropped: 4
  groups kept: 10
  hack scan: train
  warning: 27 duplicate rollout(s) within 13 ask(s) dropped
  warning: reward punishes reply length (corr -0.74); check the judge before training
  warning: reward punishes boilerplate (corr -0.32); check the judge before training
  warning: reward punishes hedging (corr -0.37); check the judge before training
  warning: 3 rollouts per ask at the median; the floor is coarse below 4, re-scan at repeats>=8 before acting on a close call
  warning: Difficulty was measured from 4 rollouts per task, so a task's band assignment can be off by about ±0.3. Use repeats=16 for a firmer band (the count the 20-80 band is measured from, Lambert 2025, chapter Reasoning).

A histogram of tasks by pass rate at k=8: the 20 to 80 percent band is shaded and kept, the all-pass and all-fail bars are grey and dropped as unanimous groups, the bars just outside the band are dropped as too hard or too easy

The warnings are part of the answer. The judge here reads the label the stand-in left behind, and the planted mistakes add words, so shorter replies really are better; on a real judge that line means check whether it grades the job or the word count.

pass@1 is the pass rate over tasks with a bootstrap interval. pass^4 is how often all four rollouts of a task pass. Headroom is pass@4 minus pass@1, the gap an RL update could close.

Your model, your key

Every role in a run is a model behind an endpoint. Say which with a backend object; its repr tells you where the call goes and which key it uses. Set it once for the process, or pass it on the call.

wai.configure(
    agent=wai.OpenAI("gpt-4.1-mini", api_key="sk-..."),        # or OPENAI_API_KEY
    judge=wai.Anthropic("claude-haiku-4-5"),                   # ANTHROPIC_API_KEY
)
print(wai.settings)

data = wai.simulate(tools=[get_order], system_prompt="Help customers with orders.", mode="rl", repeats=8)
scored = data.grade(wai.Judge(rubric="Look up the order before answering."))

Settings(agent=openai:gpt-4.1-mini, judge=anthropic:claude-haiku-4-5, simulator=default (While hosted), api_key=unset, keys=openai)

wai.Endpoint("Qwen/Qwen3-4B", url="http://localhost:8000/v1") is any OpenAI-compatible server you run; wai.Ollama("llama3") needs no key. With nothing configured, every role uses the model While hosts, on the key from wai login. The full table is on Connect your agent.

Connect your agent

A callable, an endpoint, a backend object, or the hosted model. Plus traces to aim the run.

Evals

A pass rate with an interval, a table of where the agent fails, and a CI check that turns red when it gets worse.

How it works

How the engine picks situations, plays the customer, and breaks the tools on purpose.

The five calls

simulate, grade, trust the judge, select, push: the run in order.

Get started

Guides

Concepts

Recipes

Reference

Your model, your key

Next

Connect your agent

Evals

How it works

The five calls

​Your model, your key

​Next

Connect your agent

Evals

How it works

The five calls

Your model, your key

Next