Quickstart

What you learn: simulate a stand-in agent, grade it with a one-line judge, read pass@1 with an interval, keep the rows worth training on. Needs: nothing. Takes: sixty seconds.

Running your own model? Your model and your key is the two-line version: the model as a string, the key in the provider’s usual environment variable.

First command: did my prompt rewrite help?

The question most people arrive with. Run the old prompt and the new one on the same questions, on a model on your machine, and get one verdict: the gain, its 95% range over tasks, and PASS only when the range clears zero. --demo runs a scripted model, so it needs nothing installed:

wai compare --demo

before-and-after check: 24 tasks x 4 replies x 3 runs per arm, model demo_model, temperature 0.7, seed 0 (same seed, same output)
  before d9bf2a5cae9d  pass@1 0.55 [0.47..0.63]
  after  c5bdd38064fb  pass@1 0.85 [0.80..0.90]

pass_at_1: moved (+0.306, 95% +0.253..+0.358, 24 paired tasks)
PASS
eval noise: run_std 0.015, a delta under 0.035 is noise (t(df=4)=2.78 x run_std x sqrt(1/3 + 1/3); 3 eval runs before, 3 after)
answered: 100.0% before, 100.0% after
  pass_at_1                    0.549 -> 0.854  +0.306 [+0.253..+0.358]  up  (24 paired)  noise<0.035

With your prompts, your test set and a local model it is free: no key, no credits.

ollama pull qwen3:4b-instruct
wai compare --model ollama:qwen3:4b-instruct   --before old.txt --after new.txt --tasks tasks.jsonl --reward Numeric

tasks.jsonl is one {"prompt": ..., "reference": ...} per line, and --reward is any wai.verify class or module:function. The seed (0 unless you pass --seed) fixes every sampling draw and the bootstrap, so a rerun prints the same report. The same check in Python, here on a two-line stand-in so it runs anywhere:

import whileai as wai


def my_model(messages, seed):  # stand-in for wai.Ollama("qwen3:4b-instruct")
    careful = "final number only" in messages[0]["content"]
    return "The answer is 42" if careful or seed % 2 else "About forty."


report = wai.harness.compare(
    "You are a helpful assistant.",  # before: the prompt you ship
    "Work it out, then give the final number only.",  # after: the rewrite
    tasks=[{"prompt": f"Order {i}: what is 6 * 7?", "reference": "42"} for i in range(12)],
    reward=wai.verify.Numeric(),
    model=my_model,
)
print(report["verdict"], report["compare"]["metrics"]["pass_at_1"]["ci95"])

Both prompts draw reply j of a question on the same seed, the eval runs three times a side so the report can measure its own noise (one run reads INCONCLUSIVE), and the two row sets go to wai.compare unchanged: the interval and the verdict are its own. The before-and-after recipe has the null check (the old prompt on both arms reads NO DIFFERENCE).

The loop, offline

No key, no network. seeded_agent is a stand-in agent. It answers honestly most of the time and, on a labeled fraction of rollouts, does one thing wrong on purpose: hedges, flatters, or claims success after a tool failed. Each row records what it did in seeded, so you can check that your judge catches exactly those rows before you trust it on real ones.

Write a tool

A tool is a typed function. The signature is the schema, the docstring is the description. One is enough.

import whileai as wai


@wai.tool
def get_order(order_id: str) -> dict:
    """Look up an order by id."""
    ...

Simulate

simulator=False writes the customers from templates, so no model is called. mode="rl" with repeats=4 plays every ask four times, which is what pass@k needs.

data = wai.simulate(
    wai.seeded_agent([get_order]),
    tools=[get_order],
    system_prompt="Help customers with orders.",
    simulator=False,  # no model
    mode="rl",
    repeats=4,
    repeat_policy="fixed",
    budget=64,
)

Grade and read the number

Any callable that takes a row and returns a reward is a judge. This one uses the label the stand-in agent left behind.

scored = data.grade(judge=lambda row: {"reward": int(not row["seeded"])})
print(scored.pass_at)

pass@1 0.67 [0.55..0.78] | pass^4 (pass_pow_k) 0.19 [0.00..0.38] | pass@4 1.00 [1.00..1.00] | headroom 0.33 (16 groups, k=4)

Keep the rows worth training on

select runs the gates: replies that quote the answer key the grader was given, the 20 to 80% difficulty band, unanimous groups, duplicates, truncation, a scan for what the reward is really tracking. Print it and it says what each gate dropped and why. Four of the stand-in’s planted mistakes recite the answer key, and they never reach the training file.

rows = scored.select(mode="rl")
print(rows)
rows.export("train.jsonl")

rl selection: kept 27 of 64 rows
  band 20%..80% pass rate: 0 asks dropped (0 too easy, 0 too hard)
  unanimous groups dropped: 6; duplicates dropped: 27; truncated drop: 0
  privileged leaks dropped: 4
  groups kept: 10
  hack scan: train
  warning: 27 duplicate rollout(s) within 13 ask(s) dropped
  warning: reward punishes reply length (corr -0.74); check the judge before training
  warning: reward punishes boilerplate (corr -0.32); check the judge before training
  warning: reward punishes hedging (corr -0.37); check the judge before training
  warning: 3 rollouts per ask at the median; the floor is coarse below 4, re-scan at repeats>=8 before acting on a close call
  warning: Difficulty was measured from 4 rollouts per task, so a task's band assignment can be off by about ±0.3. Use repeats=16 for a firmer band (the count the 20-80 band is measured from, Lambert 2025, chapter Reasoning).

A histogram of tasks by pass rate at k=8: the 20 to 80 percent band is shaded and kept, the all-pass and all-fail bars are grey and dropped as unanimous groups, the bars just outside the band are dropped as too hard or too easy

The warnings are part of the answer. The judge here reads the label the stand-in left behind, and the planted mistakes add words, so shorter replies really are better; on a real judge that line means check whether it grades the job or the word count.

pass@1 is the pass rate over tasks with a bootstrap interval. pass^4 is how often all four rollouts of a task pass. Headroom is pass@4 minus pass@1, the gap an RL update could close.

Your model, your key

Every role in a run is a model behind an endpoint. Say which with a backend object; its repr tells you where the call goes and which key it uses. Set it once for the process, or pass it on the call.

wai.configure(
    agent=wai.OpenAI("gpt-4.1-mini", api_key="sk-..."),        # or OPENAI_API_KEY
    judge=wai.Anthropic("claude-haiku-4-5"),                   # ANTHROPIC_API_KEY
)
print(wai.settings)

data = wai.simulate(tools=[get_order], system_prompt="Help customers with orders.", mode="rl", repeats=8)
scored = data.grade(wai.Judge(rubric="Look up the order before answering."))

Settings(agent=openai:gpt-4.1-mini, judge=anthropic:claude-haiku-4-5, simulator=default (While hosted), api_key=unset, keys=openai)

wai.Endpoint("Qwen/Qwen3-4B", url="http://localhost:8000/v1") is any OpenAI-compatible server you run; wai.Ollama("llama3") needs no key. With nothing configured, every role uses the model While hosts, on the key from wai login. The full table is on Connect your agent.

Connect your agent

A callable, an endpoint, a backend object, or the hosted model. Plus traces to aim the run.

Evals

A pass rate with an interval, a table of where the agent fails, and a CI check that turns red when it gets worse.

How it works

How the engine picks situations, plays the customer, and breaks the tools on purpose.

The five calls

simulate, grade, trust the judge, select, push: the run in order.

Get started

Guides

Concepts

Recipes

Reference

First command: did my prompt rewrite help?

The loop, offline

Your model, your key

Next

Connect your agent

Evals

How it works

The five calls

​First command: did my prompt rewrite help?

​The loop, offline

​Your model, your key

​Next

Connect your agent

Evals

How it works

The five calls

First command: did my prompt rewrite help?

The loop, offline

Your model, your key

Next