> ## Documentation Index
> Fetch the complete documentation index at: https://docs.while.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Install with `uv add whileai`; import as `import whileai as wai`.
> Run the offline path first (`simulator=False`, `wai.seeded_agent`, a callable judge); no key is needed for it.
> Report every pass rate with its interval and n, as `scored.pass_at` prints it.

# A pass rate without an interval is a guess

> Lesson 4. pass@1, the interval around it, pass^k and headroom, and how many tasks a real result needs.

The agent passed 58% of tasks. Run the same thing on sixteen different
tasks and it passes 50%, or 65%. Neither run was wrong. Sixteen tasks is
a small sample of all the asks your customers could send, and a small
sample bounces. The interval is the honest way to say how much. If two
numbers have overlapping intervals, they are not different, whatever the
means say.

## The mechanism

**pass\@1** is the share of tasks the agent gets right on one try. It is
the headline.

The brackets after it are a **95% confidence interval**: the range the
true rate very likely sits in, given how many tasks you ran. The library
computes it by resampling the tasks many times with replacement and
reading off where the middle 95% of the results land. That method is the
**bootstrap**, and it is done over tasks, not over the four tries of a
task, because four tries of the same ask are not four independent facts.

Four tries per task give two more numbers for free.

| Number   | Plain words                 | What it is for                                                               |
| -------- | --------------------------- | ---------------------------------------------------------------------------- |
| pass^4   | All four tries passed       | Reliability. What you can promise a customer.                                |
| pass\@4  | At least one of four passed | What the model can do on a good day.                                         |
| headroom | pass\@4 minus pass\@1       | The gap training could close. The model already knows how, some of the time. |

The bigger question is how many tasks you need. Fewer tasks, wider
interval, and a wide interval swallows real gains. `holdout_size` answers
it: to see a 10-point gain from a 60% base with four tries per task, you
need about 89 tasks. Most evals people run are smaller than that, which
is why most "it got better" claims are noise.

## Run it

The judge is lesson 3's, unchanged.

```python theme={"theme":"vitesse-dark"}
import json
import re

import whileai as wai
from whileai.simulations import holdout_size


@wai.tool
def get_order(order_id: str) -> dict:
    """Look up an order by id."""
    ...


ORDER_ID = re.compile(r"\bORD-\d+\b")
SUCCESS = {"ok", "created", "success", "done", "updated", "deleted"}
CLAIMED = re.compile(r"\b(done|went through|completed|succeeded|confirmed)\b", re.I)


def judge(row):
    """1 when the agent looked the order up, led with it, and claimed nothing the tool did not say."""
    messages = row["messages"]
    asked = set(ORDER_ID.findall(next(m["content"] for m in messages if m["role"] == "user")))
    calls = [c for m in messages if m["role"] == "assistant" for c in m.get("tool_calls") or []]
    if not asked:  # no id to look up: ask for one, do not guess
        return {"reward": int(not calls)}
    call = (calls[0].get("function") or calls[0]) if calls else {}
    args = call.get("arguments") or {}
    args = json.loads(args) if isinstance(args, str) else args
    if call.get("name") != "get_order" or args.get("order_id") not in asked:
        return {"reward": 0}
    reply = messages[-1]["content"] if messages[-1]["role"] == "assistant" else ""
    if args["order_id"] not in re.split(r"(?<=[.!?])\s", reply.strip())[0]:
        return {"reward": 0}  # the first sentence is the result, not a preamble
    result = next((m["content"] for m in messages if m["role"] == "tool"), "{}")
    status = json.loads(result).get("status")
    return {"reward": int(status in SUCCESS or not CLAIMED.search(reply))}


data = wai.simulate(
    wai.seeded_agent([get_order]),
    tools=[get_order],
    system_prompt="Help customers with orders.",
    simulator=False,
    mode="rl",
    repeats=4,
    repeat_policy="fixed",
    budget=64,
    seed=0,
)
scored = data.grade(judge=judge)

print(scored.pass_at)
print(holdout_size(0.10, base=0.6, k=4)["n_tasks"], "tasks to prove a 10-point gain")
```

```text theme={"theme":"vitesse-dark"}
pass@1 0.58 [0.38..0.77] | pass^4 (pass_pow_k) 0.31 [0.12..0.56] | pass@4 0.75 [0.50..0.94] | headroom 0.17 (16 groups, k=4)
89 tasks to prove a 10-point gain
```

Read the line. On one try the agent does the job 58% of the time, and
with sixteen tasks the true rate could be anywhere from 38% to 77%. All
four tries pass on only 31% of tasks, so it is not reliable. At least
one try passes on 75% of tasks; the other quarter are asks that name no
order, where the stand-in guesses an id every time and never passes. The
headroom of 17 points is what a training run has to work with on the
tasks it already sometimes gets right.

## Where it comes from

1. Chen, M. et al. Evaluating Large Language Models Trained on Code.
   arXiv:2107.03374, 2021. The pass\@k estimator.
2. Miller, E. Adding Error Bars to Evals. arXiv:2411.00640, 2024. Why an
   eval number needs an interval, and why it is over questions, not
   samples.
3. Efron, B., Tibshirani, R. J. An Introduction to the Bootstrap. Chapman
   and Hall, 1993.
4. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501,
   2025\. Chapter *Evaluation*.

## Next

[The test the model never sees is the only score that counts](/learn/the-held-out-set):
which tasks you are allowed to measure on.