> ## Documentation Index
> Fetch the complete documentation index at: https://docs.while.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Install with `uv add whileai`; import as `import whileai as wai`.
> Run the offline path first (`simulator=False`, `wai.seeded_agent`, a callable judge); no key is needed for it.
> Report every pass rate with its interval and n, as `scored.pass_at` prints it.

# SmolDataEnvs: data-analysis tasks with a reward that is a program

> SmolDataEnvs [1] is 5,394 questions about real Kaggle tables, each with one known answer.

<Note>The scripts are in the repository, not in the installed package. Clone it,
then `cd recipes/01-simulate/smol-data-envs` before running the commands below. [Browse this recipe on GitHub](https://github.com/whilehq/whileai-sdk/tree/main/recipes/01-simulate/smol-data-envs).</Note>

[SmolDataEnvs](https://huggingface.co/datasets/FineEnvs/SmolDataEnvs) \[1] is
5,394 questions about real Kaggle tables, each with one known answer. The
policy writes a Python program, the program runs next to the tables, and the
dataset's own grader compares the last line it prints to the gold. No model
grades anything, so the reward is the same on every run.

What you will learn: how to wrap an outside environment as a whileai
verifier, the two reward rules that keep a data-analysis reward honest
(an environment failure is `None`, not `0`; a shell `echo` earns nothing),
and how the graded rows become GRPO groups. You need nothing for the offline
run; the live run needs any model the SDK can call. Seconds offline, about a
minute live on 24 tasks.

## Run it

```bash theme={"theme":"vitesse-dark"}
uv add whileai
cd recipes/01-simulate/smol-data-envs
python run.py                                                       # offline: canned replies, no key
python run.py --agent openai:gpt-4.1-mini --split eval --limit 24   # live: your key, a few cents
```

The live run needs `pandas` and `pyarrow` in the same interpreter (the
programs import pandas; a few tasks read HDF5 and need `h5py`). It downloads
the split and each task's tables from Hugging Face into `raw/`, public and
with no token, once.

| flag            | default        | what it does                                                                                                                  |
| --------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `--agent`       | none (offline) | any SDK spec: `openai:<model>` (honors `OPENAI_BASE_URL`), `anthropic:<model>`, `bedrock:<id>@<region>`, `vllm:<model>@<url>` |
| `--split`       | `eval`         | `train` (5,000), `test` (250) or `eval` (144); tasks with no tables are dropped                                               |
| `--limit`       | 24             | first N tasks by `task_id`; `0` for the whole split                                                                           |
| `--k`           | 4              | rollouts per task, the GRPO group size                                                                                        |
| `--temperature` | 0.8            | sampling temperature; a group at 0 has no spread to learn from                                                                |

## The environment

```python theme={"theme":"vitesse-dark"}
from env import DataEnvReward, fixture_tasks, prompt_for, task_row_fields
from whileai.simulations.score.judging import run_judge

task = fixture_tasks()[2]  # How many respondents are older than 30 and employed?
program = """
import csv
rows = list(csv.DictReader(open("input/survey.csv")))
print(sum(1 for r in rows if int(r["age"]) > 30 and r["employed"] == "yes"))
"""
fence = "`" * 3
row = {
    "scenario_id": task["task_id"],
    "prompt": prompt_for(task),
    "final_text": f"{fence}python{program}{fence}",  # the model's reply
    **task_row_fields(task),  # gold under privileged.reference, tables, tolerances
}
scored = run_judge([row], DataEnvReward())  # a verifier honors the judge contract
print(scored.rows[0]["reward"], scored.rows[0]["reason"])
```

```text theme={"theme":"vitesse-dark"}
1 printed '<reference>': exact
```

* **The reward** (`DataEnvReward` in [env.py](https://github.com/whilehq/whileai-sdk/blob/main/recipes/01-simulate/smol-data-envs/env.py)) is a `Verifier`: 1
  when the printed answer matches under the dataset's grader, 0 when it does
  not, when the program crashed, or when the reply was a shell block.
* **Ungraded is not wrong.** A table that failed to download raises
  `VerifierError`, the row gets `reward=None`, `pass_at` skips it and
  `wai.select` drops it. Scoring it 0 would teach the policy that correct
  programs fail on an unlucky network.
* **Tolerances are the task's.** Integer answers carry `atol=0`; decimals
  carry their own `atol`/`rtol` (for example 0.05 and 1%). The grader
  ([grader.py](https://github.com/whilehq/whileai-sdk/blob/main/recipes/01-simulate/smol-data-envs/grader.py), vendored at the dataset's revision, MIT) tries
  exact, numeric, list, then Math-Verify.
* **The gold stays out of training files.** It rides in
  `privileged.reference`, which the training export never projects, and a
  reason that would quote it prints `<reference>` instead.
* **Isolation.** Programs run in a fresh temp directory, a subprocess with
  a 60 s timeout. That stops runaway loops; it is not a security boundary.
  Run a policy you do not trust in a container or a sandbox service.

## What you get

Offline, four fixture tasks walk every branch: right, right within the
tolerance (`5768` against `5768.04`), wrong value, crashed, shell, all-pass
(no gradient), and ungraded.

Live, Claude Haiku 4.5 on the first 24 eval tasks, k=4, one reply per
rollout (2026-09-25):

```text theme={"theme":"vitesse-dark"}
== why rollouts failed
    25  crashed: KeyError
    17  ran, wrong value
     4  crashed: UnicodeDecodeError
     2  crashed: ValueError
     1  timed out after 60s
     1  crashed: AttributeError

== pass@1 by difficulty (0 ungraded rollouts left out, not scored 0)
  tier     tasks  pass@1          95% CI
  easy         8    0.81      0.56..1.00
  medium      16    0.31      0.12..0.52
  all         24    0.48      0.30..0.66

== wai.select(mode='rl')
rl selection: kept 24 of 96 rows
  unanimous groups dropped: 18; duplicates dropped: 5; truncated drop: 0
  privileged leaks dropped: 0
  groups kept: 6
```

Haiku 4.5 solves 48% of these tasks in one shot (95% interval 30% to 66%,
tasks resampled; a second run of the same command gave 45%, 28% to 61%).
The biggest loss is `KeyError`: the program guesses a column name it never
looked at. Only 6 of the 24 tasks split between right and wrong, which is
where GRPO has something to learn; `select` also warns that 4 rollouts per
task place a task in the 20-80% band only to about 0.3, so train with
`--k 16`.

## Next

* Give the policy a look before it answers: a multi-turn agent with a code
  tool fixes most `KeyError` crashes. Wrap it with the
  [`bring-your-own-agent`](/recipes/01-simulate/bring-your-own-agent) contract and pass it as
  `--agent`.
* Measure whether the number means anything:
  [`is-your-eval-any-good`](/recipes/02-measure/is-your-eval-any-good).
* Train on the split groups: [`04-train/grpo`](/recipes/04-train/grpo).

## References

1. Adithya S K. SmolDataEnvs: verified data-analysis environments from
   Kaggle notebooks. Hugging Face dataset `FineEnvs/SmolDataEnvs`, 2026.
   Release notes: [FineEnvs PR #14](https://github.com/adithya-s-k/FineEnvs/pull/14).
2. Lambert et al. Tülu 3: pushing frontiers in open language model
   post-training (RLVR). [arXiv:2411.15124](https://arxiv.org/abs/2411.15124).