> ## Documentation Index
> Fetch the complete documentation index at: https://docs.while.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Install with `uv add whileai`; import as `import whileai as wai`.
> Run the offline path first (`simulator=False`, `wai.seeded_agent`, a callable judge); no key is needed for it.
> Report every pass rate with its interval and n, as `scored.pass_at` prints it.

# Hint distillation: imitate what passed, correct what failed with a hint

> Paper: Learning from Real-World Experience, Perplexity Research (Zhang et al.), Perplexity blog, September 2026.

<Note>The scripts are in the repository, not in the installed package. Clone it,
then `cd recipes/papers/hint-distill` before running the commands below. [Browse this recipe on GitHub](https://github.com/whilehq/whileai-sdk/tree/main/recipes/papers/hint-distill).</Note>

**Paper:** Learning from Real-World Experience, Perplexity Research (Zhang et al.), Perplexity blog, September 2026. [https://www.perplexity.ai/hub/blog/learning-from-real-world-experience](https://www.perplexity.ai/hub/blog/learning-from-real-world-experience)
**Book:** rejection sampling keeps the replies a judge accepts and fine-tunes on them; this recipe tests whether the rejected replies can teach too \[2].
**Claim:** replies from successful sessions get ordinary cross-entropy, and a reply that caused a tool error or a user correction gets a short hint grounded in what the model already knew; training the model toward what it predicts when it reads the hint reduces those mistakes (Perplexity: 21% fewer tool-call failures in a live A/B test) \[1].
**The change:** on the same batches as plain rejection-sampling fine-tuning (RFT), 46 failed replies with a checked hint also get forward KL toward the same model reading the hint (lambda 1).

## Recipe

1. Base: `Qwen/Qwen3-4B`, thinking on. Data: the text-to-SQL shop of [`../../04-train/text-to-sql`](/recipes/04-train/text-to-sql), the first 480 train tasks and the first 120 held out by a hash of the task id (the split of its `train_grpo_modal.py --limit 480`).
2. Experience: the base answers each train task 4 times (temperature 0.7, top\_p 0.95, 1,536 tokens), and the SQL verifier judges each reply: 870 passed, 279 ran but returned the wrong rows, 29 hit a Postgres error, 742 were cut off at the token limit.
3. Hints: each task's failed, complete replies (up to 2 distinct queries) get a hint from Claude Sonnet 5, which sees the gold query only to find the mistake. Claude Haiku 4.5, which never sees the gold, rejects any hint whose claims do not follow from the question, the schema and the error message. 46 of 192 passed; most rejects were answers that differ from the gold only by a choice the question leaves open (which columns, which reading).
4. Baseline: RFT. LoRA rank 16 on the 870 passing replies, 2 epochs, 116 steps of 16, lr 1e-4 cosine, loss `(L_CE + lambda * L_KL) / CE tokens in the batch` at lambda 0.
5. Recipe: the same loop and batches at lambda 1: the 46 hinted replies get forward KL over the full vocabulary from the model with the hint in its context (detached) to the model without it, teacher-forced on the recorded tokens. A third arm puts the KL on the answer after `</think>` only.
6. Eval: pass\@1 on the same 120 held-out tasks, 4 samples each, paired by task with a 95% bootstrap interval. GRPO on the same split is the reference row.

## Run

```bash theme={"theme":"vitesse-dark"}
python recipe.py --selftest        # free: the pure parts
python recipe.py                   # every stage, ~51 H100 min + ~$1 of Bedrock, ~$5
python recipe.py --stage hints     # one stage: experience | hints | validate | train | results
# the GRPO reference row, ~51 H100 min, ~$4:
cd ../../04-train/text-to-sql && modal run --detach train_grpo_modal.py --spawn --thinking --steps 50 --gpu H100 --learning-rate 2e-5 --beta 0.01 --max-completion-length 1536 --use-vllm --prompts-per-step 8 --micro-batch 2 --limit 480
```

The hint stage calls Bedrock with your AWS credentials (`AWS_REGION`, default `us-east-1`).

## Result

| Arm                                | pass\@1 | 95% CI          | Says "hint" with no hint | Steps | GPU min |
| ---------------------------------- | ------- | --------------- | ------------------------ | ----- | ------- |
| Base, no training                  | 0.544   | \[0.463, 0.625] | 0%                       | 0     | 2.6     |
| Baseline: RFT (lambda 0)           | 0.529   | \[0.444, 0.608] | 0%                       | 116   | 12.0    |
| Recipe: RFT + hint KL, whole reply | 0.521   | \[0.431, 0.602] | 38%                      | 116   | 12.4    |
| RFT + hint KL, answer only         | 0.533   | \[0.452, 0.617] | 0%                       | 116   | 12.2    |
| GRPO, execution reward (reference) | 0.558   | \[0.477, 0.644] | 0%                       | 50    | 50.6    |

Recipe vs baseline: -0.01 \[-0.04, +0.02]. Verdict: unresolved. One training seed per arm; a second seed on each arm, passed as `train_runs=`, would resolve it. Against GRPO the recipe is -0.04 \[-0.08, -0.00].

Before any training, the hints carry signal: the base fixes 51% of the hinted failures with the hint in its prompt and 28% without (SQL errors 19% to 35%, wrong results 36% to 64%), the same size of gain the paper reports.

## Checks

| Check                                                            | Source                                       | Result                                                                                                           |
| ---------------------------------------------------------------- | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| Eval noise: the base evaluated 2 times, `eval_variance` run\_std | \[3]                                         | run\_std 0.007 from 2 re-runs (0.533 and 0.544); any delta under the band is noise                               |
| Holdout is clean: `decontaminate(train, against=holdout)`        | \[4]                                         | 0 train rows dropped                                                                                             |
| Reward is a program, not a judge                                 | \[4]                                         | the eval executes the query on the seeded Postgres and compares result sets; the LLMs only write and check hints |
| Proxy vs target: `wai.compare(proxy=)`                           | \[5]                                         | over\_optimized false: no training reward, the eval is the target                                                |
| Length: mean completion tokens before -> after, per arm          | \[5]                                         | baseline 1018 -> 1021, recipe 1018 -> 1030                                                                       |
| Hack scan on the last training batch: `hack_scan`                | \[5]                                         | not run                                                                                                          |
| Pinned: seed, torch, transformers, peft, vllm                    | [the contract](/recipes/papers#the-contract) | seed 17, torch 2.7.1, transformers 4.54.0, peft 0.16.0, vllm 0.10.0                                              |

## Climb

| Round | What changed                           | pass\@1 | vs previous      |
| ----- | -------------------------------------- | ------- | ---------------- |
| 1     | as the paper: KL over the whole reply  | 0.521   | -0.01 vs RFT     |
| 2     | KL on the answer after `</think>` only | 0.533   | +0.01 vs round 1 |

## Learned

* On a thinking model, KL toward a hinted teacher over the thinking tokens teaches the model to invent a hint: 38% of held-out replies say "the hint says", and those pass 45% against 56% for the rest. Putting the KL on the answer only takes that to 0%.
* The hints work, but a strict grounding check leaves few of them: 46 corrections beside 870 imitations did not move pass\@1 against RFT, and both trail GRPO, which trains on the 39% of replies that run out of tokens.
* Next: a multi-turn agent that sees real tool errors mid-session, where hints come from the environment and there are more of them.

Verified 2026-09-22, whileai 0.125, torch 2.7.1 + peft 0.16.0 + vllm 0.10.0 on one H100. The run used this code as separate scripts; `recipe.py` is the same stages in one file, checked by `--selftest`. Run page: [https://while.ai/platform/runs?agent=text-to-sql-shop](https://while.ai/platform/runs?agent=text-to-sql-shop)

## References

1. Perplexity Research (Zhang, S., Wu, H., Wang, C. et al.). Learning from Real-World Experience. Perplexity blog, 2026-09-21. [https://www.perplexity.ai/hub/blog/learning-from-real-world-experience](https://www.perplexity.ai/hub/blog/learning-from-real-world-experience)
2. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter *Rejection Sampling*.
3. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter *Evaluation*.
4. Lambert, N. et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124, 2024.
5. Gao, L., Schulman, J., Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML 2023. arXiv:2210.10760.