Hint distillation: imitate what passed, correct what failed with a hint

​

Recipe

1. Base: Qwen/Qwen3-4B, thinking on. Data: the text-to-SQL shop of ../../04-train/text-to-sql, the first 480 train tasks and the first 120 held out by a hash of the task id (the split of its train_grpo_modal.py --limit 480).
2. Experience: the base answers each train task 4 times (temperature 0.7, top_p 0.95, 1,536 tokens), and the SQL verifier judges each reply: 870 passed, 279 ran but returned the wrong rows, 29 hit a Postgres error, 742 were cut off at the token limit.
3. Hints: each task’s failed, complete replies (up to 2 distinct queries) get a hint from Claude Sonnet 5, which sees the gold query only to find the mistake. Claude Haiku 4.5, which never sees the gold, rejects any hint whose claims do not follow from the question, the schema and the error message. 46 of 192 passed; most rejects were answers that differ from the gold only by a choice the question leaves open (which columns, which reading).
4. Baseline: RFT. LoRA rank 16 on the 870 passing replies, 2 epochs, 116 steps of 16, lr 1e-4 cosine, loss (L_CE + lambda * L_KL) / CE tokens in the batch at lambda 0.
5. Recipe: the same loop and batches at lambda 1: the 46 hinted replies get forward KL over the full vocabulary from the model with the hint in its context (detached) to the model without it, teacher-forced on the recorded tokens. A third arm puts the KL on the answer after </think> only.
6. Eval: pass@1 on the same 120 held-out tasks, 4 samples each, paired by task with a 95% bootstrap interval. GRPO on the same split is the reference row.

​

Run

python recipe.py --selftest        # free: the pure parts
python recipe.py                   # every stage, ~51 H100 min + ~$1 of Bedrock, ~$5
python recipe.py --stage hints     # one stage: experience | hints | validate | train | results
# the GRPO reference row, ~51 H100 min, ~$4:
cd ../../04-train/text-to-sql && modal run --detach train_grpo_modal.py --spawn --thinking --steps 50 --gpu H100 --learning-rate 2e-5 --beta 0.01 --max-completion-length 1536 --use-vllm --prompts-per-step 8 --micro-batch 2 --limit 480

​

Result

​

Checks

​

Climb

​

Learned

• On a thinking model, KL toward a hinted teacher over the thinking tokens teaches the model to invent a hint: 38% of held-out replies say “the hint says”, and those pass 45% against 56% for the rest. Putting the KL on the answer only takes that to 0%.
• The hints work, but a strict grounding check leaves few of them: 46 corrections beside 870 imitations did not move pass@1 against RFT, and both trail GRPO, which trains on the 39% of replies that run out of tokens.
• Next: a multi-turn agent that sees real tool errors mid-session, where hints come from the environment and there are more of them.

​

References

1. Perplexity Research (Zhang, S., Wu, H., Wang, C. et al.). Learning from Real-World Experience. Perplexity blog, 2026-09-21. https://www.perplexity.ai/hub/blog/learning-from-real-world-experience
2. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter Rejection Sampling.
3. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter Evaluation.
4. Lambert, N. et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124, 2024.
5. Gao, L., Schulman, J., Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML 2023. arXiv:2210.10760.