The scripts are in the repository, not in the installed package. Clone it,
then
cd recipes/papers/ptgs before running the commands below. Browse this recipe on GitHub.Result
pass@1 intervals are
wai.compare over 320 paired tasks; pass@8 intervals are a task bootstrap on seed-averaged curves and do not include seed variance. A verdict of “moved” needs the interval to exclude zero and the delta to clear the eval’s re-run band (t at 2 df x sqrt(2) x run_std = 0.051).
- Mechanism: PTGS did not reduce zero-advantage groups (0.71 vs 0.68). Heating rarely lifted a failing prompt’s pass rate, and cooling turned high-pass-rate groups all-correct, which also zeroes the advantage (the paper’s Proposition 5). Reinforce-Ada cut them to 0.41 by sampling 23.3 rollouts per prompt.
- No tax at this scale: every arm raised pass@8 over the base (0.503). The paper reports the tax for fully post-trained checkpoints at k up to 128; 80 LoRA steps on a 1.5B model at k = 8 did not reach it.
The paper
Paper: Sharpening Tax in Post-Training, Changdae Oh et al., arXiv:2610.01509, October 2026. https://arxiv.org/abs/2610.01509 Book: GRPO’s advantage is a reward minus its group’s mean, so a group whose rollouts all scored the same contributes no gradient [1][2]. Claim: RL post-training concentrates per-prompt pass rates at 0 and 1, raising pass@1 and lowering pass@k (the Sharpening Tax); sampling each prompt’s group at a temperature set by a Beta posterior over its pass rate reduces the tax and raises both [7]. The change: the rollout temperature. GRPO samples every prompt at T = 0.9; PTGS samples prompt x at T = 0.9 x 1.5^h(p̂), with p̂ drawn from Beta(2p~ + s, 2(1 - p~) + f) and log-probs taken at the same T. Reinforce-Ada keeps T = 0.9 and samples up to 32 rollouts [3].Recipe
Run
Checks
Climb
Learned
- Low-pass-rate prompts needed more samples, not a higher temperature: Reinforce-Ada cut zero-advantage groups from 0.68 to 0.41, PTGS raised them to 0.71. Theorem 3 of the paper assumes heating raises p; at 1.5B on MATH it mostly did not.
- PTGS adds no generation cost. In TRL 0.19.1 it is a per-row logits processor on
generateand a per-row temperature divisor in_get_per_token_logps. - Next: the paper’s GRPO setting (tau 1.4, p~ fixed at 0.5) and k = 64 on the holdout, so Tax_S has room to move.
References
- Shao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024.
- Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter Reinforcement Learning.
- Xiong, W. et al. Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectives. arXiv:2510.04996, 2025.
- Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter Evaluation.
- Lambert, N. et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124, 2024.
- Gao, L., Schulman, J., Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML 2023. arXiv:2210.10760.
- Oh, C. et al. Sharpening Tax in Post-Training. arXiv:2610.01509, 2026.