Talk methods: which training teaches two model copies to talk best

​

Recipe

1. Task: two copies of Qwen/Qwen2.5-1.5B-Instruct each see half the facts of a GSM8K problem and chat A, B, A, B; each copy’s last message boxes an answer. Team reward: the share of the two answers that are right. 1,024 train problems, 300 held out.
2. Baseline, GRPO: every turn gets team reward minus the mean over the problem’s 4 chats.
3. Recipe, Reinforce-Ada: a problem whose 4 chats all scored the same is replayed, 4 more chats a round, up to 3 rounds; keep 2 high and 2 low, advantage against the mean of every chat drawn.
4. RAFT: advantage 1 for a chat the team solved outright, 0 otherwise. Fine-tuning on the winners, online.
5. Mixed partners: GRPO, but in 2 of every 4 chats B is the untrained base (LoRA off); those B turns are generated, not trained.
6. Every arm: TRL GRPO machinery + LoRA r=32, 80 steps, 8 problems x 4 chats, lr 1e-4, on-policy, no KL, 3 seeds (17, 18, 19). Eval: 2 chats on each of the 300 held-out problems, both answers graded; and each trained model as A with the untrained base as B.

​

Run

python recipe.py --selftest      # the split, the chat and the credit per method, offline
python recipe.py --reuse         # all four arms, three seeds; reuses the GRPO arm if cached
python recipe.py --arm raft      # one method
python post.py                   # the Experiments card: one run per method and a chart

​

Result

​

Checks

​

Climb

​

Learned

• Reinforce-Ada is the reliable way to teach the talk: every seed learned it, where GRPO lost one seed in three. The mean gap (0.38 against 0.30) needs more seeds to call.
• Fine-tuning on winning chats learns the shortest messages (91 chars on one seed) and uses the partner’s facts least. Mixing in an untrained partner did not make the skill travel.
• Next: five seeds for Ada and GRPO so the reliability gap can resolve, then Ada with a mixed-partner pool.

​

References

1. Shao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024.
2. Xiong, W. et al. Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectives. arXiv:2510.04996, 2025.
3. Dong, H. et al. RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment. arXiv:2304.06767, 2023.
4. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter Evaluation.
5. Gao, L., Schulman, J., Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML 2023. arXiv:2210.10760.
6. Park, C. et al. MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning. arXiv:2502.18439, 2025.