Team talk: train a model to reason as a team talking to itself

​

Recipe

1. Base: Qwen/Qwen2.5-1.5B-Instruct. Data: GSM8K, 512 train prompts from the train split, 120 held out from the test split.
2. Reward, both arms: the binary outcome, MathEqual against the GSM8K gold number. A program, not a judge.
3. Baseline: TRL GRPO + LoRA r=32, 40 steps, 12 prompts x 4 rollouts per step, lr 1e-4, on-policy, no KL, reward minus the group mean with no std division. Plain prompt.
4. Recipe: the same trainer and settings under the team prompt (three speakers, one turn per line, Lead writes \boxed{}). Both arms get the same 512-token completion cap, so a conversation has to fit where a monologue does.
5. Eval: pass@1 on the 120 held-out tasks, 4 samples per task, two training seeds per arm (17 and 18), paired delta with a 95% interval (wai.compare, train_runs=). Twice: each arm under its own prompt, then both under the plain prompt. The second is the memory test: with the team prompt gone, whatever talk and accuracy remain is in the weights.

​

Run

python recipe.py --selftest                 # the talk counter and the grader, offline, no GPU and no key
python recipe.py                            # both arms, two seeds each, four L40S in parallel
python recipe.py --arm recipe --steps 80    # one arm, longer

​

Result

​

Checks

​

Climb

​

Learned

• A prompt cannot keep a 1.5B model talking under an answer-only reward. The talk cost 13 points before training, so GRPO removed it within 40 steps and the arm converged on the baseline’s monologue. Kim et al. see talk grow under the same reward, on models whose talk already pays; this one’s did not.
• Nothing was lost either: the team arm ends where GRPO ends, with and without its prompt. The prompt was a detour, not a handicap.
• Next: make the talk pay by giving it a listener. Separate copies post to a shared message board and read each other’s notes before answering, and the reward counts the team’s answer, so a note matters only if another copy uses it (Park et al. [5], Pappu et al. [4]). Or a model large enough that its own talk already helps.

​

References

1. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter Reasoning.
2. Shao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024.
3. Kim, J., Lai, S., Scherrer, N., Agüera y Arcas, B., Evans, J. Reasoning Models Generate Societies of Thought. arXiv:2601.10825, 2026.
4. Pappu, A. et al. Self-Organizing Agent Teams Learn to Reason Together. arXiv:2609.22682, 2026.
5. Park, C. et al. MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning. arXiv:2502.18439, 2025.
6. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter Evaluation.
7. Gao, L., Schulman, J., Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML 2023. arXiv:2210.10760.