Skip to main content
The scripts are in the repository, not in the installed package. Clone it, then cd recipes/papers/message-board before running the commands below. Browse this recipe on GitHub.
Paper: MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning, Chanwoo Park et al., arXiv:2502.18439, February 2025. https://arxiv.org/abs/2502.18439 Book: RL on a verifiable reward pays for the answer, so a reader trained that way learns to use a teammate’s note only when the note makes its own answer right more often [1][2]. Claim: copies of a model that answer, read each other and answer again collaborate better after RL on the team’s outcome, and a model trained alone does not learn to collaborate (Park et al.). Talk inside one reply was trained away because nothing read it (team-talk); a board gives the talk a reader. The change: what a reader sees. Both arms draft the same four notes per problem and train the reader with GRPO. The baseline reader sees only its own note. The recipe reader sees all four.

​
Recipe

  1. Base: Qwen/Qwen2.5-1.5B-Instruct. Data: GSM8K, 512 train prompts from the train split, 120 held out from the test split.
  2. Round one, both arms: four copies of the current policy each post a note (key steps and a boxed answer, at most 256 tokens). No gradient: the notes are context.
  3. Round two, both arms: each copy reads its board and writes the final answer. TRL GRPO + LoRA r=32 trains this step, 40 steps, 12 problems x 4 readers, lr 1e-4, on-policy, no KL. Reward: the reader’s own answer, MathEqual against the GSM8K gold. Nothing pays for agreeing with the board.
  4. The one change: the baseline’s reader j sees note j; the recipe’s reader j sees notes 1 to 4. Same notes drawn, same tokens.
  5. Eval: the same two rounds on the 120 held-out problems, pass@1 over the 4 readers, two training seeds per arm, paired delta with a 95% interval (wai.compare, train_runs=). A majority vote over the board is the no-reading reference.

​
Run

​
Result

Recipe vs baseline: +0.029 [-0.023, +0.085] across both training seeds. Verdict: flat. Rescued: the reader’s own note was wrong and its answer is right. Misled: the note was right and the answer is wrong. Vote: the board’s most common answer, no reading. Reading the board beats voting on it in every row: trained shared-board readers end 20 and 13 points above the vote over the same notes, and are misled 2 to 4 times in 100. But the own-note reader rescues almost as often (13 and 24 in 100): most of the gain is a second pass over the problem, not the teammates. The notes are the weak link. Asked for three lines, the model writes full solutions and 256 tokens cut most of them before the answer (right 30 to 57 of 100), so the board carries fewer answers than it could.

​
Checks

Nothing in this table is ticked by hand: every cell is written by recipe.py into results.json.

​
Climb

​
Learned

  • Copies of a 1.5B model do read a board: untrained readers gain 5 points from seeing all four notes, trained readers beat the board’s vote by 13 to 20 points and are rarely misled. Unlike team-talk, nothing was trained away.
  • Training the reader on the board does not beat training it on its own note: a second look at the problem does most of the work, and the shared board adds 3 points that two seeds cannot tell from zero.
  • Next: train the writers too. Reward a note by whether the readers who saw it got the answer right (Park et al. [3]; leave-one-out credit), so notes get short and carry their answer. Then a harder set where one copy’s insight matters more than a second pass.
Verified 2026-09-23, whileai 0.125, TRL 0.19.1 + PEFT 0.16.0 on torch 2.7.1. 248.7 GPU minutes over five containers, 8.29onL40S(plusabout8.29 on L40S (plus about 1.30 on a first run that lost a container). Run page: https://while.ai/platform/training/run_bbc9c4191065e453. Experiment: https://while.ai/platform/experiments/message-board

​
References

  1. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter Reasoning.
  2. Shao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024.
  3. Park, C. et al. MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning. arXiv:2502.18439, 2025.
  4. Pappu, A. et al. Self-Organizing Agent Teams Learn to Reason Together. arXiv:2609.22682, 2026.
  5. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter Evaluation.
  6. Gao, L., Schulman, J., Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML 2023. arXiv:2210.10760.
Last modified on September 23, 2026