Skip to main content
The scripts are in the repository, not in the installed package. Clone it, then cd recipes/papers/board-writers before running the commands below. Browse this recipe on GitHub.
Paper: MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning, Chanwoo Park et al., arXiv:2502.18439, February 2025. https://arxiv.org/abs/2502.18439 Book: a policy-gradient update credits a token only through the reward its rollout earns; a note that another rollout reads earns nothing unless the reward is routed back to it [1][2]. Claim: training every turn of a discussion on the team’s outcome, not only the last one, is what makes copies of a model collaborate (Park et al.). In message-board only the readers were trained, and the notes stayed long, cut off before their answer, and barely more useful than rereading your own. The change: the writers’ advantage. Both arms train notes and readers in one batch. The baseline’s notes sit in the batch at zero advantage; the recipe pays each note the mean reward of the two readers who read it.

​
Recipe

  1. Base: Qwen/Qwen2.5-1.5B-Instruct. Data: MATH train, levels 3 to 5, 1,024 problems; MATH-500 levels 3 to 5, 300 held out (disjoint by construction).
  2. Round one: four copies of the current policy each post a note (key steps and a boxed answer, at most 256 tokens).
  3. Round two, a ring: reader j reads notes j and j + 1, so every note has exactly two readers. A reader is paid its own answer (MathEqual against the gold).
  4. Baseline: TRL GRPO + LoRA r=32, 80 steps, 12 problems x (4 notes + 4 readers), lr 1e-4, on-policy, no KL, reward minus group mean. Notes are in the batch at zero advantage, so the loss is normalized over the same tokens.
  5. Recipe: the same, with each note’s advantage its two readers’ mean reward minus the mean over the four notes. Nothing about the note’s own answer enters.
  6. Eval: the same two rounds on the 300 held-out problems, pass@1 over the 4 readers, three training seeds per arm, paired delta with a 95% interval (wai.compare, train_runs=).

​
Run

​
Result

Recipe vs baseline: -0.017 [-0.048, +0.014] across three training seeds. Verdict: flat, and pointing the wrong way. Training the readers alone is steady: 0.32, 0.33, 0.35. Paying the notes is not: 0.30, 0.23, 0.34, and the 0.23 seed ends below the untrained base. The notes did not become short and answer-bearing in any seed. They stayed long or got longer (744 chars on seed 17), and the share that close a boxed answer fell to 8 to 16 in 100, where the baseline’s notes kept 14 to 31. The one seed whose notes got short (451 chars) had the worst vote over its board (0.07) and the most misled readers (0.04). Why the credit did not teach: the mean absolute note advantage was 0.08 to 0.09 all run. On a problem where two readers score 1 and 0, every note on the ring gets roughly the same credit, because each reader reads two notes and every note is read by two readers. The signal a note gets is mostly how hard the problem was, not what the note said. And notes and readers share one set of weights, so reader training moves the notes in both arms (the baseline’s notes drift too, from 0.26 to between 0.14 and 0.31 with an answer).

​
Checks

Nothing in this table is ticked by hand: every cell is written by recipe.py into results.json.

​
Climb

​
Learned

  • Readers learn from a board on hard problems too: from 0.26 untrained to 0.32 to 0.35 on MATH levels 3 to 5, three seeds within 3.5 points of each other.
  • Paying a note its readers’ mean does not make better notes at this size. The credit is too flat to say which note helped, the notes drifted long in two seeds and short-and-wrong in the third, and one seed ended below the base.
  • Next: sharper credit. Leave one note out and re-read (the difference is that note’s own effect, at 5x the reading cost), or pay a note for its own correct boxed answer plus what its readers did, so the easy signal comes first. The in-budget alternative is a larger model, whose untrained notes already carry answers.
Verified 2026-09-23, whileai 0.125, TRL 0.19.1 + PEFT 0.16.0 on torch 2.7.1. 590.2 GPU minutes over seven containers, $39.35 on H100. Run page: https://while.ai/platform/training/run_65abd81dd729e5f2. Experiment: https://while.ai/platform/experiments/board-writers

​
References

  1. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter Reasoning.
  2. Shao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024.
  3. Park, C. et al. MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning. arXiv:2502.18439, 2025.
  4. Hendrycks, D. et al. Measuring Mathematical Problem Solving With the MATH Dataset. NeurIPS 2021. arXiv:2103.03874.
  5. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter Evaluation.
  6. Gao, L., Schulman, J., Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML 2023. arXiv:2210.10760.
Last modified on September 23, 2026