Skip to main content
The scripts are in the repository, not in the installed package. Clone it, then cd recipes/papers/talk-to-solve before running the commands below. Browse this recipe on GitHub.
Paper: MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning, Chanwoo Park et al., arXiv:2502.18439, February 2025. https://arxiv.org/abs/2502.18439 Book: a policy-gradient update pays a token only through the reward its rollout earns, so a message is learned only when the reward it is paid depends on what the partner did with it [1][2]. Claim: copies of a model that answer and discuss collaborate better after RL on the team’s outcome, and every turn of the discussion should be trained, not only the last (Park et al.). The three recipes before this one (team-talk, message-board, board-writers) never needed the talk: one copy could solve the problem alone. Here it cannot. The change: the channel. Both arms play the same chat and train every message on the team’s outcome. The baseline’s messages are never delivered: each copy sees “(no message)” where its partner’s words would be.

​
Recipe

  1. Base: Qwen/Qwen2.5-1.5B-Instruct. Data: GSM8K problems whose fact sentences split into two halves that each hold a number; copy A gets every other fact starting with the first, copy B the rest, both get the question. 1,024 train problems from the train split, 300 held out from the test split.
  2. The chat: A, B, A, B. A message is at most 160 tokens; each copy’s last message (at most 384) boxes an answer.
  3. Reward: the mean of A’s and B’s final answers (MathEqual against the GSM8K gold). Every message in the chat is paid it, minus the mean over the problem’s 4 chats.
  4. Both arms: TRL GRPO + LoRA r=32, 80 steps, 8 problems x 4 chats x 4 turns, lr 1e-4, on-policy, no KL, two chats per micro-batch. The one change is whether messages are delivered.
  5. Eval: 2 chats on each of the 300 held-out problems, both copies’ answers graded (4 answers a problem), three training seeds per arm, paired delta with a 95% interval (wai.compare, train_runs=). The talk is measured too: how much of what only it knows a copy sends, how much of what only its partner knows its answer uses, first messages that ask, agreement. And each trained model plays A against the untrained base as B.

​
Run

​
Result

Trained chat vs trained with the channel cut: +0.365 [+0.321, +0.409] across three training seeds. Verdict: moved. The copies learned to talk. With the channel cut, training cannot get past 2 to 4 in 100, because half the facts is not enough. With the channel on, two of three seeds end at 0.40 and 0.35 from an untrained 0.11: each copy sends nearly every number only it has (92 to 96 in 100, from 67) and builds its answer from its partner’s numbers (72 to 79 in 100, from 30). They stopped asking: once both sides send everything unprompted, a question is a wasted turn. Two caveats. Seed 18 learned to send (79 in 100) but not to use (46 in 100), wrote the longest messages (698 chars) and ended at 0.15, so the skill is learned on most seeds, not all. And it is a protocol between two trained copies, not a skill one copy carries alone: a trained A with the untrained base as B scores 0.10 to 0.16, about where two untrained copies are.

​
Checks

Nothing in this table is ticked by hand: every cell is written by recipe.py into results.json.

​
Climb

​
Learned

  • RL on the team’s outcome teaches two copies of a 1.5B model to communicate when the task needs it: send what only you know, use what your partner sent. The team goes from 11 to 35-40 in 100 on two of three seeds; with the channel cut it stays at 2 to 4.
  • What they learn is a shared protocol. Paired with an untrained partner, a trained copy is back near the untrained pair. And one seed learned only half of it: sending without using.
  • Next: longer chats where asking pays (facts split three ways, a question that needs a follow-up), then train against a mix of partners (trained and untrained) so the skill survives a new teammate.
Verified 2026-09-23, whileai 0.125, TRL 0.19.1 + PEFT 0.16.0 on torch 2.7.1. 567.8 GPU minutes over seven containers, 37.85onH100(plusabout37.85 on H100 (plus about 25 on a first run whose chat arm ran out of memory at step 45 with four chats per micro-batch; every arm was rerun at two). Run page: https://while.ai/platform/training/run_109c64a5b5e988be. Experiment: https://while.ai/platform/experiments/talk-to-solve

​
References

  1. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter Reasoning.
  2. Shao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024.
  3. Park, C. et al. MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning. arXiv:2502.18439, 2025.
  4. Cobbe, K. et al. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168, 2021.
  5. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter Evaluation.
  6. Gao, L., Schulman, J., Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML 2023. arXiv:2210.10760.
Last modified on September 24, 2026