> ## Documentation Index
> Fetch the complete documentation index at: https://docs.while.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Install with `uv add whileai`; import as `import whileai as wai`.
> Run the offline path first (`simulator=False`, `wai.seeded_agent`, a callable judge); no key is needed for it.
> Report every pass rate with its interval and n, as `scored.pass_at` prints it.

# Board writers: pay a note by what it did for the copies that read it

> Paper: MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning, Chanwoo Park et al., arXiv:2502.18439, February 2025.

<Note>The scripts are in the repository, not in the installed package. Clone it,
then `cd recipes/papers/board-writers` before running the commands below. [Browse this recipe on GitHub](https://github.com/whilehq/whileai-sdk/tree/main/recipes/papers/board-writers).</Note>

**Paper:** MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning, Chanwoo Park et al., arXiv:2502.18439, February 2025. [https://arxiv.org/abs/2502.18439](https://arxiv.org/abs/2502.18439)
**Book:** a policy-gradient update credits a token only through the reward its rollout earns; a note that another rollout reads earns nothing unless the reward is routed back to it \[1]\[2].
**Claim:** training every turn of a discussion on the team's outcome, not only the last one, is what makes copies of a model collaborate (Park et al.). In [message-board](/recipes/papers/message-board) only the readers were trained, and the notes stayed long, cut off before their answer, and barely more useful than rereading your own.
**The change:** the writers' advantage. Both arms train notes and readers in one batch. The baseline's notes sit in the batch at zero advantage; the recipe pays each note the mean reward of the two readers who read it.

## Recipe

1. Base: `Qwen/Qwen2.5-1.5B-Instruct`. Data: MATH train, levels 3 to 5, 1,024 problems; MATH-500 levels 3 to 5, 300 held out (disjoint by construction).
2. Round one: four copies of the current policy each post a note (key steps and a boxed answer, at most 256 tokens).
3. Round two, a ring: reader j reads notes j and j + 1, so every note has exactly two readers. A reader is paid its own answer (`MathEqual` against the gold).
4. Baseline: TRL GRPO + LoRA r=32, 80 steps, 12 problems x (4 notes + 4 readers), lr 1e-4, on-policy, no KL, reward minus group mean. Notes are in the batch at zero advantage, so the loss is normalized over the same tokens.
5. Recipe: the same, with each note's advantage its two readers' mean reward minus the mean over the four notes. Nothing about the note's own answer enters.
6. Eval: the same two rounds on the 300 held-out problems, pass\@1 over the 4 readers, three training seeds per arm, paired delta with a 95% interval (`wai.compare`, `train_runs=`).

## Run

```bash theme={"theme":"vitesse-dark"}
python recipe.py --selftest      # the ring, the credit and the counters, offline
python recipe.py                 # both arms, three seeds, plus a base-eval container, seven H100
python recipe.py --reuse         # rerun only the containers that failed
```

## Result

| Arm                                                               | pass\@1 | 95% CI        | pass\@k | Notes with an answer | Note chars | Vote | Rescued | Steps | GPU min |
| ----------------------------------------------------------------- | ------- | ------------- | ------- | -------------------- | ---------- | ---- | ------- | ----- | ------- |
| Base, no training                                                 | 0.26    | \[0.23, 0.30] | 0.46    | 0.26                 | 628        | 0.14 | 0.19    | 0     | 0       |
| Baseline, readers trained (seed 17; 18: 0.33, 19: 0.35)           | 0.32    | \[0.27, 0.36] | 0.46    | 0.31                 | 627        | 0.23 | 0.18    | 80    | 56.0    |
| Recipe, readers and writers trained (seed 17; 18: 0.23, 19: 0.34) | 0.30    | \[0.26, 0.34] | 0.49    | 0.08                 | 744        | 0.12 | 0.24    | 80    | 57.8    |

Recipe vs baseline: **-0.017 \[-0.048, +0.014]** across three training seeds. Verdict: **flat**, and pointing the wrong way.

Training the readers alone is steady: 0.32, 0.33, 0.35. Paying the notes is not: 0.30, 0.23, 0.34, and the 0.23 seed ends below the untrained base. The notes did not become short and answer-bearing in any seed. They stayed long or got longer (744 chars on seed 17), and the share that close a boxed answer fell to 8 to 16 in 100, where the baseline's notes kept 14 to 31. The one seed whose notes got short (451 chars) had the worst vote over its board (0.07) and the most misled readers (0.04).

Why the credit did not teach: the mean absolute note advantage was 0.08 to 0.09 all run. On a problem where two readers score 1 and 0, every note on the ring gets roughly the same credit, because each reader reads two notes and every note is read by two readers. The signal a note gets is mostly how hard the problem was, not what the note said. And notes and readers share one set of weights, so reader training moves the notes in both arms (the baseline's notes drift too, from 0.26 to between 0.14 and 0.31 with an answer).

## Checks

Nothing in this table is ticked by hand: every cell is written by `recipe.py` into `results.json`.

| Check                                                            | Source                                       | Result                                                                                                              |
| ---------------------------------------------------------------- | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| Eval noise: the base evaluated 3 times, `eval_variance` run\_std | \[5]                                         | run\_std 0.010 from 3 re-runs (0.26, 0.26, 0.24); the recipe's seed spread (0.23 to 0.34) is ten times that         |
| Holdout is clean: `decontaminate(train, against=holdout)`        | \[5]                                         | 0 of 1,024 train rows dropped                                                                                       |
| Reward is a program, not a judge                                 | \[5]                                         | `MathEqual` (Math-Verify) against the MATH gold; a note is paid only through its readers                            |
| Proxy vs target: `wai.compare(proxy=)`                           | \[6]                                         | `proxy=None`: the training reward is the target; over\_optimized false                                              |
| Length: mean reader completion before -> after, per arm          | \[6]                                         | 1,047 chars base -> 953 baseline, 1,062 recipe                                                                      |
| Hack scan on the last training batch: `hack_scan`                | \[6]                                         | nothing above the floor                                                                                             |
| Pinned: seed, torch, transformers, trl, peft                     | [the contract](/recipes/papers#the-contract) | training seeds 17, 18, 19, `--seed 0` for the data; torch 2.7.1, transformers 4.54.0, trl 0.19.1, peft 0.16.0; H100 |

## Climb

| Round | What changed                                                  | pass\@1                                                | vs previous                    |
| ----- | ------------------------------------------------------------- | ------------------------------------------------------ | ------------------------------ |
| 1     | notes paid their two readers' mean on a ring, 3 seeds per arm | baseline 0.32 / 0.33 / 0.35, recipe 0.30 / 0.23 / 0.34 | -0.017 \[-0.048, +0.014], flat |

## Learned

* Readers learn from a board on hard problems too: from 0.26 untrained to 0.32 to 0.35 on MATH levels 3 to 5, three seeds within 3.5 points of each other.
* Paying a note its readers' mean does not make better notes at this size. The credit is too flat to say which note helped, the notes drifted long in two seeds and short-and-wrong in the third, and one seed ended below the base.
* Next: sharper credit. Leave one note out and re-read (the difference is that note's own effect, at 5x the reading cost), or pay a note for its own correct boxed answer plus what its readers did, so the easy signal comes first. The in-budget alternative is a larger model, whose untrained notes already carry answers.

Verified 2026-09-23, whileai 0.125, TRL 0.19.1 + PEFT 0.16.0 on torch 2.7.1. 590.2 GPU minutes over seven containers, \$39.35 on H100. Run page: [https://while.ai/platform/training/run\_65abd81dd729e5f2](https://while.ai/platform/training/run_65abd81dd729e5f2). Experiment: [https://while.ai/platform/experiments/board-writers](https://while.ai/platform/experiments/board-writers)

## References

1. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter *Reasoning*.
2. Shao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024.
3. Park, C. et al. MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning. arXiv:2502.18439, 2025.
4. Hendrycks, D. et al. Measuring Mathematical Problem Solving With the MATH Dataset. NeurIPS 2021. arXiv:2103.03874.
5. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter *Evaluation*.
6. Gao, L., Schulman, J., Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML 2023. arXiv:2210.10760.