Trim SFT data to what the test can see

​

The loop

import whileai as wai

rows = wai.select(scored, mode="sft", target=16000)
base = evaluate(base_model)  # per-item 0/1 scores on your test

scores = {}
for n in rows.trim_sizes():  # 64, 128, 256, ... len(rows)
    scores[n] = [train_and_eval(rows.trim(n), seed) for seed in (0, 1)]  # per-item scores per seed

kept = rows.trim(scores, base=base)
print(kept.report["trim"])
kept.export("train.jsonl")

verdict: saturated; N* = 5,208 rows (next doubling gains < 0.010)
  fit: base + (a - base) * n^c / (d + n^c); base 0.296, a 0.714, c 0.92, d 147
  error predicting n=16000 from the rest: rectified 0.007, power 0.014, logit 0.064
  N* 90% range 2,239 rows .. 6,214 rows; inside the sizes trained in 100% of draws
  last doubling gained +0.007 [+0.000, +0.016]
  smallest gain this test can detect: 0.021
  note: min_gain 0.010 is under what this test can detect (0.021): add items or seeds

​

Which rows

​

How many rows

​

How sure

• Range on N*. A bootstrap resamples test items (the same items at every size, a paired design) and seeds within each size, refits, and reports the 5 to 95% range [12]. The verdict is saturated only when N* falls inside the sizes you trained in at least 90% of draws.
• Can the test see min_gain? The paired detectable gain between the two largest sizes is 2.8 * sqrt(2 * seed_var / k + var(item diffs) / test_n) [12]. If min_gain is smaller, the print says so: add items or seeds.
• Forecasts. An N* past twice the largest size is labeled forecast. Downstream scores often break from a clean curve [14], and about 15% of learning curves are ill-behaved [15].
• Other verdicts. not identifiable means over 10% of refits failed, or the power exponent fell under 0.1 (fine-tuning exponents of 0.015 to 0.15 are reported in loss [17]). flat means no rising curve fits. Anything but saturated or flat keeps every row.

​

References

1. Xia et al. “Rethinking Data Selection at Scale: Random Selection is Almost All You Need.” arXiv:2410.09335, 2024.
2. Diddee, Ippolito. “Chasing Random: Instruction Selection Strategies Fail to Generalize.” arXiv:2410.15225, 2024.
3. Ivison et al. “Large-Scale Data Selection for Instruction Tuning.” arXiv:2503.01807, 2025.
4. “Measuring Data Diversity for Instruction Tuning” (NovelSum). ACL 2025, arXiv:2502.17184.
5. Xia et al. “LESS: Selecting Influential Data for Targeted Instruction Tuning.” arXiv:2402.04333, 2024.
6. Lin et al. “Selecting Large Language Model to Fine-tune via Rectified Scaling Law.” ICML 2024, arXiv:2402.02314.
7. Hestness et al. “Deep Learning Scaling is Predictable, Empirically.” arXiv:1712.00409, 2017.
8. Yuan et al. “Scaling Relationship on Learning Mathematical Reasoning with Large Language Models.” arXiv:2308.01825, 2023.
9. Abbas et al. “SemDeDup: Data-efficient learning at web-scale through semantic deduplication.” arXiv:2303.09540, 2023.
10. Lin, Bilmes. “Multi-document Summarization via Budgeted Maximization of Submodular Functions.” NAACL 2010.
11. Alabdulmohsin, Neyshabur, Zhai. “Revisiting Neural Scaling Laws in Language and Vision.” NeurIPS 2022, arXiv:2209.06640.
12. Miller. “Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations.” arXiv:2411.00640, 2024.
13. Provost, Jensen, Oates. “Efficient Progressive Sampling.” KDD 1999.
14. Lourie, Hu, Cho. “Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check.” arXiv:2507.00885, 2025.
15. Yan, Mohr, Viering. “LCDB 1.1.” NeurIPS 2025 Datasets and Benchmarks, arXiv:2505.15657.
16. “Data Repetition Beats Data Scaling in Long-CoT SFT.” arXiv:2602.11149, 2026.
17. Zhang et al. “When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method.” ICLR 2024, arXiv:2402.17193.