RESEARCH · RESEARCH · #1408
Paper (arXiv:2610.00234v1): ordering and LR schedule determine which convention a model commits to
This paper proves and empirically measures that when a model is trained on the same problems written under two incompatible but correct conventions, the data arrangement (ordering) and the learning-rate schedule enter multiplicatively to determine which convention the model commits to. Using ten orderings and families of schedulers (constant, decayed, cosine), the authors show large allocation shifts (allocation share 0.04–0.87) and a 12.29σ arrangement switch in convention choice, while a convention-agnostic score keeps combined accuracy roughly conserved (within 9.7%); explicitly marking the convention in the prompt largely collapses the switch and recovers most of the union performance.
KEY POINTS
- This paper proves and empirically measures that when a model is trained on the same problems written under two incompatible but correct conventions, the data arrangement (ordering) and the learning-rate schedule enter multiplicatively to determine which convention the model commits to.
- Using ten orderings and families of schedulers (constant, decayed, cosine), the authors show large allocation shifts (allocation share 0.04–0.87) and a 12.29σ arrangement switch in convention choice, while a convention-agnostic score keeps combined accuracy roughly conserved (within 9.7%); explicitly marking the convention in the prompt largely collapses the switch and recovers most of the union performance.
- This matters because dataset ordering and scheduler choice can make a model commit to one correct convention without changing aggregate capability as measured by convention-agnostic metrics, affecting pretraining design and benchmark interpretation; prompt annotation is an effective mitigation.
WHY IT MATTERS
This matters because dataset ordering and scheduler choice can make a model commit to one correct convention without changing aggregate capability as measured by convention-agnostic metrics, affecting pretraining design and benchmark interpretation; prompt annotation is an effective mitigation.