RESEARCH · RESEARCH · #2617
A comprehensive framework for all stages of large-model training (arXiv:2610.33451v1)
This arXiv preprint proposes a unified stage division and a six-dimensional analytical framework (data, objective, optimization, system constraints, evaluation, risk/governance) covering data/tokenization, pretraining, and post-training alignment for LLMs and MLLMs, systematically reviewing 395 references (2017–2026). The authors present a cross-stage effects map, a decision tree, at least ten verifiable open problems, and three core findings: scaling-law comparability depends on data quality/mixture; preference-alignment methods (RLHF, DPO) and RL with verifiable rewards shift post-training boundaries; and multimodal training challenges tokenizer design, objectives, and evaluation.
KEY POINTS
- This arXiv preprint proposes a unified stage division and a six-dimensional analytical framework (data, objective, optimization, system constraints, evaluation, risk/governance) covering data/tokenization, pretraining, and post-training alignment for LLMs and MLLMs, systematically reviewing 395 references (2017–2026).
- The authors present a cross-stage effects map, a decision tree, at least ten verifiable open problems, and three core findings: scaling-law comparability depends on data quality/mixture; preference-alignment methods (RLHF, DPO) and RL with verifiable rewards shift post-training boundaries; and multimodal training challenges tokenizer design, objectives, and evaluation.
- A unified, cross-stage survey that highlights data quality as central to scaling laws and shows how alignment and multimodality reshape training pipelines is useful for researchers and practitioners designing or comparing large-model training regimes.
WHY IT MATTERS
A unified, cross-stage survey that highlights data quality as central to scaling laws and shows how alignment and multimodality reshape training pipelines is useful for researchers and practitioners designing or comparing large-model training regimes.