RESEARCH · RESEARCH · #2561
Hierarchical agent distillation for LLMs (arXiv:2610.31918v1)
The paper introduces a hierarchical agent distillation framework that separates two levels of "on-policiness"—the policy that produces interaction histories and the policy that generates the token prefix of the current action—and lets practitioners control their pairing, mixture, and schedule. In a three-stage self-distillation study across three models and four interactive environments, the authors find that student-induced histories paired with teacher-generated action prefixes give the strongest standalone signal, fully on-policy data performs worst alone but helps in mixtures, and front-loading teacher contexts while delaying on-policy contexts improves results.
KEY POINTS
- The paper introduces a hierarchical agent distillation framework that separates two levels of "on-policiness"—the policy that produces interaction histories and the policy that generates the token prefix of the current action—and lets practitioners control their pairing, mixture, and schedule.
- In a three-stage self-distillation study across three models and four interactive environments, the authors find that student-induced histories paired with teacher-generated action prefixes give the strongest standalone signal, fully on-policy data performs worst alone but helps in mixtures, and front-loading teacher contexts while delaying on-policy contexts improves results.
- Provides actionable principles for where, how much, and when to follow the student in LLM agent distillation, affecting training recipes for interactive agents.
WHY IT MATTERS
Provides actionable principles for where, how much, and when to follow the student in LLM agent distillation, affecting training recipes for interactive agents.