NEWS · RESEARCH · #231
PROOF-Gen: From optimized training data to improved distillation for tool-calling
The paper presents PROOF-Gen, a method that focuses on generating optimized training data to improve supervised fine-tuning of student models distilled from teacher-generated trajectories, aiming to overcome limitations of the common generate-and-filter pipeline that leaves hard failure cases unaddressed. The authors report that on τ 2-bench, 57% of teacher trials fail and roughly two-thirds of those failures are near-misses (most tool calls correct but subsequently undone), motivating the need to extract signal from failures rather than discarding them.
KEY POINTS
- The paper presents PROOF-Gen, a method that focuses on generating optimized training data to improve supervised fine-tuning of student models distilled from teacher-generated trajectories, aiming to overcome limitations of the common generate-and-filter pipeline that leaves hard failure cases unaddressed.
- The authors report that on τ 2-bench, 57% of teacher trials fail and roughly two-thirds of those failures are near-misses (most tool calls correct but subsequently undone), motivating the need to extract signal from failures rather than discarding them.
- If effective, PROOF-Gen could make distillation pipelines more data-efficient and reduce recurring costs by turning teacher failures into useful training signal rather than discarding them.
WHY IT MATTERS
If effective, PROOF-Gen could make distillation pipelines more data-efficient and reduce recurring costs by turning teacher failures into useful training signal rather than discarding them.