RESEARCH · RESEARCH · #3622
A Spectral Framework for Universal Proxy Regularization in Model Merging (arXiv:2611.17728v1)
The paper proposes a spectral framework that characterizes when a task-agnostic proxy distribution can substitute for task data during model merging via two metrics—spectral coverage (α) and leakage (β)—and derives a unified parameter-interference bound showing smaller (α,β) reduces interference. It introduces a proxy regularizer with three instantiations (J: Jacobian-vector exact, F: Fisher frozen curvature, O: output-drift) that coincide under NTK linearization, analyzes their departures for nonlinear fine-tuning, and validates the theoretical predictions on CLIP.
KEY POINTS
- The paper proposes a spectral framework that characterizes when a task-agnostic proxy distribution can substitute for task data during model merging via two metrics—spectral coverage (α) and leakage (β)—and derives a unified parameter-interference bound showing smaller (α,β) reduces interference.
- It introduces a proxy regularizer with three instantiations (J: Jacobian-vector exact, F: Fisher frozen curvature, O: output-drift) that coincide under NTK linearization, analyzes their departures for nonlinear fine-tuning, and validates the theoretical predictions on CLIP.
- This gives a theoretical and practical account of when task-agnostic proxy data suffices for privacy-preserving model merging and provides concrete regularizers with provable behavior, helping reduce cross-task interference without joint training.
WHY IT MATTERS
This gives a theoretical and practical account of when task-agnostic proxy data suffices for privacy-preserving model merging and provides concrete regularizers with provable behavior, helping reduce cross-task interference without joint training.