Tech Meridian ← LIVE FEED
PROMY MERIDIAN RU

RESEARCH · RESEARCH · #3622

A Spectral Framework for Universal Proxy Regularization in Model Merging (arXiv:2611.17728v1)

The paper proposes a spectral framework that characterizes when a task-agnostic proxy distribution can substitute for task data during model merging via two metrics—spectral coverage (α) and leakage (β)—and derives a unified parameter-interference bound showing smaller (α,β) reduces interference. It introduces a proxy regularizer with three instantiations (J: Jacobian-vector exact, F: Fisher frozen curvature, O: output-drift) that coincide under NTK linearization, analyzes their departures for nonlinear fine-tuning, and validates the theoretical predictions on CLIP.

KEY POINTS

  1. The paper proposes a spectral framework that characterizes when a task-agnostic proxy distribution can substitute for task data during model merging via two metrics—spectral coverage (α) and leakage (β)—and derives a unified parameter-interference bound showing smaller (α,β) reduces interference.
  2. It introduces a proxy regularizer with three instantiations (J: Jacobian-vector exact, F: Fisher frozen curvature, O: output-drift) that coincide under NTK linearization, analyzes their departures for nonlinear fine-tuning, and validates the theoretical predictions on CLIP.
  3. This gives a theoretical and practical account of when task-agnostic proxy data suffices for privacy-preserving model merging and provides concrete regularizers with provable behavior, helping reduce cross-task interference without joint training.

WHY IT MATTERS

This gives a theoretical and practical account of when task-agnostic proxy data suffices for privacy-preserving model merging and provides concrete regularizers with provable behavior, helping reduce cross-task interference without joint training.

SOURCES & TIMELINE

1