RESEARCH · RESEARCH · #2460
GRAM: Group-wise Rank-Aware Model Merging (arXiv:2610.29110v1)
This paper introduces GRAM, a method that groups geometrically compatible task updates using principal-angle geometry and then applies an SVD-based merge within each group using only fine-tuned weights. GRAM reduces a failure mode called 'subspace interference' and outperforms prior model-merging techniques on benchmarks including GLUE with Flan-T5, an 8-task CLIP vision suite, and medical instruction-tuned Qwen2-7B-Instruct with LoRAs.
KEY POINTS
- This paper introduces GRAM, a method that groups geometrically compatible task updates using principal-angle geometry and then applies an SVD-based merge within each group using only fine-tuned weights.
- GRAM reduces a failure mode called 'subspace interference' and outperforms prior model-merging techniques on benchmarks including GLUE with Flan-T5, an 8-task CLIP vision suite, and medical instruction-tuned Qwen2-7B-Instruct with LoRAs.
- Grouping task updates by their subspace geometry lets merges retain task-relevant directions and reduces aggregation error, which matters for deploying aggregated models from locally fine-tuned weights in privacy- or regulation-sensitive settings.
WHY IT MATTERS
Grouping task updates by their subspace geometry lets merges retain task-relevant directions and reduces aggregation error, which matters for deploying aggregated models from locally fine-tuned weights in privacy- or regulation-sensitive settings.