RESEARCH · RESEARCH · #1642
JIVEAdapter: an additive multi-task low-rank adapter via Joint and Individual Variation Explained
The paper introduces JIVEAdapter, an additive low-rank adapter that decomposes each weight update into a shared Joint component and per-task Individual components, enforces near-orthogonality between them, and adaptively allocates rank across shared and task-specific pools. Evaluated with DeBERTaV3-base on GLUE and SuperGLUE, JIVEAdapter matches strong single- and multi-task low-rank baselines at the same per-task effective rank and supports freezing a jointly learned Joint for reuse with cheap per-task updates.
KEY POINTS
- The paper introduces JIVEAdapter, an additive low-rank adapter that decomposes each weight update into a shared Joint component and per-task Individual components, enforces near-orthogonality between them, and adaptively allocates rank across shared and task-specific pools.
- Evaluated with DeBERTaV3-base on GLUE and SuperGLUE, JIVEAdapter matches strong single- and multi-task low-rank baselines at the same per-task effective rank and supports freezing a jointly learned Joint for reuse with cheap per-task updates.
- Separating and freezing a shared joint component while adaptively allocating task-specific rank enables more interpretable, reusable, and parameter-efficient multi-task fine-tuning without extra modules like MoE.
WHY IT MATTERS
Separating and freezing a shared joint component while adaptively allocating task-specific rank enables more interpretable, reusable, and parameter-efficient multi-task fine-tuning without extra modules like MoE.