RESEARCH · RESEARCH · #3971
BOS Summary: inference-only method that makes the BOS token carry input context to boost reasoning
The paper proposes BOS Summary, a training-free, inference-time modification that relaxes causal masking for the beginning-of-sequence (BOS) query row at a single transformer layer during prefill so BOS aggregates all visual and textual inputs into an input-specific context; other tokens keep standard causal attention. The change requires no extra parameters or decoding overhead and reportedly improves multi-step reasoning across several LLM and LVLM families and benchmarks, outperforming direct/CoT prompting and recent sink-intervention methods, while analysis shows it also reduces BOS sink dominance and redistributes attention toward content tokens.
KEY POINTS
- The paper proposes BOS Summary, a training-free, inference-time modification that relaxes causal masking for the beginning-of-sequence (BOS) query row at a single transformer layer during prefill so BOS aggregates all visual and textual inputs into an input-specific context; other tokens keep standard causal attention.
- The change requires no extra parameters or decoding overhead and reportedly improves multi-step reasoning across several LLM and LVLM families and benchmarks, outperforming direct/CoT prompting and recent sink-intervention methods, while analysis shows it also reduces BOS sink dominance and redistributes attention toward content tokens.
- A simple, training-free inference intervention that repurposes the persistent BOS pathway can materially improve multi-step reasoning and reveals how changing sink content reshapes attention allocation in transformers.
WHY IT MATTERS
A simple, training-free inference intervention that repurposes the persistent BOS pathway can materially improve multi-step reasoning and reveals how changing sink content reshapes attention allocation in transformers.