Tech Meridian ← ENTITY INDEX
PROMY MERIDIAN RU

TOPIC · ENTITY #5703

transformer

Related event timeline, sources and context from the news index.

EVENT TIMELINE

2

RESEARCH · 1 SOURCE · arXiv cs.AI

BOS Summary: inference-only method that makes the BOS token carry input context to boost reasoning

The paper proposes BOS Summary, a training-free, inference-time modification that relaxes causal masking for the beginning-of-sequence (BOS) query row at a single transformer layer during prefill so BOS aggregates all visual and textual inputs into an input-specific context; other tokens keep standard causal attention. The change requires no extra parameters or decoding overhead and reportedly improves multi-step reasoning across several LLM and LVLM families and benchmarks, outperforming direct/CoT prompting and recent sink-intervention methods, while analysis shows it also reduces BOS sink dominance and redistributes attention toward content tokens.

7.0

RESEARCH · 1 SOURCE · arXiv cs.AI

Attention-Aware Routing (AAR) for MoE models (arXiv:2609.20974v1)

The paper introduces Attention-Aware Routing (AAR), which augments Mixture-of-Experts routers with temporal and spectral features derived from a sliding window of attention weights while keeping the base transformer frozen and training only routing parameters. On OLMoE, AAR improves GSM8K accuracy by +3.37 percentage points over a routing-only SFT baseline, demonstrates that routing updates propagate to reshape subsequent-layer attention without changing attention weights directly, reduces long diverging generations for incorrect answers, and shows strong depth sensitivity that separates retrieval and reasoning behavior across layers.

7.0