Tech Meridian ← LIVE FEED
PROMY MERIDIAN RU

RESEARCH · RESEARCH · #3813

DNeo paper introduces MoFA feature encoding to speed parallel draft verification

New arXiv preprint (arXiv:2611.21925v1) proposes DNeo, an architecture for speculative-diffusion parallel drafting that uses Mixture-of-Features Attention (MoFA), a rejection memory, and several architectural tweaks (2‑tap causal convolutions, prefix-only cross-attention, non-uniform layer selection) to address lossy, memory-less, and uniform-sampling shortcomings of prior feature encoders. The authors report DNeo achieves about 1.3× faster generation than the prior state of the art (DFlash-2) without increasing drafter footprint or degrading generation quality.

KEY POINTS

  1. New arXiv preprint (arXiv:2611.21925v1) proposes DNeo, an architecture for speculative-diffusion parallel drafting that uses Mixture-of-Features Attention (MoFA), a rejection memory, and several architectural tweaks (2‑tap causal convolutions, prefix-only cross-attention, non-uniform layer selection) to address lossy, memory-less, and uniform-sampling shortcomings of prior feature encoders.
  2. The authors report DNeo achieves about 1.3× faster generation than the prior state of the art (DFlash-2) without increasing drafter footprint or degrading generation quality.
  3. Faster, higher-fidelity feature encoding for parallel draft verification can materially reduce LLM inference latency and improve practical throughput for multi-token speculative decoding.

WHY IT MATTERS

Faster, higher-fidelity feature encoding for parallel draft verification can materially reduce LLM inference latency and improve practical throughput for multi-token speculative decoding.

SOURCES & TIMELINE

1