NEWS · CODING · #2659
Inclusion AI speeds Ling-3.0-flash inference on DGX Spark with Humming, FP8 LM-head, and NEXTN
Inclusion AI evaluated Ling-3.0-flash (124B MoE, 5.1B active params/token) on NVIDIA DGX Spark at batch size 1 and identified memory-bound decoding dominated by MoE routing and the LM head. Replacing FlashInfer MXFP4 MoE kernels with the Humming library cut time-per-token from 27.71 ms to 25.35 ms (9.3%), online FP8 quantization of the LM head halved its weight-read volume (together with Humming yielding a 21.7% decode throughput gain), and multi-token speculative decoding via NEXTN reached 3.90 accepted tokens per round on a synthetic workload (185.1% throughput); the three optimizations combined gave a 42.5% throughput increase and 24.5% mean-latency reduction on GSM8K with no statistically detectable accuracy change. Deployment code and instructions are in the Ling cookbook.
KEY POINTS
- Inclusion AI evaluated Ling-3.0-flash (124B MoE, 5.1B active params/token) on NVIDIA DGX Spark at batch size 1 and identified memory-bound decoding dominated by MoE routing and the LM head.
- Replacing FlashInfer MXFP4 MoE kernels with the Humming library cut time-per-token from 27.71 ms to 25.35 ms (9.3%), online FP8 quantization of the LM head halved its weight-read volume (together with Humming yielding a 21.7% decode throughput gain), and multi-token speculative decoding via NEXTN reached 3.90 accepted tokens per round on a synthetic workload (185.1% throughput); the three optimizations combined gave a 42.5% throughput increase and 24.5% mean-latency reduction on GSM8K with no statistically detectable accuracy change.
- Deployment code and instructions are in the Ling cookbook.
WHY IT MATTERS
This demonstrates practical kernel, quantization, and speculative-decoding changes that materially improve MoE inference throughput and latency on DGX Spark (batch‑1) without measurable accuracy loss, providing actionable deployment guidance.