Inclusion AI speeds Ling-3.0-flash inference on DGX Spark with Humming, FP8 LM-head, and NEXTN
Inclusion AI evaluated Ling-3.0-flash (124B MoE, 5.1B active params/token) on NVIDIA DGX Spark at batch size 1 and identified memory-bound decoding dominated by MoE routing and the LM head. Replacing FlashInfer MXFP4 MoE kernels with the Humming library cut time-per-token from 27.71 ms to 25.35 ms (9.3%), online FP8 quantization of the LM head halved its weight-read volume (together with Humming yielding a 21.7% decode throughput gain), and multi-token speculative decoding via NEXTN reached 3.90 accepted tokens per round on a synthetic workload (185.1% throughput); the three optimizations combined gave a 42.5% throughput increase and 24.5% mean-latency reduction on GSM8K with no statistically detectable accuracy change. Deployment code and instructions are in the Ling cookbook.