Amazon SageMaker details fused MoE kernel optimizations for NVIDIA Hopper WGMMA constraints
Amazon SageMaker engineers describe kernel-level optimizations for fused Mixture-of-Experts (MoE) models, focusing on performance bottlenecks introduced by NVIDIA Hopper’s WGMMA instruction which fixes the M dimension at 64. The post explains how decode-time workloads (often M=1) and the large N dimensions of fused MoE (e.g., topk=8 aggregating many experts) create operand-shape asymmetry that starves Tensor Core lanes, and presents implemented optimizations and measured benefits using a representative MoE model.