NEWS · CODING · #1911
Amazon SageMaker details fused MoE kernel optimizations for NVIDIA Hopper WGMMA constraints
Amazon SageMaker engineers describe kernel-level optimizations for fused Mixture-of-Experts (MoE) models, focusing on performance bottlenecks introduced by NVIDIA Hopper’s WGMMA instruction which fixes the M dimension at 64. The post explains how decode-time workloads (often M=1) and the large N dimensions of fused MoE (e.g., topk=8 aggregating many experts) create operand-shape asymmetry that starves Tensor Core lanes, and presents implemented optimizations and measured benefits using a representative MoE model.
KEY POINTS
- Amazon SageMaker engineers describe kernel-level optimizations for fused Mixture-of-Experts (MoE) models, focusing on performance bottlenecks introduced by NVIDIA Hopper’s WGMMA instruction which fixes the M dimension at 64.
- The post explains how decode-time workloads (often M=1) and the large N dimensions of fused MoE (e.g., topk=8 aggregating many experts) create operand-shape asymmetry that starves Tensor Core lanes, and presents implemented optimizations and measured benefits using a representative MoE model.
- Because Hopper’s WGMMA enforces a fixed M=64 mapping, MoE decode workloads with tiny M and very large N can leave Tensor Cores underutilized; kernel engineering on SageMaker targets the biggest leverage point to cut latency and cost for MoE inference.
WHY IT MATTERS
Because Hopper’s WGMMA enforces a fixed M=64 mapping, MoE decode workloads with tiny M and very large N can leave Tensor Cores underutilized; kernel engineering on SageMaker targets the biggest leverage point to cut latency and cost for MoE inference.