Dense vs. MoE Models: Active Parameters, Throughput, and When to Choose Each
An NVIDIA Developer post compares dense and Mixture-of-Experts (MoE) architectures, showing how a 30B-parameter model can activate only about 3B parameters per token and discussing the resulting capacity and throughput trade-offs; Nemotron 3.5 Lightning is used as an illustrative example.