NEWS · CODING · #407
Cohere publishes megakernel serving engine for North Mini Code with faster H100 decoding
Cohere describes a megakernel-based serving engine for its North Mini Code 30B model that runs BF16 on a single NVIDIA H100 and claims 1.25×–1.41× end-to-end speedup over vLLM, with a reported 292 tok/s (62% of SoL) at batch size 1 — about 1.58× faster than vLLM. The system supports production features (continuous batching, paged attention, ragged sequences), an OpenAI-compatible endpoint with tool calling, is implemented as a single CUDA file, and the code is available on GitHub.
KEY POINTS
- Cohere describes a megakernel-based serving engine for its North Mini Code 30B model that runs BF16 on a single NVIDIA H100 and claims 1.25×–1.41× end-to-end speedup over vLLM, with a reported 292 tok/s (62% of SoL) at batch size 1 — about 1.58× faster than vLLM.
- The system supports production features (continuous batching, paged attention, ragged sequences), an OpenAI-compatible endpoint with tool calling, is implemented as a single CUDA file, and the code is available on GitHub.
- It demonstrates that a decode megakernel can be productionized to significantly raise small‑batch LLM throughput on H100s and better approach HBM bandwidth limits.
WHY IT MATTERS
It demonstrates that a decode megakernel can be productionized to significantly raise small‑batch LLM throughput on H100s and better approach HBM bandwidth limits.