Tech Meridian ← ENTITY INDEX
PROMY MERIDIAN RU

TOPIC · ENTITY #17457

decode phase

Related event timeline, sources and context from the news index.

EVENT TIMELINE

2

RESEARCH · 1 SOURCE · arXiv cs.AI

GAUSS: workload-aware analytical and Monte‑Carlo simulator for LLM inference serving

GAUSS is a workload-aware simulator (arXiv:2610.37989v1) that predicts full distributions of time-to-first-token (TTFT) and time-between-tokens (TBT) plus sustainable request rates for LLM inference. It couples phase-specific latency surfaces for prefill and decode with a Lindley-recurrence queueing model and uses ensemble Monte‑Carlo sampling to capture tail latencies and evaluate operating points, routing, and scheduling policies without relying on single-trace replay.

7.0

CODING · 1 SOURCE · AWS Machine Learning

Amazon SageMaker details fused MoE kernel optimizations for NVIDIA Hopper WGMMA constraints

Amazon SageMaker engineers describe kernel-level optimizations for fused Mixture-of-Experts (MoE) models, focusing on performance bottlenecks introduced by NVIDIA Hopper’s WGMMA instruction which fixes the M dimension at 64. The post explains how decode-time workloads (often M=1) and the large N dimensions of fused MoE (e.g., topk=8 aggregating many experts) create operand-shape asymmetry that starves Tensor Core lanes, and presents implemented optimizations and measured benefits using a representative MoE model.

7.0