RESEARCH · RESEARCH · #2873
GAUSS: workload-aware analytical and Monte‑Carlo simulator for LLM inference serving
GAUSS is a workload-aware simulator (arXiv:2610.37989v1) that predicts full distributions of time-to-first-token (TTFT) and time-between-tokens (TBT) plus sustainable request rates for LLM inference. It couples phase-specific latency surfaces for prefill and decode with a Lindley-recurrence queueing model and uses ensemble Monte‑Carlo sampling to capture tail latencies and evaluate operating points, routing, and scheduling policies without relying on single-trace replay.
KEY POINTS
- GAUSS is a workload-aware simulator (arXiv:2610.37989v1) that predicts full distributions of time-to-first-token (TTFT) and time-between-tokens (TBT) plus sustainable request rates for LLM inference.
- It couples phase-specific latency surfaces for prefill and decode with a Lindley-recurrence queueing model and uses ensemble Monte‑Carlo sampling to capture tail latencies and evaluate operating points, routing, and scheduling policies without relying on single-trace replay.
- By modeling phase-specific latencies and using ensemble Monte‑Carlo sampling, GAUSS can predict tail latency and capacity trade-offs that trace-driven simulators under-sample, aiding deployment and scheduler decisions for LLM services.
WHY IT MATTERS
By modeling phase-specific latencies and using ensemble Monte‑Carlo sampling, GAUSS can predict tail latency and capacity trade-offs that trace-driven simulators under-sample, aiding deployment and scheduler decisions for LLM services.