GUIDE · MODELS · #3546
NVIDIA tutorial: serving GLM-5.3 across two DGX Stations with vLLM
NVIDIA published a tutorial demonstrating how to serve the 753B-parameter GLM-5.3 (NVFP4 quantized MoE model) locally across two DGX Stations with GB300 GPUs using vLLM. The guide shows a TP=2 tensor-parallel setup over ConnectX-8 RDMA (no CPU weight offload), with Station A hosting an OpenAI-compatible API endpoint and Station B acting as a worker, and includes steps for checkpoint transfer (≈464 GB), networking checks, and baseline load measurements.
KEY POINTS
- NVIDIA published a tutorial demonstrating how to serve the 753B-parameter GLM-5.3 (NVFP4 quantized MoE model) locally across two DGX Stations with GB300 GPUs using vLLM.
- The guide shows a TP=2 tensor-parallel setup over ConnectX-8 RDMA (no CPU weight offload), with Station A hosting an OpenAI-compatible API endpoint and Station B acting as a worker, and includes steps for checkpoint transfer (≈464 GB), networking checks, and baseline load measurements.
- This matters because it documents a practical multi‑node GPU setup (RDMA + tensor parallelism) to serve a frontier MoE model that exceeds a single GB300 GPU's memory, enabling local deployment and benchmarking of very large models.
WHY IT MATTERS
This matters because it documents a practical multi‑node GPU setup (RDMA + tensor parallelism) to serve a frontier MoE model that exceeds a single GB300 GPU's memory, enabling local deployment and benchmarking of very large models.