RESEARCH · RESEARCH · #2700
arXiv:2610.36329v1 — 100 ms GPU monitoring finds major underutilization in a university cluster
The paper presents a lightweight monitoring tool (using NVIDIA DCGM) that records GPU compute metrics at 100 ms intervals and reports results from a deployment on an 89-node, 319-GPU university cluster over a 10-day period covering 7,399 jobs by 150 users. Key findings: >70% of jobs use under 20% of GPU memory, 50% average below 5% SM utilization, 80% of jobs mismatch requested vs actual runtimes, and multi-GPU jobs show highly periodic communication patterns; the authors release the monitoring code on GitHub.
KEY POINTS
- The paper presents a lightweight monitoring tool (using NVIDIA DCGM) that records GPU compute metrics at 100 ms intervals and reports results from a deployment on an 89-node, 319-GPU university cluster over a 10-day period covering 7,399 jobs by 150 users.
- Key findings: >70% of jobs use under 20% of GPU memory, 50% average below 5% SM utilization, 80% of jobs mismatch requested vs actual runtimes, and multi-GPU jobs show highly periodic communication patterns; the authors release the monitoring code on GitHub.
- Fine-grained measurements show clear opportunities to improve GPU utilization via better scheduling and multitasking without hardware changes, which could reduce queue times and costs.
WHY IT MATTERS
Fine-grained measurements show clear opportunities to improve GPU utilization via better scheduling and multitasking without hardware changes, which could reduce queue times and costs.