NEWS · COMPANIES · #770
NVIDIA Dynamo-Triton 26.07 enables TensorRT multi-device inference across GPUs
NVIDIA has enabled TensorRT multi-device inference in Dynamo-Triton release 26.07 (TensorRT multi-device is supported starting in TensorRT 11.0), allowing a single KIND_MODEL instance to own multiple GPUs and present one gRPC endpoint while using NCCL-backed distributed collectives. In a Cosmos 3 Nano video-generation demo, distributing a 44,160-token transformer across up to eight GPUs reduced end-to-end generation latency from 156.6s to 34.2s and produced a reported transformer RPC speedup of 6.09x; users can download Dynamo-Triton 26.07 from NGC and follow the TensorRT multi-device guide to reproduce the setup.
KEY POINTS
- NVIDIA has enabled TensorRT multi-device inference in Dynamo-Triton release 26.07 (TensorRT multi-device is supported starting in TensorRT 11.0), allowing a single KIND_MODEL instance to own multiple GPUs and present one gRPC endpoint while using NCCL-backed distributed collectives.
- In a Cosmos 3 Nano video-generation demo, distributing a 44,160-token transformer across up to eight GPUs reduced end-to-end generation latency from 156.6s to 34.2s and produced a reported transformer RPC speedup of 6.09x; users can download Dynamo-Triton 26.07 from NGC and follow the TensorRT multi-device guide to reproduce the setup.
- This makes it easier to deploy latency-sensitive generative AI workloads across multiple GPUs without client-side rank orchestration, reducing time-to-result for production services.
WHY IT MATTERS
This makes it easier to deploy latency-sensitive generative AI workloads across multiple GPUs without client-side rank orchestration, reducing time-to-result for production services.