NEWS · MODELS · #500
TensorRT Edge-LLM runs Qwen3.6-27B on Jetson AGX Thor, completes MLPerf Edge Agentic 6.4× faster
NVIDIA's TensorRT Edge-LLM ran Qwen3.6-27B on a single Jetson AGX Thor Developer Kit and achieved 52.33 tokens/sec in the MLPerf Inference v6.1 Edge Agentic performance workload, completing all 1,007 turns in 24 minutes 36 seconds — 6.4× faster than the llama.cpp Jetson reference (2h37m). The submission used NVFP4 quantization for weights/activations, FP8 for the KV cache, tree-based multi-token prediction, and KV-cache/recurrent-state reuse to accelerate long-context agent decoding.
KEY POINTS
- NVIDIA's TensorRT Edge-LLM ran Qwen3.6-27B on a single Jetson AGX Thor Developer Kit and achieved 52.33 tokens/sec in the MLPerf Inference v6.1 Edge Agentic performance workload, completing all 1,007 turns in 24 minutes 36 seconds — 6.4× faster than the llama.cpp Jetson reference (2h37m).
- The submission used NVFP4 quantization for weights/activations, FP8 for the KV cache, tree-based multi-token prediction, and KV-cache/recurrent-state reuse to accelerate long-context agent decoding.
- This shows techniques (NVFP4 quantization, FP8 KV cache, MTP, KV reuse) and optimized kernels can enable practical, much faster on-device agentic LLM inference with long contexts on constrained edge GPUs.
WHY IT MATTERS
This shows techniques (NVFP4 quantization, FP8 KV cache, MTP, KV reuse) and optimized kernels can enable practical, much faster on-device agentic LLM inference with long contexts on constrained edge GPUs.