vLLM

2026-10-30

COMPANIES · 1 SOURCE · NVIDIA Developer

NVIDIA packages Dynamo 1.4.2 into validated Kubernetes runtime recipes with AICR

NVIDIA's DSX AI Factory Platform adds Dynamo 1.4.2 and its cluster dependencies into reproducible, validated runtime recipes via the NVIDIA AI Cluster Runtime (AICR). AICR pins component versions, captures GPU/DR A, scheduling, networking and monitoring requirements, offers a five-step render/deploy/validate flow (snapshot, generate recipe, render bundle, deploy via Helmfile/GitOps, validate), and documents routing options (TCP + ZMQ) including a Gateway API / EPP ingress path; NATS is now opt-in.

6.0

2026-10-28

RESEARCH · 1 SOURCE · arXiv cs.AI

WAKMan — workload-aware KV-cache manager for LLM servers

WAKMan is a workload-aware KV-cache manager (arXiv:2610.36355v1) that selects an eviction policy from live traffic without inspecting prompts or knowing request types to improve prefix-caching hit rates for LLM servers. Evaluated across mixtures of single-turn, multi-turn and agentic traffic, WAKMan raises hit rate over LRU by about 5 percentage points on average and achieves the lowest mean and worst-case optimality gap among tested policies.

6.0

2026-10-25

COMPANIES · 1 SOURCE · InfoQ AI, ML & Data Engineering

Google open-sources ML Drift GPU engine for on-device AI inference

Google released ML Drift, an Apache-2.0 open-source, cross-platform GPU compute engine for on-device AI and ML inference that abstracts multiple low-level APIs (OpenGL ES, OpenCL, Metal, WebGPU). ML Drift powers LiteRT, adds unified shaders via tensor virtualization, extensible custom ops, 5D tensor support, and hardware-specific optimizations for Arm, Intel, and Qualcomm GPUs; Google says it replaces the legacy TensorFlow Lite GPU delegate and reports lower memory overhead and app-level latency improvements in Chrome, YouTube Shorts, Photos, Adobe apps, and Snapchat.

8.0

2026-10-22

COMPANIES · 1 SOURCE · NVIDIA Developer

NVIDIA adds official eight-node DGX Spark clusters with 1 TB unified memory to run frontier models

At GTC Berlin NVIDIA announced official support for eight-node DGX Spark clusters, scaling unified memory to 1 TB and enabling local execution of frontier models including the full GLM 5.3 (753B) from Z.ai. NVIDIA also introduced a Spark-optimized collective backend called RoCEnante (merged into FlashInfer) that reduces all-reduce latency—reporting a 4.43x decode speedup at eight nodes (vs 3.05x without it)—and published benchmarks (e.g., Qwen3.8-27B NVFP4 scaling from ~12 to 52 tokens/sec, GLM 5.3 ~80 TPS, and 7.79x pretraining speedup at 97.4% efficiency for a nanochat model); NeMo Switchyard and vLLM integration are highlighted for multi-model routing and distributed inference, and NVIDIA Sync eight-node support is due later this year.

7.0

2026-10-21

MODELS · 1 SOURCE · Hugging Face

IBM updates Granite Speech suite with granite-speech-5.0-2b and two revised models

IBM released three updates to its Granite Speech collection: a new speech-language model, granite-speech-5.0-2b, and revised granite-speech-5.0-470m-turboctc and granite-speech-4.1-2b-plus. The 5.0-2b model (trained on ~390,000 hours of public audio) expands multilingual transcription and translation and improves English ASR robustness; the Turbo CTC 470M revision adds multilingual support, native punctuation, and faster runtime (~19,870 RTFx); the 4.1-2b-plus revision improves speaker attribution, timestamps, long-form WER, and keyword biasing. All models ship under Apache 2.0 with integrations for Transformers, vLLM, llama.cpp and mlx-audio.

7.0

2026-10-20

MODELS · 1 SOURCE · The Decoder

Deepseek open-sources DeepInfer inference engine used by V4.1-Flash

Deepseek has released DeepInfer, its internal inference framework, under the Apache License 2.0. DeepInfer — which builds on vLLM with optimized compute kernels, distributed multi-chip processing, and faster text-generation techniques — powers the company's V4.1-Flash model; Deepseek also published deepseek-recipe to replicate its API token processing and plans to open-source code for future models.

7.0

2026-10-20

MODELS · 1 SOURCE · AWS Machine Learning

GPT-OSS-20B fine-tuned for multilingual reasoning on Amazon SageMaker HyperPod

Teams fine-tuned the open-weight reasoning model GPT-OSS-20B on a multilingual chain-of-thought dataset (HuggingFaceH4/Multilingual-Thinking) using supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) on Amazon SageMaker HyperPod with EKS, LoRA, TRL/veRL, vLLM, Ray/KubeRay and Amazon FSx for Lustre. Results: the base model almost always produced English chains-of-thought; SFT corrected most of the language behavior gap, and GRPO provided additional improvements only when the desired behavior had non-zero probability in the base model; a Qwen2.5-7B control experiment supported the same conclusion.

6.0

2026-10-19

CODING · 1 SOURCE · AWS Machine Learning

vLLM adds push-based NIXL connector to speed KV transfer for disaggregated inference

vLLM introduced a push-based NIXL connector that moves KV-transfer coordination off the decode critical path for prefill/decode disaggregated inference. In benchmarks on EC2 P5en instances with EFA, push mode reportedly reduced time-to-first-token by up to 3× and improved time-per-output-token by up to 30% for Qwen3-32B, with results varying by workload and configuration.

6.0

2026-10-19

CODING · 1 SOURCE · AWS Machine Learning

vLLM adds bi-directional KV transfer to reduce TTFT for multi-turn agent workloads

vLLM implemented bi-directional key-value (KV) cache transfer so decode nodes can return cached KV blocks to prefill nodes across turns, avoiding repeated recomputation of earlier responses and context. In 1P1D Amazon EC2 P5en+EFA benchmarks the change reduced time-to-first-token (TTFT) by up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8; results vary by workload and config.

7.0

2026-10-18

MODELS · 4 SOURCES · Google · Google DeepMind · The Verge AI · The Decoder

Google launches Gemini 4 Argon, a new frontier model, globally

Google today made Gemini 4 Argon globally available to developers and enterprises via the API in AI Studio and the Gemini platform, and to Google AI Ultra/Pro subscribers in Antigravity, the Gemini app, and AI Mode in Search. Google positions Argon as a high‑precision frontier model for complex multi‑step tasks across coding, finance, legal, cybersecurity, and science, citing top ranking on the Vals Index, example deployments inside Google (Chrome refactor, ADK optimizations), technical speedups (xTB and Gemma inference via vLLM), and introductory pricing ($2 per million input tokens, $10 per million output tokens, with cached inputs 95% off).

9.0

2026-10-16

MODELS · 1 SOURCE · AWS Machine Learning

Reflection releases Beam 1.0 — a 501B-parameter open-weight agentic coding model available on AWS

Reflection AI released Beam 1.0, an open-weight sparse mixture-of-experts model (501 billion total parameters, 23 billion active parameters, 1M token effective context) for agentic coding and long-horizon workflows. Beam 1.0 weights are published on Hugging Face under an Apache 2.0 license and the model is available via Amazon SageMaker JumpStart in US East (N. Virginia and Ohio) with deployment options including JumpStart real-time endpoints, SageMaker HyperPod Inference, and self-managed runtimes via SGLang or vLLM; checkpoints are offered in BF16, FP8, and NVFP4 formats.

8.0

2026-10-16

MODELS · 1 SOURCE · Hugging Face

Reflection releases Beam — 501B MoE open model with 1M-token context (Apache‑2.0)

Reflection published Beam, a 501B-parameter mixture-of-experts (MoE) language model on Hugging Face under the Apache‑2.0 license. Beam runs ~23B parameters per token, supports text input with a 1M-token context window, targets coding and agentic use cases, ships with a Rust performance harness and day‑0 support for transformers, vLLM and SGLang, and includes recipes for inference, SFT and RL.

8.0

2026-10-15

MODELS · 1 SOURCE · Hugging Face

Granite Switch 4.1 integrates decision adapters and fastest OpenASR model into one endpoint

Granite Switch 4.1 (8B and 30B) bundles a general Decision aLoRA, a specialized Guardian aLoRA that the team reports outperforms several safety-focused decision models on related tasks, and Granite-Speech-5.0-Turbo-CTC (tied to the OpenASR leaderboard) into a single hosted endpoint. The adapters are cache-efficient Activated LoRAs that reuse the base model's KV cache, keeping VRAM use similar to the base model while enabling fast switching between chat, decision, and speech-to-text workloads; a Hugging Face demo (“Speech-to-Decisions”) shows real-time transcription, summaries and safety screening on one endpoint.

7.0

2026-10-15

MODELS · 4 SOURCES · Hugging Face · NVIDIA Developer · The Decoder · TechCrunch AI

Meta releases Muse Spark Open — 816B-parameter open-weight multimodal MoE model

Meta published Muse Spark Open, an open-weight mixture-of-experts (MoE) model with ~816 billion parameters (≈42B active per token) that accepts text, images, audio, and video and supports up to a 1,048,576-token context window. Meta says the model is optimized for agentic coding workflows (reading, writing, debugging code and tool calls); checkpoints are available in BF16 (safetensors, ~1.5 TB) and an NVFP4 4-bit format tuned for NVIDIA Blackwell GPUs.

9.0

2026-10-14

CODING · 1 SOURCE · InfoQ AI, ML & Data Engineering

Google's Antigravity SDK adds local model support for hybrid cloud–local agent workflows

Google added support for local models to the Antigravity SDK, letting developers run agent 'builders' on-device while the higher‑context 'architect' can remain in the cloud. The initial implementation targets running Gemma 4 26B A4B via LiteRT, but the SDK also supports OpenAI‑compatible local inference servers (LocalOpenAIAgentConfig) such as Ollama, LM Studio, and vLLM; Google reports an example where ~97.2% of tokens were consumed locally and no source code was uploaded to the cloud.

7.0

2026-10-12

RESEARCH · 1 SOURCE · NVIDIA Developer

NVIDIA and Dao AI Lab publish ReplaySSM to speed SSM-based hybrid-model decoding

NVIDIA and the Dao AI Lab (Princeton) released ReplaySSM (paper on arXiv and code on GitHub), a technique that caches recent SSM inputs instead of full state to halve state memory traffic and speed SSM decoding kernels by ~1.46–1.80× (batch 256 on Blackwell Ultra B300). ReplaySSM also simplifies speculative-decoding rollback (1.85–2.10× higher speculative throughput on 120B-class models) and enables mixed-precision state (three of four tested models stay within ~2 percentage points of FP32 on GPQA‑Diamond); it's integrated into TensorRT‑LLM, vLLM, and FlashInfer and runs in production in GLM‑5.3‑Flash and Kimi‑K3.

8.0

2026-10-12

CODING · 1 SOURCE · AWS Machine Learning

AWS launches Ray LLM DLC for multi-node LLM inference on Amazon EKS

AWS published a guide and released a Ray LLM Deep Learning Container (DLC) that supports deploying a single large language model across multiple GPU nodes on Amazon EKS, using a disaggregated prefill/decode setup. The Ray LLM DLC packages vLLM, Ray Serve LLM, NIXL for KV-cache transfer, and the EFA stack, exposes an OpenAI‑compatible endpoint, and includes a helper (build_pd_openai_app) plus example code in a repository.

6.0

2026-10-01

RESEARCH · 1 SOURCE · arXiv cs.AI

Decode-Latency Feedback Prefill (DLFP): a model-free controller for prefill chunking

The paper (arXiv:2609.38386v1) introduces Decode-Latency Feedback Prefill (DLFP), a model-free controller implemented in vLLM that adaptively resizes prefill chunks overlapping active decodes. On Qwen3-0.6B (BF16) running on a single NVIDIA A100 80 GB, three paired 100-request trials reported mean reductions in P99 inter-token latency of 27.7% (paired 95% CI 21.0%–34.3%) with exact output agreement and unchanged SLO compliance, at the cost of a 34.8% mean increase in P99 time-to-first-token; DLFP did not generalize to Qwen3-8B, Qwen3-32B, or a two-GPU tensor-parallel setup, a failure traced to using an asynchronous scheduler-call interval as a proxy for GPU iteration completion.

6.0

2026-09-29

COMPANIES · 1 SOURCE · NVIDIA Developer

NVIDIA releases VSS Blueprint 3.3 with Build Vision Agent skill and Adaptive EVS

NVIDIA’s Metropolis Video Search and Summarization (VSS) Blueprint 3.3 adds the vss-build-vision-ai skill to compose multi-workflow visual AI deployments from a single prompt and introduces Adaptive Efficient Video Sampling (EVS) to prune redundant visual tokens and batch VLM work. NVIDIA cites a demo that deployed a bottling-line agent in under 30 minutes on an RTX PRO 6000 Blackwell and reports up to ~80% fewer VLM input tokens for a 60-minute summary and a 46% increase in concurrent real-time VLM streams in their tests; the repository and agent skills are available and a live demo is scheduled for Oct. 1 at 9 a.m. PT.

7.0

2026-09-28

CODING · 1 SOURCE · AWS Machine Learning

Deploy streamed Qwen3-TTS on SageMaker using vLLM-Omni v1.5 for real-time voice

AWS published a Part 1 tutorial showing how to deploy the Qwen3-TTS text-to-speech model on Amazon SageMaker AI using the vLLM-Omni v1.5 Deep Learning Container (DLC). The guide demonstrates sending text and receiving audio chunks over a single persistent bidirectional connection (SageMaker bidirectional streaming) via the vLLM-Omni native WebSocket route (v1/audio/speech/stream) and includes a Gradio example; the series will also cover other specialized DLCs like WhisperX and llama.cpp.

6.0

2026-09-25

COMPANIES · 1 SOURCE · AWS Machine Learning

Run SkyRL GRPO post-training of Qwen3‑VL‑8B on Amazon SageMaker HyperPod

Amazon demonstrates running the open-source SkyRL framework on SageMaker HyperPod (EKS) to post-train a Qwen3‑VL‑8B vision‑language model with Group Relative Policy Optimization (GRPO). Using a HyperPod Ray cluster with shared FSx storage, the workflow improved maze solve rate from 43.75% to over 95% on a fixed 64‑maze evaluation set and uses colocated vLLM inference and FSDP-sharded policy training with LoRA adapters.

6.0

2026-09-22

COMPANIES · 1 SOURCE · AWS Machine Learning

Amazon SageMaker AI adds built-in concurrency sweeps to right-size generative AI endpoints

Amazon describes using concurrency sweeps—integrated in SageMaker AI Inference Recommendations—to benchmark generative AI endpoints by increasing simultaneous request levels and measuring throughput and latency. The post walks through deploying the NVIDIA Nemotron-3 Nano 30B on an ml.g7e.2xlarge with the vLLM container and running CreateAIBenchmarkJob (with workload defined via CreateAIWorkloadConfig) to discover saturation points and right-size capacity.

6.0

2026-09-21

RESEARCH · 1 SOURCE · arXiv cs.AI

RBS-Attention: radius-adaptive dual-branch sparse prefill for long-context LLMs

arXiv preprint arXiv:2609.20971v1 introduces RBS-Attention, a training-free sparse-prefill method that combines a centroid base branch with a radius-based rescue branch to avoid 'mean dilution' when selecting sparse blocks for long-context self-attention. On H100 GPUs the method reports up to 20.65× standalone prefill-attention speedup, 11.92× vLLM prefill-attention speedup and 5.97× end-to-end time-to-first-token speedup at 128K on Qwen3-30B-A3B-Instruct-2507-FP8, while retaining near-dense accuracy (88.65 vs 89.52 RULER on dense Qwen3-32B); evaluations include LongBench-v2, InfiniteBench and Video-MME.

7.0

2026-09-18

COMPANIES · 1 SOURCE · NVIDIA Developer

NVIDIA releases AIPerf, a multiprocess LLM benchmarking tool

NVIDIA AIPerf replaces GenAI-Perf with a ground-up multiprocess architecture designed to avoid client-side bottlenecks for high-concurrency LLM inference benchmarking. It supports 15+ endpoint types, public datasets and trace replay formats (ShareGPT, Mooncake, Baseten, WEKA/AgentX), configurable arrival patterns, and reports TTFT, ITL, latency percentiles and GPU telemetry when DCGM or pynvml are available.

6.0

2026-09-18

CODING · 1 SOURCE · AWS Machine Learning

Hugging Face publishes six open-source Skills to deploy models on Amazon SageMaker AI via coding agents

Hugging Face published six open-source 'Skills' (GitHub) that let coding agents orchestrate end-to-end deployment of Hugging Face models to Amazon SageMaker AI. The skills automate selecting the correct serving container from AWS Deep Learning Containers, creating real-time or serverless endpoints with autoscaling and CloudWatch alarms, and provide verified teardown paths; they run using Python and the AWS CLI and are designed to prevent fragile or costly agent-made deployment mistakes.

6.0

2026-09-16

CODING · 1 SOURCE · Cohere

Cohere publishes megakernel serving engine for North Mini Code with faster H100 decoding

Cohere describes a megakernel-based serving engine for its North Mini Code 30B model that runs BF16 on a single NVIDIA H100 and claims 1.25×–1.41× end-to-end speedup over vLLM, with a reported 292 tok/s (62% of SoL) at batch size 1 — about 1.58× faster than vLLM. The system supports production features (continuous batching, paged attention, ragged sequences), an OpenAI-compatible endpoint with tool calling, is implemented as a single CUDA file, and the code is available on GitHub.

7.0

2026-09-09

MODELS · 1 SOURCE · AWS Machine Learning

Deploying Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod with vLLM

AWS Machine Learning provides a walkthrough for deploying Qwen3.8-2.4T-A95B, a 2.4‑trillion‑parameter open‑weight model, on Amazon SageMaker HyperPod using vLLM. The guide covers cluster provisioning, NVFP4 quantization, and creating an OpenAI‑compatible endpoint with built‑in reasoning, tool calling, and native MTP speculative decoding.

7.0

2026-03-16

MODELS · 1 SOURCE · Mistral AI

Mistral releases Small 4: 119B MoE multimodal model with 256k context (Apache 2.0)

Mistral announced Mistral Small 4, a 119B-parameter hybrid Mixture-of-Experts model (128 experts, 4 active) that accepts text and image inputs, offers a 256k context window, and includes a configurable reasoning_effort parameter; it is released under the Apache 2.0 license. The company says Small 4 unifies capabilities from its Magistral, Pixtral, and Devstral lines, targets chat, coding/agentic, and complex-reasoning use cases, claims substantial latency and throughput gains versus Mistral Small 3, and is available across vLLM, llama.cpp, SGLang, Transformers and other runtimes.

9.0