AWS Trainium

2026-10-21

CODING · 1 SOURCE · AWS Machine Learning

Train Qwen3 on AWS Trainium using torchtitan

A torchtitan walkthrough demonstrates three ways to train Qwen3 on a single AWS Trainium trn2.48xlarge: (1) via Transformers v5 backend (AutoModelForCausalLM) reading/writing Hugging Face safetensors; (2) using torchtitan’s native Qwen3Model via config.model_spec; and (3) adding NKI (Neuron Kernel Interface) mixture-of-experts kernels from torchtitan-neuron and attaching custom kernels with the override registry. The post covers environment setup (Neuron PyTorch container or AWS DL AMI), installing torchtitan-neuron, torchrun launch patterns, checkpoint loading/saving (safetensors), and reports a verified step-1 loss example for Qwen3-4B-Instruct-2507 (6.54 from published weights vs 12.43 from random init).

5.0

2026-10-21

CODING · 1 SOURCE · Hugging Face

Profiling TorchNeuron on AWS Trainium — system-level walkthrough (Part 1)

This tutorial shows how to profile PyTorch workloads on AWS Trainium using TorchNeuron (torch-neuronx). It covers configuring torch.profiler with NeuronConfig and NeuronProfiler, exporting matched system/device traces (NEFF/NTFF), inspecting execution with Neuron Explorer, and compares three RMSNorm implementations (naive PyTorch ops, fused torch.rms_norm, and torch.compile) with reproducible examples in the torch-neuronx examples directory.

6.0

2026-10-20

CODING · 1 SOURCE · Hugging Face

Hugging Face launches TorchNeuron — native PyTorch backend for AWS Trainium (public beta Oct 20, 2026)

TorchNeuron is a new PyTorch plugin from Hugging Face that exposes Neuron as a native PyTorch device for AWS Trainium, enabling eager execution, torch.compile and native distributed APIs (DTensor, FSDP, TP). It brings Transformers, Diffusers and other Hugging Face libraries to Trainium without Optimum Neuron-specific forks, increases out-of-the-box text-generation model compatibility (89% of top 100 vs 46% with Optimum Neuron v0.4.5), and will be available as a public beta starting Oct 20, 2026 (pip install torch-neuronx).

8.0

2026-10-19

CODING · 1 SOURCE · AWS Machine Learning

vLLM adds push-based NIXL connector to speed KV transfer for disaggregated inference

vLLM introduced a push-based NIXL connector that moves KV-transfer coordination off the decode critical path for prefill/decode disaggregated inference. In benchmarks on EC2 P5en instances with EFA, push mode reportedly reduced time-to-first-token by up to 3× and improved time-per-output-token by up to 30% for Qwen3-32B, with results varying by workload and configuration.

6.0

2026-10-19

CODING · 1 SOURCE · AWS Machine Learning

vLLM adds bi-directional KV transfer to reduce TTFT for multi-turn agent workloads

vLLM implemented bi-directional key-value (KV) cache transfer so decode nodes can return cached KV blocks to prefill nodes across turns, avoiding repeated recomputation of earlier responses and context. In 1P1D Amazon EC2 P5en+EFA benchmarks the change reduced time-to-first-token (TTFT) by up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8; results vary by workload and config.

7.0

2026-09-22

STARTUPS · 1 SOURCE · The Decoder

Biological Computing's neuron-derived 'Neural Dynamics Adapter' to be offered via AWS

San Francisco startup The Biological Computing Co. (TBC) says it has a small proprietary software module — the Neural Dynamics Adapter — derived from measurements of cultured cortical neurons that can plug into open-source diffusion video models; TBC and Amazon Web Services plan to offer the adapter-backed text‑to‑video model on Trainium and via SageMaker/AWS Marketplace, with only an early-access signup available so far. According to TBC's press release and tests on the Oasis Minecraft model, the adapter (≈156k parameters, <0.1% of base size) reportedly preserves image detail longer and yields up to 5× faster generation and ~80% lower inference cost versus the unnamed base model, though TBC does not disclose the base model and the commercial offering is not yet public.

6.0