NEWS · MODELS · #2299
i14 trains and serves Laplace 3B linear-attention model on AWS Trainium2 with TorchNeuron Native
i14 ported its Laplace 3B 3B-parameter linear-attention decoder (using Gated Linear Attention and a fixed 29.5 MB recurrent state) to AWS Trainium2 via TorchNeuron Native and native PyTorch operators. On a trn2.48xlarge instance the team reported ~120,000 training tokens/sec and serving throughput of 11,544 output tokens/sec across 1,280 concurrent sequences after replacing a Triton kernel, optimizing MuonFSDP2 collectives, and statically unrolling the GLA recurrence for the Trainium2 compiler.
KEY POINTS
- i14 ported its Laplace 3B 3B-parameter linear-attention decoder (using Gated Linear Attention and a fixed 29.5 MB recurrent state) to AWS Trainium2 via TorchNeuron Native and native PyTorch operators.
- On a trn2.48xlarge instance the team reported ~120,000 training tokens/sec and serving throughput of 11,544 output tokens/sec across 1,280 concurrent sequences after replacing a Triton kernel, optimizing MuonFSDP2 collectives, and statically unrolling the GLA recurrence for the Trainium2 compiler.
- Demonstrates that custom linear-attention architectures with fixed-size recurrent state can be efficiently trained and served on Trainium2 using native PyTorch tooling, reducing KV-cache growth and improving concurrency and predictable serving costs.
WHY IT MATTERS
Demonstrates that custom linear-attention architectures with fixed-size recurrent state can be efficiently trained and served on Trainium2 using native PyTorch tooling, reducing KV-cache growth and improving concurrency and predictable serving costs.