Mixture-of-Experts (MoE)

2026-10-31

RESEARCH · 1 SOURCE · arXiv cs.AI

LRMoE: demand-side load recalibration for memory-aware MoE training

A new arXiv paper (arXiv:2610.37720v1) introduces LRMoE, a memory-aware training system for sparse Mixture-of-Experts models that jointly plans retention quotas and recalibrates backward demand to reduce communication, computation, and buffer use on memory-constrained GPUs. Evaluated on Qwen3-30B-A3B and GLM4.5-106B-A12B, LRMoE reportedly cuts cumulative MoE-layer training time to fixed-loss targets by 21.2–44.8% versus the SYMI baseline.

7.0

2026-10-16

MODELS · 1 SOURCE · Hugging Face

Reflection releases Beam — 501B MoE open model with 1M-token context (Apache‑2.0)

Reflection published Beam, a 501B-parameter mixture-of-experts (MoE) language model on Hugging Face under the Apache‑2.0 license. Beam runs ~23B parameters per token, supports text input with a 1M-token context window, targets coding and agentic use cases, ships with a Rust performance harness and day‑0 support for transformers, vLLM and SGLang, and includes recipes for inference, SFT and RL.

8.0

2026-10-15

MODELS · 4 SOURCES · Hugging Face · NVIDIA Developer · The Decoder · TechCrunch AI

Meta releases Muse Spark Open — 816B-parameter open-weight multimodal MoE model

Meta published Muse Spark Open, an open-weight mixture-of-experts (MoE) model with ~816 billion parameters (≈42B active per token) that accepts text, images, audio, and video and supports up to a 1,048,576-token context window. Meta says the model is optimized for agentic coding workflows (reading, writing, debugging code and tool calls); checkpoints are available in BF16 (safetensors, ~1.5 TB) and an NVFP4 4-bit format tuned for NVIDIA Blackwell GPUs.

9.0

2026-10-13

CODING · 1 SOURCE · AWS Machine Learning

Amazon SageMaker details fused MoE kernel optimizations for NVIDIA Hopper WGMMA constraints

Amazon SageMaker engineers describe kernel-level optimizations for fused Mixture-of-Experts (MoE) models, focusing on performance bottlenecks introduced by NVIDIA Hopper’s WGMMA instruction which fixes the M dimension at 64. The post explains how decode-time workloads (often M=1) and the large N dimensions of fused MoE (e.g., topk=8 aggregating many experts) create operand-shape asymmetry that starves Tensor Core lanes, and presents implemented optimizations and measured benefits using a representative MoE model.

7.0

2026-10-07

RESEARCH · 1 SOURCE · arXiv cs.AI

JIVEAdapter: an additive multi-task low-rank adapter via Joint and Individual Variation Explained

The paper introduces JIVEAdapter, an additive low-rank adapter that decomposes each weight update into a shared Joint component and per-task Individual components, enforces near-orthogonality between them, and adaptively allocates rank across shared and task-specific pools. Evaluated with DeBERTaV3-base on GLUE and SuperGLUE, JIVEAdapter matches strong single- and multi-task low-rank baselines at the same per-task effective rank and supports freezing a jointly learned Joint for reuse with cheap per-task updates.

6.0

2026-10-01

MODELS · 1 SOURCE · Hugging Face

Olmo launches Olmo-core 3 — open, scalable MoE training infrastructure

Olmo has released Olmo-core 3, a redesigned open mixture-of-experts (MoE) training framework intended to scale MoE models into the trillion-parameter range while preserving computational efficiency. The stack replaces an FSDP-based implementation with a DDP-centered system, combines expert/pipeline parallelism and other optimizations, reports a 2.7× throughput uplift on a 47B-parameter MoE across eight NVIDIA B300 GPUs, and shows MXFP8 delivering about 21% higher throughput than BF16 in controlled tests.

7.0

2026-09-25

COMPANIES · 1 SOURCE · AWS Machine Learning

Amazon describes EKS + EFA + DeepEP architecture to scale MoE reinforcement learning with up to 40% higher throughput

An Amazon post details an architecture that combines Amazon EKS, Elastic Fabric Adapter (EFA), and DeepEP to address the heterogeneous compute, high-bandwidth communication, and orchestration challenges of post-training MoE reinforcement learning (including RLHF and GRPO). The write-up explains balancing rollout generation and tightly coupled policy training, optimizing Expert Parallelism (EP) communication over EFA, and reports up to a 40% throughput improvement using DeepEP.

6.0

2026-09-21

RESEARCH · 1 SOURCE · arXiv cs.AI

Attention-Aware Routing (AAR) for MoE models (arXiv:2609.20974v1)

The paper introduces Attention-Aware Routing (AAR), which augments Mixture-of-Experts routers with temporal and spectral features derived from a sliding window of attention weights while keeping the base transformer frozen and training only routing parameters. On OLMoE, AAR improves GSM8K accuracy by +3.37 percentage points over a routing-only SFT baseline, demonstrates that routing updates propagate to reshape subsequent-layer attention without changing attention weights directly, reduces long diverging generations for incorrect answers, and shows strong depth sensitivity that separates retrieval and reasoning behavior across layers.

7.0

2026-09-16

MODELS · 1 SOURCE · Cohere

Cohere releases open-source "Transcribe Arabic" ASR model claiming best open-weight Arabic accuracy

Cohere published Transcribe Arabic, an open-source Arabic automatic speech recognition (ASR) model (Apache 2.0) based on its 2B ASR work. Cohere says the model achieves a 25.87 WER on the Hugging Face Arabic ASR leaderboard, outperforming Meta’s OmniASR-LLM-7B and OpenAI’s Whisper Large V3, and was preferred to Whisper in about 96% of human evaluation tests; weights are available on Hugging Face and via the Cohere API/Model Vault.

8.0

2026-09-16

RESEARCH · 1 SOURCE · Cohere

Cohere introduces hardware-aware Dynamic Speculative Decoding (DSD) to adapt draft-token count to GPU constraints

Cohere presents Dynamic Speculative Decoding (DSD), an extension of speculative decoding that selects the number of draft tokens K based on hardware regime (memory-bandwidth vs compute-bound) and model type (dense vs MoE). DSD uses offline profiling of Acceptance Length (AL) and Inter-Token Latency (ITL) to choose an optimal K per condition, aiming to preserve SD speedups in production workloads and large-scale RL rollouts.

7.0

2026-09-15

MODELS · 1 SOURCE · NVIDIA Developer

Dense vs. MoE Models: Active Parameters, Throughput, and When to Choose Each

An NVIDIA Developer post compares dense and Mixture-of-Experts (MoE) architectures, showing how a 30B-parameter model can activate only about 3B parameters per token and discussing the resulting capacity and throughput trade-offs; Nemotron 3.5 Lightning is used as an illustrative example.

6.0