RESEARCH · RESEARCH · #2059
MLX enables local and multi‑Mac inference of Muse Spark Open (816B) on Mac Studio
The MLX array framework added Apple Silicon support and demonstrates running frontier LLMs locally and in a distributed setup on Mac Studio. Using the latest MLX release and RDMA over Thunderbolt, a four‑node Mac Studio cluster runs Muse Spark Open (816B, MoE, NVFP4) at >36 tokens/s; smaller models (Qwen 3.8 and Mistral Medium 3.5) can fit on a single 512GB Mac Studio. Benchmarks with mlx_lm.benchmark report up to 4.6× faster time‑to‑first‑token and ~1.6× higher tokens/s on M5 Ultra versus M3 Ultra, and modest peak power (~200–300W) for inference.
KEY POINTS
- The MLX array framework added Apple Silicon support and demonstrates running frontier LLMs locally and in a distributed setup on Mac Studio.
- Using the latest MLX release and RDMA over Thunderbolt, a four‑node Mac Studio cluster runs Muse Spark Open (816B, MoE, NVFP4) at >36 tokens/s; smaller models (Qwen 3.8 and Mistral Medium 3.5) can fit on a single 512GB Mac Studio.
- Benchmarks with mlx_lm.benchmark report up to 4.6× faster time‑to‑first‑token and ~1.6× higher tokens/s on M5 Ultra versus M3 Ultra, and modest peak power (~200–300W) for inference.
WHY IT MATTERS
Demonstrates that commodity Apple Silicon desktops can run and scale frontier LLMs (including an 816B MoE) with good latency, throughput, and energy efficiency, lowering barriers for private/local AI research and development.