llama.cpp

2026-10-21

MODELS · 1 SOURCE · Hugging Face

IBM updates Granite Speech suite with granite-speech-5.0-2b and two revised models

IBM released three updates to its Granite Speech collection: a new speech-language model, granite-speech-5.0-2b, and revised granite-speech-5.0-470m-turboctc and granite-speech-4.1-2b-plus. The 5.0-2b model (trained on ~390,000 hours of public audio) expands multilingual transcription and translation and improves English ASR robustness; the Turbo CTC 470M revision adds multilingual support, native punctuation, and faster runtime (~19,870 RTFx); the 4.1-2b-plus revision improves speaker attribution, timestamps, long-form WER, and keyword biasing. All models ship under Apache 2.0 with integrations for Transformers, vLLM, llama.cpp and mlx-audio.

7.0

2026-10-20

CODING · 1 SOURCE · Hugging Face

AnyLanguageModel 1.0 released with swift-huggingface and swift-xet 1.0, enabling local and hosted models on Apple platforms

AnyLanguageModel 1.0 is out alongside swift-huggingface 1.0 and swift-xet 1.0, offering a drop-in API that lets Swift apps on iOS, macOS, visionOS and Linux use Foundation Models-compatible local models (via llama.cpp, MLX, Core ML) or hosted providers (Hugging Face, Anthropic Messages API, OpenRouter and OpenAI-compatible endpoints). New features include guided generation across providers, image support using Foundation Models 27 syntax, reasoning transcript handling, short-lived API key closures, and migration guidance for existing apps.

6.0

2026-10-15

MODELS · 4 SOURCES · Hugging Face · NVIDIA Developer · The Decoder · TechCrunch AI

Meta releases Muse Spark Open — 816B-parameter open-weight multimodal MoE model

Meta published Muse Spark Open, an open-weight mixture-of-experts (MoE) model with ~816 billion parameters (≈42B active per token) that accepts text, images, audio, and video and supports up to a 1,048,576-token context window. Meta says the model is optimized for agentic coding workflows (reading, writing, debugging code and tool calls); checkpoints are available in BF16 (safetensors, ~1.5 TB) and an NVFP4 4-bit format tuned for NVIDIA Blackwell GPUs.

9.0

2026-10-12

CODING · 1 SOURCE · AWS Machine Learning

Deploy llama.cpp decision models on Amazon SageMaker AI with AWS DLC and SemIf

AWS published a how-to showing how to use the AWS Deep Learning Container (DLC) for llama.cpp to serve OpenAI‑compatible chat completions via SageMaker InvokeEndpoint, and how to extend that image with SemIf (formerly OpenJev) to return option probabilities and logits for structured decision tasks. The post includes CPU and GPU deployment options, a child‑image build path using CodeBuild/ECR for SemIf, and a benchmark comparing CPU vs GPU latency for SemIf workloads.

6.0

2026-09-28

CODING · 1 SOURCE · AWS Machine Learning

Deploy streamed Qwen3-TTS on SageMaker using vLLM-Omni v1.5 for real-time voice

AWS published a Part 1 tutorial showing how to deploy the Qwen3-TTS text-to-speech model on Amazon SageMaker AI using the vLLM-Omni v1.5 Deep Learning Container (DLC). The guide demonstrates sending text and receiving audio chunks over a single persistent bidirectional connection (SageMaker bidirectional streaming) via the vLLM-Omni native WebSocket route (v1/audio/speech/stream) and includes a Gradio example; the series will also cover other specialized DLCs like WhisperX and llama.cpp.

6.0

2026-09-24

MODELS · 1 SOURCE · AWS Machine Learning

AWS packages WhisperX DLC for SageMaker to deliver speaker‑labeled, per‑word timestamps

AWS provides a WhisperX Deep Learning Container (DLC) for Amazon SageMaker that bundles OpenAI Whisper with wav2vec2 forced alignment and speaker diarization to produce per-word timestamps and speaker labels. The GPU-ready container conforms to SageMaker’s serving contract, supports real-time and asynchronous endpoints, and outputs json, verbose_json, srt, and vtt formats (example image tag: whisperx:3.8.6-cu128-amzn2023-sagemaker).

6.0

2026-09-24

MODELS · 1 SOURCE · Hugging Face

Liquid AI publishes DSpark drafter for LFM2.5-VL-3B, accelerating VLM decoding up to 3.13×

Liquid AI released a DSpark draft model for its vision‑language model LFM2.5‑VL‑3B that adds speculative decoding to speed up token decoding (up to 3.13× on-device, 2.66× on H100) while increasing model size by ~280M parameters (≈8.9%). The drafter ships with day‑one integrations for llama.cpp, MLX‑VLM and SGLang and is available on Hugging Face in Safetensors and GGUF formats; measured end‑to‑end gains ranged up to 2.62× on edge and 2.27× on GPU across MMSpec tasks.

7.0

2026-09-22

MODELS · 1 SOURCE · Hugging Face

Transformers adds GGUF (llama.cpp quant) support on main

Hugging Face's transformers (main branch) can now load and run GGUF quantized checkpoints using llama.cpp/ggml kernels, aiming for efficient local inference (initially optimized for Apple Silicon and Qwen3.5 models). The integration also enables serving GGUF checkpoints via transformers-serve and falls back to dequantization if compatible kernels are unavailable.

7.0

2026-09-16

MODELS · 1 SOURCE · NVIDIA Developer

TensorRT Edge-LLM runs Qwen3.6-27B on Jetson AGX Thor, completes MLPerf Edge Agentic 6.4× faster

NVIDIA's TensorRT Edge-LLM ran Qwen3.6-27B on a single Jetson AGX Thor Developer Kit and achieved 52.33 tokens/sec in the MLPerf Inference v6.1 Edge Agentic performance workload, completing all 1,007 turns in 24 minutes 36 seconds — 6.4× faster than the llama.cpp Jetson reference (2h37m). The submission used NVFP4 quantization for weights/activations, FP8 for the KV cache, tree-based multi-token prediction, and KV-cache/recurrent-state reuse to accelerate long-context agent decoding.

7.0

2026-03-16

MODELS · 1 SOURCE · Mistral AI

Mistral releases Small 4: 119B MoE multimodal model with 256k context (Apache 2.0)

Mistral announced Mistral Small 4, a 119B-parameter hybrid Mixture-of-Experts model (128 experts, 4 active) that accepts text and image inputs, offers a 256k context window, and includes a configurable reasoning_effort parameter; it is released under the Apache 2.0 license. The company says Small 4 unifies capabilities from its Magistral, Pixtral, and Devstral lines, targets chat, coding/agentic, and complex-reasoning use cases, claims substantial latency and throughput gains versus Mistral Small 3, and is available across vLLM, llama.cpp, SGLang, Transformers and other runtimes.

9.0