github llama.cpp · 2h ago · inference

llama.cpp v0.6.0 adds extended batch API, GLM-5.3-Flash support, and decision model endpoints

from English

llama.cpp v0.6.0 introduces the new llama_batch_ext extended batch API for mixed token and embedding inputs, along with support for the GLM-5.3-Flash 320B hybrid model and the Clef decision model.

The new llama_batch_ext API supports per-token state embeddings for MTP and deepstack models.
Qwen4Exp now offers high-quality support with MTP speculative decoding, providing approximately 1.5x decode speedup on DGX Spark.
A new /v1/systemone server API supports five decision models: laya, julia-1, lev, openjev, and kev.
Metal gains a tensor-API flash attention kernel for F16 KV and MMA mat-mul kernels up to 3x faster on Apple GPUs.
The Web UI receives an overhaul with a Hugging Face Hub data layer and model download pipeline.

This release expands llama.cpp's capabilities for multimodal processing, speculative decoding performance, and decision model integration.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Multimodal

Read original

github llama.cpp · 7d ago · 44 views

llama.cpp b11240 adds typed multimodal input support to /v1/embeddings

The llama.cpp server now supports typed content (vision, audio, video) input for the /v1/embeddings endpoint. This update enables multimodal embedding requests by accepting OpenAI-style wrapped content arrays, allowing text and image_url parts to be processed together.

media MarkTechPost · 10d ago · 46 views

Liquid AI releases LFM2.5-VL-3B-DSpark speculative drafter for up to 3.13x faster decoding

Liquid AI has released LFM2.5-VL-3B-DSpark, an experimental speculative-decoding draft model for its LFM2.5-VL-3B vision-language model. The drafter adds approximately 280M parameters and accelerates token generation without altering the model's output distribution.

lab Hugging Face Blog · 11d ago · 38 views

Liquid AI releases LFM2.5-VL-DSpark to accelerate vision-language model inference

Liquid AI has released the LFM2.5-VL-DSpark draft model, a speculative decoding drafter designed to accelerate inference for the LFM2.5-VL-3B vision-language model. The drafter captures hidden states from fixed layers of the target model to generate candidate tokens, adding only 280M parameters (8.9% overhead) while maintaining identical dimensionality across modalities.

lab Liquid AI · 11d ago · 39 views

Liquid AI releases DSpark draft model for LFM2.5-VL-3B

Liquid AI has released an experimental DSpark draft model for its vision-language model LFM2.5-VL-3B, enabling faster decoding on edge devices and GPUs without changing output quality.

arxiv arXiv cs.CL · 18d ago · 64 views

DeepSeek releases V4.1-Flash with 890 bytes/token KV cache footprint

DeepSeek has released DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts model designed to address the high costs of long-context computation and large KV caches in agentic workloads.