The llama.cpp project has added support for the Qwen3.8-Flash-Next (qwen4exp) model architecture, implementing its specific components including hyper-connections, gated delta nets, Mixture of Experts, and PLE n-gram hash embeddings.

  • Adds GGUF plumbing for qwen4_exp, including tensors for low-rank hyper-connections and PLE embeddings.
  • Implements the decode graph with hyper-connection residual streams and MoE blocks, validated against vLLM with high top-1 agreement.
  • Introduces host-side hashing for PLE n-gram embeddings using 64-bit integers due to large multiplier values.
  • Adds optional indexer key cache support in llama_memory_hybrid to enable QSA sparse attention for hybrid models.

This addition enables the loading and inference of Qwen3.8-Flash-Next models within llama.cpp, extending its architecture coverage.