The llama.cpp project introduces support for the Granite-Switch architecture, a dense model variant featuring N embedded LoRA adapters that are selected per-token via control tokens. This implementation replaces a previous proof-of-concept global sticky index with an in-graph causal "router" attention mechanism to resolve adapter indices.
- The new router uses a single-head causal attention layer to recover the adapter index within the graph, fixing concurrency issues where multiple sequences previously leaked adapter states into one another.
- By storing router keys and values in the per-sequence KV cache, concurrent requests are now isolated without requiring global state management.
- The architecture supports mid-sequence switching, allowing different adapters to be activated for specific tasks like answerability or query rewriting within a single sequence.
- A known limitation remains that once an adapter fires within a sequence, it stays active until the sequence ends, as the router gain is flat and lacks recency bias.
This change enables efficient multi-task inference on CPU by dynamically switching between specialized adapters without reloading models, while maintaining isolation for concurrent batched requests.