A user is requesting assistance with optimizing the KV cache configuration in llama-server to mitigate long prefill times when running large models like Qwen 3.5 122B on a Strix Halo system.
- The user has configured --cache-ram 16384, --ctx-checkpoints 72, and --checkpoint-min-step 4096 to manage VRAM and context windows.
- A full cache miss at 100k context results in approximately 10-20 minutes of prefill time, which is considered extremely problematic.
- Checkpoints are sometimes missed after user prompts, particularly with agentic workflows like Crush coding agent or Codex.
- Older checkpoints appear to disappear over time, leaving only a few active context checkpoints and causing reprocessing during larger misses.
- The user is considering K/V quantization (e.g., q8) to potentially double the number of available checkpoints but questions its impact on quality for models already in q4.
The user hopes for community input on fine-tuning cache settings and resolving checkpoint miss issues to improve performance during fast agentic iterations.