A bug in llama-server causes the first query after a process restart to discard rehydrated KV caches and perform a full prompt re-prefill, rendering the save/restore feature useless. The root cause is that checkpoint metadata was not persisted to disk, preventing the server from rolling back to valid cache positions.
- The fix persists checkpoint metadata to a versioned sidecar file during slot save and reloads it during restore.
- This allows the server to reuse cached states for delta queries instead of recomputing them.
- Benchmarks show a ~720x speedup for delta prefill and ~167x end-to-end resume compared to full re-prefill.
- The patch is available as PR #206 in the llama-cpp-turboquant fork.
This change restores the intended functionality of long-context work on budget hardware by ensuring cached states are actually utilized after a restart.