A bug in llama-server causes the first query after a process restart to discard rehydrated KV caches and perform a full prompt re-prefill, rendering the save/restore feature useless. The root cause is that checkpoint metadata was not persisted to disk, preventing the server from rolling back to valid cache positions.

  • The fix persists checkpoint metadata to a versioned sidecar file during slot save and reloads it during restore.
  • This allows the server to reuse cached states for delta queries instead of recomputing them.
  • Benchmarks show a ~720x speedup for delta prefill and ~167x end-to-end resume compared to full re-prefill.
  • The patch is available as PR #206 in the llama-cpp-turboquant fork.

This change restores the intended functionality of long-context work on budget hardware by ensuring cached states are actually utilized after a restart.