The llama.cpp project released version b9908, which introduces a hard limit for the server's prompt cache RAM usage. Previously, the cache could temporarily exceed specified limits by retaining entries during save operations.
- The --cache-ram flag now strictly prevents saving state if it exceeds the RAM limit.
- Old cache entries are evicted as necessary to make room for new ones.
- Token-limit cleanup can now evict the last remaining cache entry instead of preserving one.
- macOS Apple Silicon KleidiAI support is disabled in this release.
This change ensures that the prompt cache respects memory constraints more reliably, preventing temporary overflows.