The developer of the local AI harness OpenFox has implemented a feature called "speculative cache warming" that processes the system prompt and tools array while the user types their input. This optimization allows the model to begin processing static context data before the actual prompt is submitted, effectively reducing the latency experienced when starting a new session.
- The system prompt (containing AGENTS.md and preferences) and tools array are processed in the background during typing.
- When the user sends the prompt, only the remaining prompt text needs to be processed by the LLM.
- At a processing speed of 500 tokens per second, this method saves approximately 10 to 20 seconds of wait time.
This change makes the interaction with local LLMs more responsive and reduces the idle time typically associated with initializing a new session.