The Hindcast framework evaluates LLM forecasters by replaying resolved Polymarket prediction markets against a frozen snapshot of public Reddit, grading models as if they existed at a specific past date. This approach prevents two common channels of answer leakage: retrieval of post-event reports and training data contamination from events that occurred after the model's release.
- Hindcast scores forecasts against both the actual outcome and the market price at the cutoff date, which serves as a human baseline derived from the same historical information.
- The evaluation uses per-market cutoffs to ensure the test remains valid as new markets are added and models improve.
- Retrieval assistance only improves performance when Reddit discussions preceded the event; in cases where the archive contained only speculation, retrieval degrades accuracy.
By closing these leaks, Hindcast ensures that evaluations measure genuine foresight rather than recall of resolved information.