The author addresses the issue of non-deterministic LLM outputs caused by inference stack batch-invariance, even at temperature zero. To ensure reliable testing, they recommend repeating runs to calculate a pass rate rather than relying on a single result.
- Use repeated runs to determine a pass rate instead of a flaky single result.
- Implement structured outputs for consistent formatting.
- Employ a calibrated judge for evaluation.