Researchers propose a protocol to mitigate p-hacking in large language model (LLM) research by preregistering experiments and running confirmatory analyses on the first eligible LLM released after the commitment. This approach prevents researchers from tuning prompts or parameters to achieve desired results, as the target model does not exist at the time of preregistration.
- The protocol requires finalizing procedures on current models, preregistering the analysis plan with a set of eligible future models, and executing the analysis on the first eligible model released afterward.
- Evaluation across 20 models from four providers showed that the protocol blocked successful p-hack transfer in 73.9% and 72.7% of cases for two tasks with known true values.
- An independent test following the protocol confirmed its effectiveness, with hacking failing to carry over in 6 out of 7 configurations on the first eligible model released after preregistration.
This method helps ensure the integrity of LLM-based research by making it difficult to manipulate results through iterative tuning, as configurations that hack one model often do not transfer to the next.