Researchers propose an evaluation framework that uses regularized regression on item text embeddings to predict psychometric properties, addressing the cold start problem in item calibration. The study applies this method to a mathematics item bank (EEDI) and a medical licensure benchmark (BEA 2024), introducing reliability and design ceilings as performance upper bounds.
- Item difficulty is highly predictable from text, achieving a repeated cross-validated R squared of 0.53, which represents about 57% of its reliability ceiling.
- Discrimination and pseudo guessing parameters appear less predictable, but this stems from low target reliability rather than weak text signal strength.
- Text uniformly recovers 57 to 63% of the reliable variance across difficulty targets, while the 3PL pseudo guessing parameter has a reliability ceiling near zero.
- On the BEA benchmark, embedding-based regression matches leaderboard RMSE despite explaining almost no variance, highlighting the need for scale-free metrics.
- A single train and test split can inflate apparent accuracy by 0.1 to 0.15 in R squared, underscoring the necessity of repeated cross-validation.
The authors consider this important because it demonstrates that text embeddings can effectively predict item difficulty and provides a rigorous framework for benchmarking calibration support applications.