Energy-Based Transformers as Predictors of Reading Difficulty

This study introduces energy-based transformers as a novel measure for predicting human reading difficulty, establishing a formal link between transformer models and associative memory literature like Hopfield networks.

• Evaluated across Natural Stories, UCL eye-tracking, and UCL self-paced reading corpora, the energy measure provides significant fit beyond surprisal in all three datasets.
• In a controlled experiment on relative clause processing, energy at a single layer captures the well-known object/subject asymmetry.
• Evidence suggests the energy measure subsumes effects attributable to both attention entropy and surprisal, potentially serving as a unified predictor.

The authors consider this important because it offers a single unified predictor for reading difficulty where multiple complementary measures were previously required.