A study using Olmo 3 and Olmo Hybrid open weights finds hybrid models outperform transformers on open-class content words and opening delimiters. The gains are less consistent for closed-class function words and closing delimiters, with hybrids excelling in semantic state tasks like pronoun memory and entity tracking, while transformers perform better on bracket-matching tasks. These results suggest recurrent layers enhance state-aware predictions, while attention supports n-gram and syntactic pattern recognition.
Token-Level Comparison of Transformers and Hybrid Models
from English