An ablation study in OLMo-core compared integrating Engram onto a standard Transformer versus a 3 GDN-layer + 1 attention-layer hybrid backbone. The analysis revealed that fixing signed overflow and padded token ID indexing failures substantially narrowed the performance gap between the two architectures.
- Transformer + Engram achieved slightly lower cross-entropy loss and perplexity than the hybrid.
- The hybrid processed more tokens per second but suffered from choppier gradients and lower MFU due to compiler constraints on model dimension.
- Fixing indexing failures and moving static tensors into registered buffers lowered loss for both backbones.
- GPU memory remained flat for both runs despite elevated CUDA allocation retries in the hybrid.
The results indicate that initial performance differences were partly execution artifacts rather than inherent architectural advantages, highlighting the importance of engineering constraints like throughput and compiler compatibility in research.