BCMT: Blockwise Causal Memory Transformer for efficient long-context modeling

A researcher introduces BCMT, a new architecture for long-context language modeling that replaces dense global self-attention with blockwise operations. The model combines local causal attention with adaptive contextual summaries and causal memory propagation to efficiently capture long-range dependencies.

• Computational complexity is reduced from O(T²) to O(TL) for a fixed block size.
• The implementation includes the full architecture, an H-only ablation, and a standard Transformer baseline.
• Experiments on WikiText-103 show validation perplexities close to dense Transformers with higher throughput and lower GPU memory usage.

The project is open source, providing PyTorch code and training scripts for community review.