FlashAttention-3/4 optimizations do not improve RTX GPU performance

An exploration of applying FlashAttention-3 and FlashAttention-4 optimizations to consumer RTX GPUs reveals that these datacenter-focused improvements offer no performance gains. The author rebuilt attention kernels from scratch to test if FA-3/4 techniques could outperform the existing FA-2 fallback on consumer hardware.

• Faster tensor-core instructions (WGMMA) central to FA-3 are unavailable on RTX GPUs.
• Tensor Memory Accelerator (TMA) is present in sm_120 but does not help because transport is not the bottleneck.
• Warp specialization results in negative performance, increasing latency from 206us to 213us.
• FA-4's simulated exponentiation optimization is irrelevant as RTX 5090 remains tensor-core bound rather than SFU-bound.

The study concludes that FlashAttention-2 represents the performance ceiling for regular attention on consumer silicon, as further gains would require sacrificing accuracy for lower precision.

This analysis focuses on prefill and compute-bound regimes; decoding against large KV caches remains a separate memory-bound challenge where split-KV techniques are more relevant.