FlashAttention-4 addresses the performance limitations of Blackwell's 4-bit floating-point (FP4) tensor cores by introducing new paths for noncausal inference and causal training. The method, called Direct-P, maps scores directly to FP4 probabilities to bypass softmax conversion overheads that typically dominate once matrix products shrink.
- Direct-P achieves up to 2.13x the bfloat16 (BF16) forward throughput on an NVIDIA GB200 for noncausal inference.
- The causal path passes forward quantization directly into backward, reconstructing probabilities from saved quantized queries and keys using 8-bit floating-point (FP8) gradient operands.
- This causal approach accelerates a complete single-GPU 8-billion-parameter update by up to 1.14x.
- Matched distributed training retains FP8 probabilities and values, as every tested MXFP4 probability/value training trajectory diverges.
The authors consider this important because it enables efficient hardware-aware FP4 attention on Blackwell architectures, overcoming the dependencies that prevent automatic speedups from FP4 tensor cores.