Profiling in PyTorch (Part 3): Attention is all you profile

Source: HuggingFace Date: 2026-07-10 URL: https://huggingface.co/blog/torch-attention-profile

Summary

Third installment of HuggingFace’s PyTorch profiling series, using torch.profiler + Perfetto traces to compare attention backends on an A100: Math (reference, 20 kernels, 7.2ms), Efficient/cutlass (1 kernel, ~278µs), Flash/FlashAttention-2 (1 kernel, ~147µs, fastest despite only 13% occupancy), and cuDNN (1 kernel, ~186µs, cost shifted to kernel-planning). A minor fix — swapping masked_fill() for the in-place masked_fill_() — eliminates a hidden Memcpy kernel per forward pass. The methodology throughout: form an explicit hypothesis about the trace before opening it, treat any mismatch as the finding.

Implications

• Inference/quantization advances. Concrete, reproducible numbers on attention backend selection (Flash winning on latency despite low occupancy) are directly useful for anyone tuning inference stacks on consumer/prosumer hardware rather than datacenter A100/H100 fleets.
• Not tied to a specific vendor or model release — a craft/tooling signal rather than a competitive one. Value is methodological (hypothesis-driven trace reading) as much as the specific numbers, which will drift with PyTorch/driver versions.