Profiling in PyTorch (Part 2): From nn.Linear to a Fused MLP

Source: HuggingFace Date: 2026-06-11 URL: https://huggingface.co/blog/torch-mlp-fusion

Summary

HuggingFace’s second PyTorch profiling tutorial walks through kernel-level analysis of an MLP with GeGLU activation on an A100, comparing three execution modes: eager (nn.Linear), torch.compile, and hand-tuned Liger kernels. torch.compile collapses GeLU, mul, and reshape into a single kernel keeping intermediates in registers; Liger provides a shape-agnostic alternative avoiding recompilation. Measured latency: compiled at 89.4 µs (shape-specific), Liger at 92.8 µs (shape-agnostic).

Implications

• Feeds the floor/substrate fight thread: the gap between torch.compile and hand-tuned kernels is now ~4%, meaning the compiler is nearly capturing manual optimization — that compression raises the question of when the substrate just absorbs what is now specialist knowledge.
• Practical signal for ML infrastructure teams: Liger is the pragmatic default for production workloads with variable batch shapes; torch.compile wins only if shapes are stable.
• Relevant to dev-tooling: profiler-literacy content from HuggingFace is part of a broader push to make GPU optimization accessible to practitioners who don’t read PTX.