PyTorch: How torch.compile achieves up to 10× speedup through kernel fusion

PyTorch’s compiler torch.compile achieves speedups of up to 10× on GPUs without a single change to the model code. The key mechanism behind this speedup is kernel fusion — a technique for merging multiple operations into a single GPU kernel to eliminate costly writes of intermediate values to global memory.

Why does GPU memory slow down model training?

GPU computing does not charge a uniform cost for all operations. Launching each GPU kernel carries a fixed overhead, and every write of intermediate values to global memory is hundreds of times slower than operations in the processor’s own registers. Without a compiler, PyTorch executes each operation (multiply, add, sigmoid) as a separate kernel with its own writes and reads to and from global memory.

Concretely, three consecutive pointwise operations (x * w, + b, .sigmoid()) without fusion require 5 reads and 3 writes — a total of 8 full tensor I/O operations. After kernel fusion: 3 reads + 1 write = 4 operations, which is a 50% reduction in memory traffic.

How does torch.compile automatically fuse kernels?

torch.compile uses the Inductor compiler, which analyzes the computation graph and applies multiple types of fusion:

• Vertical fusion (most common): merges dependent operations so that intermediate values remain in GPU registers. Eliminates all temporary tensors that would otherwise need to go into global memory.
• Pointwise fusion: merges element-wise operations (multiply → add → sigmoid) into a single Triton kernel.
• Reduction fusion: combines reduction operations (e.g., summations) with surrounding operations — critical for batch normalization.
• GEMM + Epilogue fusion: adds bias and activation functions directly at the end of matrix multiplication.
• Horizontal fusion: executes independent operations on the same input (e.g., sin(x) and cos(x)) in parallel within a single kernel.

What happens to kernels in practice?

The Triton kernel generated by fusion keeps all intermediate values in GPU registers and only writes the result at the end:

Load x, Load w, Load b
→ tmp = x * w       (stays in registers)
→ tmp = tmp + b     (stays in registers)
→ tmp = sigmoid(tmp)(stays in registers)
Store result        (the only write to global memory)

Inductor automatically names fused kernels using the convention triton_poi_fused_* (pointwise) and triton_per_fused_* (reduction), which is visible by setting the TORCH_LOGS="output_code" variable.

Results and application

The PyTorch community documents speedups of up to 10× in real models thanks to the combination of fusion, elimination of kernel launch overhead, and reduced memory traffic. All that is needed is a single line of code:

model = torch.compile(model)

The Inductor compiler then independently builds the optimal Triton or CUDA kernel, without the need for manual GPU code writing. This automation makes torch.compile one of the most important tools for production ML training and inference in the PyTorch ecosystem.