AMD Quark delivers MXFP4 quantization for FLUX.1-dev on Instinct MI350 with 1.92x speedup

AMD has extended MXFP4 quantization support to the image generation domain. The enabling tool is Quark, AMD’s quantization framework, integrated with the xDiT framework for parallel execution of diffusion models. Measurements conducted on the AMD Instinct MI350 GPU (gfx950 architecture) demonstrate significant speedups without measurable quality loss in generated images.

MXFP4 quantization for diffusion models on AMD Instinct

Image generation with diffusion models falls into memory-intensive workloads — a large number of parameters pass through successive denoise steps, and in real-time inference or batch processing every millisecond of latency directly affects service economics. FP4 quantization offers a dramatic reduction in memory footprint, but without structured scaling the numerical precision loss is too large and manifests as artifacts in generated images.

Quark Release 0.12 implements MXFP4 with a per-layer scheme using native AITER GEMM kernels that operate directly in MXFP4 format. There is no dequantization phase between layers — the kernel receives MXFP4 input and produces output without conversion back to FP16 or BF16, eliminating the memory bandwidth overhead that would otherwise partially negate the advantage of lower precision.

Test environment: PyTorch 2.9.1 with ROCm backend, AITER 0.1.10, Quark Release 0.12, Docker image rocm/pytorch-xdit:v26.5.

What is MXFP4 and why does it eliminate dequantization overhead?

FP4 representation is stored in 4 bits per element, which compared to BF16 (16 bits) gives a 4x smaller model memory footprint. The problem with standard FP4 is its narrow dynamic range: without information about data magnitude, fine numerical differences between activations disappear in rounding.

MXFP4 (Microscaling FP4) addresses this with block-level scaling: every block of 32 elements shares a common scaling factor stored in higher precision. This is a standardized approach defined within the broader Microscaling (MX) specification, enabling FP4 compression with a dynamic range comparable to FP8 formats. AMD’s AITER GEMM kernels implement this format natively on the MI350 architecture, meaning the GPU operates directly on MXFP4 tensors without software conversions in the hot path.

Measured speedup on one and two MI350 GPUs

The reference model is FLUX.1-dev (black-forest-labs), generated at a resolution of 1024x768 pixels, 20 inference steps, guidance scale 3.5. Quality validation was performed on 100 samples from the COCO 2017 dataset using openai/clip-vit-base-patch16.

On a single GPU with torch.compile, the MXFP4 ASM configuration achieves a 1.92x speedup over the BF16 eager baseline — latency drops from 2.054 s/image to 1.069 s/image. Compared to the compiled BF16 variant, the speedup is 1.41x. In eager mode without compilation, MXFP4 delivers 1.15x faster generation (1.779 s/image vs. 2.054 s/image).

On a 2-GPU configuration using Ulysses parallelism, the compiled MXFP4 variant reaches an optimal 0.855 s/image at a batch size of 16, representing 1.23x better throughput compared to the BF16 equivalent (1.052 s/image). At batch size 1 the speedup is slightly lower, 1.21x, due to lower parallel utilization.

Generated image quality remains unchanged

The BF16 eager baseline achieves a CLIP score of 30.98. The compiled MXFP4 ASM variant achieves 31.84 — numerically higher than the baseline, within the statistical variability of the measurement procedure on 100 samples. All four tested configurations (BF16 eager, BF16 compiled, MXFP4 eager, MXFP4 compiled) retain CLIP scores within a range of ±0.5 points.

This is practically relevant: CLIP scores measure the semantic alignment between image and description, which is perceptually closer to the user’s quality assessment than metrics like PSNR that measure pixel similarity. Confirming that MXFP4 does not degrade CLIP score means that quantization introduces no semantic artifacts that would be visible to the end user.

Integration with the xDiT framework leaves room for extension to 4 or 8 GPUs without changing the quantization configuration — AMD Quark 0.12 supports both FP8 and MXFP4 formats on the same MI350 infrastructure. For operators running image generation services, the 1.92x speedup translates directly into doubling capacity per GPU or a proportional reduction in infrastructure costs at the same load. The accessibility of the approach — standard Docker image, Diffusers integration, xDiT Ulysses parallelism without model modifications — makes MXFP4 quantization a viable option for production deployments without requiring specialized engineering knowledge of AMD hardware.