AMD: New ATOM Inference Engine for Instinct GPUs Brings OpenAI-Compatible API and MoE Optimizations

AMD has introduced ATOM, an inference engine designed for Instinct GPUs that directly targets production serving of large language models on AMD hardware.

What does ATOM offer and where does it fit in AMD’s stack?

ATOM exposes an OpenAI-compatible API and orchestrates KV cache, scheduling, and parallelism during inference. An inference engine is the layer that receives requests and manages model execution on GPUs. ATOM sits at the top of AMD’s stack: ROCm as the platform, AITER for kernel acceleration, MoRI for RDMA communication between nodes, and ATOM as the serving layer. RDMA (Remote Direct Memory Access) enables direct memory transfers between devices without burdening the CPU.

Which models and parallelism modes does ATOM support?

ATOM supports tensor (TP), data (DP), and expert (EP) parallelism, and is especially optimized for MoE (Mixture of Experts) models. Explicitly listed models include DeepSeek V2 through V4, Mixtral, Qwen3-MoE, Kimi-K2.5, and MiniMax-M2. Expert parallelism distributes individual MoE “experts” across multiple GPUs, which is key to efficiently serving large MoE architectures.

How does ATOM accelerate inference?

ATOM offers quantization in FP8, MXFP4, INT8, and INT4 formats, with automatic detection from the HuggingFace model configuration. Quantization reduces the precision of weights to speed up inference and reduce memory consumption. Additionally, ATOM uses MTP speculative decoding with an EAGLE proposer, prefix cache sharing, and piecewise compilation for faster processing.

How is ATOM used in practice?

ATOM can run standalone or as a plugin for vLLM and SGLang, two popular LLM serving libraries. AMD is also publishing a public benchmark dashboard with nightly performance tracking, offering a transparent signal about progress in serving on Instinct GPUs as an alternative to the NVIDIA stack.