AMD: Speculative Decoding on MI300X Speeds Up Inference by 4.3x
On the ROCm blog, AMD presented an implementation of speculative decoding on Instinct MI300X GPUs. The technique, in which a smaller draft model predicts tokens and a larger one verifies them, achieved up to 4.3 times higher throughput compared to classic autoregressive generation.
This article was generated using artificial intelligence from primary sources.
On its ROCm blog, AMD published an implementation of speculative decoding on Instinct MI300X accelerators. Speculative decoding is a technique for accelerating the inference of large language models in which a smaller draft model predicts a sequence of tokens, and a larger target model then verifies them in parallel. This avoids slow token-by-token generation without loss of output quality.
How was the system set up?
AMD used Llama-3.2-1B-Instruct as the draft model and Llama-3.1-70B-Instruct as the target. The distribution spanned five MI300X GPUs: four to shard the large target model and one dedicated to the draft model. The software layer consisted of ROCm 7.2, PyTorch 2.9.1 and FlashInfer, all within a Docker environment with automated setup.
How big is the speedup?
Measurements on the alpaca, c4, ultrafeedback and humaneval benchmarks showed a clear gain. Autoregressive generation delivered 52.32 tokens per second, standard speculative decoding 138.06, and the advanced variant as much as 225.86 tokens per second. That means generation 4.32 times faster than the baseline approach and 1.64 times faster than conventional speculative decoding.
Why does it matter?
Higher throughput directly reduces the cost of serving models and the latency for users. The demonstration shows that AMD hardware, paired with a mature ROCm software stack, can deliver acceleration techniques that were until now tied to competing platforms.
Frequently Asked Questions
- What is speculative decoding?
- It is a technique for accelerating LLM inference in which a smaller draft model predicts tokens in advance and a larger target model verifies them in parallel, reducing the number of slow steps.
- How much speedup was achieved on MI300X?
- AMD reports a throughput of 225.86 tokens per second, which is 4.32 times faster than autoregressive generation and 1.64 times faster than standard speculative decoding.
Sources
📬 AI news in your inbox
A daily digest built your way — pick topics, sources and cadence. One-click unsubscribe.
Related news
AMD: hipVS — GPU vector search for Instinct, cuVS-compatible, with agentic RAG demo
NVIDIA: performance per watt as the key metric — Blackwell GB300 up to 25× more efficient than Hopper
AMD: GEAK Agent Automatically Optimized DeepSeek-V4 MLA Kernel on MI355 with Up to 9× Speedup