AI Models & Releases
Why AI Is Getting Faster Without Getting Smarter: Speculative Decoding Explained
AI models are not getting smarter every time they respond faster. A technique called speculative decoding lets large language models produce the same outputs in a fraction of the time, and understanding how it works tells you something important about where AI is headed.
Key takeaways
- Speculative decoding uses a small, fast draft model to guess tokens that a large target model then verifies in parallel, delivering 2 to 3 or more times faster responses with mathematically identical outputs.
- The speedup exploits a hardware reality: GPU compute cores sit idle during token generation because the process is memory-bandwidth bound, leaving spare capacity for parallel verification.
- The technique has moved from research papers into production systems at major AI companies and is now a standard feature in open-source inference frameworks.
- Speedup is not universal. Tasks with predictable outputs like code and structured data benefit most; highly creative tasks with low draft acceptance rates benefit less.
- For anyone reasoning about AI progress, speculative decoding illustrates a key principle: speed improvements and capability improvements travel on separate tracks, and distinguishing between them is essential for forming accurate predictions about what AI announcements actually mean.
You have probably noticed that AI assistants have gotten noticeably snappier over the past couple of years, even when the underlying models have not changed. The responses feel quicker. The wait between pressing send and seeing the first word appear has shrunk. For most users, the natural assumption is that the model itself improved. Often, that is not what happened at all. The speedup frequently comes from a different kind of advance, one that lives in the infrastructure layer rather than the intelligence layer. The technique at the center of this story is called speculative decoding, and it is one of the most practically consequential ideas to come out of AI research in recent years.
The Fundamental Bottleneck: Why Large Language Models Are Slow by Default
To understand why speculative decoding matters, you first need to understand the unusual way large language models generate text. Every time a model produces a response, it does so one token at a time. A token is roughly a word or a fragment of a word. To generate a sentence that is twelve tokens long, the model must run twelve separate forward passes through its full network of billions of parameters.
This is called autoregressive decoding, and it creates a significant hardware problem. During each forward pass, the GPU must read all of the model's weights from memory. For a large model, that can mean reading on the order of a terabyte of data for every single word it produces. The overall speed at which the model generates text is therefore limited not by the raw calculation power of the GPU, but by how quickly data can be fed to it. Researchers call this being memory-bandwidth bound rather than compute-bound.
The practical consequence is that modern GPUs sit mostly idle in terms of raw computation during inference. Their processing cores are fast enough to do far more math per second than the memory system can keep them supplied with data. Speculative decoding was designed specifically to exploit that gap.
The Core Idea: A Fast Guesser and a Careful Checker
Speculative decoding was formalized in research published by Google and DeepMind in 2023, though earlier variations of the idea existed. The core insight is elegant: instead of having the large model generate every token by itself, use a smaller, faster model to guess what the large model would say, then let the large model verify multiple guesses at once.
Here is how it works in practice. A lightweight draft model, sometimes called a speculator, generates a short sequence of candidate tokens very quickly. The large target model then takes all of those candidate tokens and evaluates them in a single forward pass. It accepts the ones that match what it would have produced on its own and rejects the first one that does not fit. When a token is rejected, the target model replaces it with a corrected sample, and the process starts over.
The critical point is step two. Because all of the draft tokens are already known at verification time, they are only being checked rather than generated from scratch, and the target model can evaluate them in parallel within a single forward pass. This means multiple tokens can be accepted for roughly the cost of one or two large model forward passes, rather than requiring a separate forward pass for each token.
The algorithm guarantees something remarkable: the output distribution of the combined system is mathematically identical to what the target model would have produced on its own. The responses are not approximations or slightly degraded versions of what the big model would say. They are exactly what the big model would say, just delivered faster.
How Much Faster Are We Actually Talking?
Early results from Google and DeepMind's concurrent 2023 papers reported speedups in the range of roughly 2 to 2.5 times over standard autoregressive generation. Intel and the Weizmann Institute later reported their own speculative decoding research achieving up to 2.8 times faster inference without loss of output quality.
The speedup is not uniform. It depends heavily on how well the draft model can predict what the target model would say. If the draft model has a high acceptance rate, meaning many of its guesses pass the target model's verification, many tokens get accepted in each cycle and the speedup is substantial. If the draft model guesses poorly on a given task, the target model rejects most tokens, and the overhead of running two models can actually slow things down compared to the baseline.
This acceptance rate is the key variable. The draft model must be significantly smaller than the target model, typically by a factor of ten to fifty times, to achieve net acceleration. The draft model must also share a compatible vocabulary and general language understanding with the target. Mismatched model families produce poor acceptance rates and may yield no benefit.
The EAGLE Family: Making Draft Models Much Smarter
The original speculative decoding setup pairs a standalone small model with a large target model. Researchers quickly identified the main limitation: a generic small model trained independently will not predict a specific large model's outputs as accurately as a model trained to mimic that specific target.
A series of methods called EAGLE addressed this directly. Rather than using a separate independent draft model, EAGLE trains a lightweight draft head that conditions on the target model's own internal representations, specifically the output features of the target model's last transformer layer before the language modeling head. Because the draft sees what the target was processing at its deepest layer, it can make predictions that are closely aligned with the target's distribution.
EAGLE's first version achieved 2.7 to 3.5 times latency speedup on large models. The second version, EAGLE-2, added dynamic draft trees: instead of proposing a simple linear sequence of tokens, the draft explores branching token paths and retains only the most confident branches. The target model then verifies the entire tree in a single pass. Research on EAGLE-3 reported speedups in the range of approximately 3 to 6.5 times compared to standard autoregressive generation, with improvements in the average number of accepted tokens per verification step.
NVIDIA's technical documentation describes EAGLE-3 as a technique that uses a lightweight autoregressive prediction head attached to the target model's internal layers to generate candidate tokens, eliminating the need for a separate standalone draft model entirely.
From Research Paper to Production Reality
Speculative decoding has moved well beyond academic benchmarks. Google confirmed that speculative decoding enables AI Overviews in Google Search to produce results faster while maintaining the same quality of responses, and that it remains a significant part of their ongoing optimizations for that product. Google published their foundational paper in 2022 and have continued to build on the technique since.
The technique is now supported as a standard feature in the major open-source inference frameworks. Popular serving systems like vLLM and SGLang include built-in support for speculative decoding, making it accessible to any team running their own models rather than only to organizations with the resources to build custom infrastructure from scratch. Major hardware vendors including NVIDIA and Intel have published technical documentation and code for deploying it in production.
The broader adoption pattern is telling. What started as a clever academic paper has become a standard optimization layer in AI infrastructure because it delivers real, measurable latency improvements with no change to model outputs and no need to retrain or replace the underlying model.
When Speculative Decoding Works Best (and When It Does Not)
Speculative decoding is not a universal fix. The technique delivers the most benefit when the text being generated is somewhat predictable: tasks involving code, structured data like JSON or SQL, summarization, and document-grounded question answering tend to produce high acceptance rates because the small draft model can anticipate the large model's choices reliably.
Creative writing, open-ended brainstorming, and tasks requiring unusual or highly specific vocabulary are harder for draft models to predict accurately. In those scenarios, the acceptance rate drops, fewer tokens are accepted per cycle, and the speedup diminishes. Some inference configurations see the speed actually slow down in these cases.
Batch size also matters. Speculative decoding works best when serving single requests or small batches, because the GPU has spare computational resources available for parallel verification. At very large batch sizes, the GPU becomes compute-bound rather than memory-bound, and the extra work of running the draft model and verifying its outputs competes with other requests for the same compute budget.
The Probability Lens: What This Means for Thinking About AI Progress
Here is where thinking in probabilities becomes directly useful. When you see a headline announcing that a well-known AI assistant just got significantly faster, the naive prediction is that someone made a smarter model. The technically informed prediction is more nuanced: the smartness of the model and the speed of its delivery are increasingly separate engineering problems, and advances in either one can happen independently of the other.
Speculative decoding is a clean illustration of this separation. The intelligence, reasoning quality, and accuracy of the large target model are completely unchanged by the technique. The distribution of outputs is mathematically guaranteed to be identical. What changes is the infrastructure. Speed improvements can now arrive on a different cadence from capability improvements, which means the question "did the AI get better or just faster?" is a genuinely important one to ask whenever a product announces a performance upgrade.
For anyone trying to form accurate predictions about the AI landscape, this matters. Inference optimization is a distinct research front from pretraining scaling, fine-tuning, and reasoning techniques. Progress on one front does not predict progress on another. A company can deliver a dramatically faster AI experience without having advanced the underlying model at all, and vice versa. Keeping these categories distinct is one of the simplest and most useful habits for getting smart about AI.
The Broader Inference Optimization Landscape
Speculative decoding exists alongside a family of other inference optimization techniques, including quantization (reducing the numerical precision of model weights), dynamic batching (grouping multiple user requests together to share computation), and custom GPU kernels that squeeze more performance out of specific hardware. These techniques can often be combined, and researchers have demonstrated that pairing speculative decoding with quantization, for example, can compound the benefits.
The common thread across all of these approaches is the same insight that makes speculative decoding work: large language model inference is memory-bandwidth bound during token generation, which means there is almost always idle computational capacity that better algorithms can exploit. The research community is actively mapping the theoretical limits of how much speedup speculative decoding can achieve before it hits a fundamental ceiling, and the gap between current methods and that ceiling suggests there is still meaningful room to improve.
For users and practitioners, the practical upshot is that AI responses will continue to get faster even as the underlying models stay constant, and that speed improvements carry a different kind of value than capability improvements. Faster responses reduce latency in user-facing products, lower the cost of serving each query, and make it economically viable to deploy larger and more capable models that would otherwise be too slow for real-time use. In that sense, speculative decoding is not just an engineering footnote. It is one of the mechanisms that determines which AI capabilities are actually deployable at scale.
Sources
- Looking Back at Speculative Decoding (Google Research Blog)
- An Introduction to Speculative Decoding for Reducing Latency in AI Inference (NVIDIA Technical Blog)
- Intel and Weizmann Institute Speed AI with Speculative Decoding Advance (Intel Newsroom)
- A Hitchhiker's Guide to Speculative Decoding (PyTorch Blog)
- How Speculative Decoding Delivers Faster LLM Inference (Red Hat Developer)
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models (arXiv)
- EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (OpenReview / ICML 2024)
- EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees (arXiv)
- Speculative Decoding Explained (Data Processing Club)
- Speculative Decoding Speed-of-Light: Optimal Lower Bounds via Branching Random Walks (arXiv)
- Why LLM Inference Is Memory-Bound (Not Compute-Bound) (Medium)
- Speculative Decoding in Practice: How EAGLE3 Makes LLMs Faster Without Changing Their Outputs (Hugging Face Blog)
Frequently asked questions
