AI_SLANG_ENTRY
Speculative Decoding Meaning
The LLM speed trick where a smaller draft model guesses upcoming tokens and a larger model checks them, like autocomplete with a quality inspector.
What does Speculative Decoding mean?
The LLM speed trick where a smaller draft model guesses upcoming tokens and a larger model checks them, like autocomplete with a quality inspector.
Speculative decoding speeds up generation by letting a cheap model draft several tokens while the main model verifies them in batches. The goal is lower latency without accepting lower-quality guesses.
Origin and usage
Moved from inference research into builder vocabulary as teams looked for ways to reduce latency in large language model apps without changing the final output distribution.
Source type: paper. Last checked: 2026-07-11.
Technical LLM inference term covered in survey and systems papers; slang value comes from builders using it as shorthand for draft-and-verify generation speedups.
Examples
- Speculative decoding made the chatbot feel faster without switching to a smaller model.
- Latency went down after we tuned the draft model for speculative decoding.