VANILLA SPEC DECODING (2023)
Draft LLM
separate small model
guesses from scratch
→
[draft tokens]
↓
[target verifies]
EAGLE-3
TARGET MODEL
low
mid
high
→
Fuse → Draft Layer
↓
Target LM Head
↓
[draft tokens]
draft reuses target features
TRAINING-TIME TEST
without
training teaches step 1 only. at step 2 the draft consumes its own step-1 output, a distribution it never saw
→ acceptance collapses
with
during training the draft unrolls several steps and consumes its own outputs → training distribution = inference distribution
→ more data, longer acceptance