Faster Text Generation with Self-Speculative Decoding

Source: HuggingFace Date: 2024-11-20 URL: https://huggingface.co/blog/layerskip

Summary

Library update and research summary: Self-speculative decoding (LayerSkip) integrated into Transformers via assistant_early_exit parameter. Uses early layers to draft tokens, later layers to verify — no separate draft model needed. Speedups on LayerSkip-trained models: Llama2 70B layer 10 exit 2.06x, Llama3 8B layer 4 exit 1.83x, Llama3.2 1B layer 4 exit 1.80x. Key caveat: only works on models trained with the LayerSkip recipe (layer dropout + early exit loss); pre-trained models without this training won’t benefit.

Implications

Transformers library trajectory. Self-speculative decoding being a single parameter (assistant_early_exit) in generate() is the right API design — it’s zero-friction for users of compatible models. The 2x+ speedup at 70B scale without a separate draft model is the practical advantage over traditional speculative decoding (which requires maintaining and serving two models simultaneously).

Open-weights ecosystem health. The hard requirement — models must be trained with the LayerSkip recipe — means the speedup is only available for models specifically designed for it. This creates an incentive for open-weights model authors to include LayerSkip training in their pipelines, but the installed base of pre-trained models is unaffected. Watch for future releases of popular model families with LayerSkip variants to see if this becomes a standard training recipe inclusion.