A New Pruning Method Lets Language Models Skip Work Token by Token
A new paper introduces WIDE, a token-level pruning method that cuts language-model computation while preserving more accuracy than prior dynamic approaches.
What the paper proposes
Trimming unneeded computation from large language models is one of the main ways teams try to make them cheaper to run. Older, static pruning methods remove the same parts of the model for every input, which is easy to speed up on real hardware but can hurt accuracy badly once the cuts get aggressive, because the pruning ignores what each specific input actually needs. Newer, dynamic methods try to adapt to each input instead, but according to the paper they have mostly stayed limited to coarse, layer-level choices, and turning that adaptivity into real speed on hardware has remained difficult.
WIDE, described by its authors in the paper on arXiv as the first end-to-end differentiable token-level dynamic width pruning framework built for both prefill and decode, tries to close that gap. Instead of deciding once per layer, WIDE lets each individual token choose its own attention-head groups and feed-forward channel groups to use, pushing the pruning decision down to what the paper calls neuron-block-level granularity. A two-stage training pipeline is used to teach the model these token-by-token sparse patterns. To make the fine-grained decisions actually fast in practice, the authors also built what they call a pruning-kernel co-design: it splits the work of skipping computation into mask reordering, block-level skipping that works across hardware, and finer intra-block skipping tuned to specific hardware.
How the reported results compare
According to the paper, at 50% sparsity WIDE delivers a 55.1% performance boost compared with the strongest existing dynamic depth pruning method, measured under calibration-only settings. Under prefill and decoding workloads, the authors report kernel-level speedups of up to 1.98x for prefill and 4.95x for decoding, which they describe as close to the theoretical maximum for this kind of sparsity. They also report 1.68x and 1.55x end-to-end acceleration from the same pruning-kernel design.
Key facts
- WIDE is presented as the first end-to-end differentiable token-level dynamic width pruning framework covering both prefill and decode.
- Each token can pick its own attention-head groups and feed-forward channel groups, moving pruning decisions to neuron-block-level granularity instead of stopping at the layer level.
- At 50% sparsity, the paper reports a 55.1% performance boost over the strongest existing dynamic depth pruning baseline under calibration-only settings.
- Kernel-level speedups reach up to 1.98x for prefill and 4.95x for decoding, alongside 1.68x and 1.55x end-to-end acceleration.
- The authors have released their training and kernel code on GitHub under the EIT-NLP/LLM-Pruning repository.
Why it matters
Running a large language model in production spends most of its cost on inference, not training, so any method that trims that cost without giving up much accuracy is useful to teams operating these systems at scale. This matters because static pruning is easy to deploy but can be too blunt, while earlier dynamic methods were adaptive but hard to actually speed up on real chips. WIDE's pitch is a middle path: token-level decisions for quality, paired with a kernel design meant to turn those decisions into real, measured speedups rather than only theoretical ones.
A preprint, not a settled result
This work is a preprint posted to arXiv on 30 July 2026 and has not been through peer review, so the reported numbers come from the authors' own benchmarks rather than independent verification. The comparison to "the state-of-the-art" dynamic depth pruning method is also scoped to calibration-only settings, a detail worth keeping in mind before treating the 55.1% figure as a general result across every setup.
Check your understanding
What does WIDE let each individual token dynamically select, according to the paper?
What kernel-level speedup does the paper report for decoding at 50% sparsity?