Prefix Sliding cuts long-rollout inference time by up to 3×

• Prefix Sliding keeps the task and tool prefix plus a recent reasoning window in attention, while evicting older thought tokens, as Muennighoff's paper announcement describes.
• The authors report up to 3x faster inference without retraining, with Muennighoff's throughput chart showing accuracy against thinking time on the same Qwen3 model.
• Reinforcement learning can spend a full-attention memory budget on much longer rollouts, which Muennighoff's RL post says can produce higher reward.
• Its defining choice is to retain the entire task-and-tools prefix as global context, a distinction Muennighoff's clarification draws from related streaming and sliding-window schemes.

The paper makes a hard claim about reasoning traces: most of their middle can expire. The runnable Qwen3 example sets a 4,096-token window for a 32,000-token generation, and the team published evaluation artifacts alongside it.

Muennighoff in his explanation of token retention defines the preserved prefix as the task, system instruction, and tool definitions. A second region holds only the latest reasoning tokens, so his implementation description caps generation cost at the prefix length plus the window length.

The attention pattern has three parts:

• Persistent prefix: task framing, instructions, and available tools.
• Sliding window: the recent tokens the model is actively using to reason.
• Evicted middle: older intermediate reasoning tokens outside both regions.

The paper's Figure 1 plots accuracy against average thinking time on AIME25, GPQA Diamond, and MATH500. It attributes the speed advantage to generating more tokens in the same time, rather than making each generated token better.

The no-training result matters because the intervention is an attention-and-memory policy applied to an existing generative model, rather than a new reasoning checkpoint.

During reinforcement learning, Prefix Sliding keeps the per-token memory budget bounded while a rollout grows. The paper reports reasoning traces beyond 100,000 tokens under this training path.

That result remains preliminary: Muennighoff in a later reply said the RL runs were decent but needed substantially more scaling and more comparisons with related approaches.

Muennighoff in comparing alternatives lists the trade-offs that the fixed prefix is meant to avoid:

• Vanilla sliding window: eventually drops instructions and tool definitions. Adding intermittent full attention restores the memory failure.
• Compaction or summaries: add a summarization pass, process summary tokens twice, and lose some context.
• Last-K retention: also loses context and processes tokens twice.

Muennighoff in the project's acknowledgements credits support from the Laude Institute, NVIDIA's Academic Grant Program, and Prime Intellect. The released code announcement links to an implementation that changes the vLLM, FlashAttention, and RL stack, rather than merely exposing a standard runtime setting.

The repository setup pins custom vLLM and FlashAttention branches and estimates the build at about 10 hours, mostly during editable installation. Its published inference snippet uses Qwen/Qwen3-1.7B, sets SWF=4096, and enables the sliding window through model configuration.