Exploration-Distillation separates RL exploration from student training

• ExpDis separates aggressive exploration from student training: the announcement introduces a disposable explorer whose successful traces become training data.
• Reasoning gains persist through pass@64: in the authors' experiments, ExpDis beats DAPO at matched wall-clock time.
• General capabilities were preserved on the evaluated suite, according to the capability comparison, while adding novelty rewards directly to DAPO degraded them.

Throwaway explorers are the neat trick: the paper lets them deteriorate because they are never deployed. The code includes a CPU-only end-to-end test, while the checkpoint release includes a DAPO control trained for four times as many steps.

Saif Punjwani and Micah Goldblum, Columbia University researchers, initialize separate explorer and student instances from the same checkpoint. Their released pipeline has three stages:

1. Explorer RL: Train with DAPO, combining correctness rewards with a weighted novelty bonus.
2. Rejection sampling: Accept only correct final \boxed{} answers that terminate within 32,768 tokens and contain no repetition loops. Keep the shortest accepted trace per problem, capped at 500 traces per round.
3. Student training: Apply supervised fine-tuning (SFT) to the accepted traces, then run DAPO without the novelty bonus.

A math reward can remain high while knowledge outside its scope deteriorates, as the thread explains. The two policies communicate through filtered trajectories, without transferring the explorer's updated weights.

Goldblum also floated training on the collected traces with RL and an importance-sampling correction in a follow-up.

The explorers use Random Network Distillation (RND). The paper's formulation has four ingredients:

• Random targets: Frozen, randomly initialized MLPs attached to selected intermediate layers.
• Predictors: Trainable MLPs that learn to reproduce those targets' outputs.
• Features: Mean-pooled hidden states from completion tokens, encoded without the prompt.
• Novelty signal: Root-mean-square prediction error, averaged across layers. Unfamiliar traces receive larger bonuses.

The bonus applies only to verifier-correct completions. Rewarding novelty on incorrect outputs destabilized training in the authors' experiments because novel garbage could earn additional reward.

The multi-round implementation scales exploration along two axes while keeping the RL update budget fixed:

• Parallel explorers: Start from the same checkpoint with different seeds; pool their traces before filtering.
• Repeated rounds: The previous student initializes both the next explorers and the next student.
• Fresh prompts: Each round receives a disjoint shard of the training problems.
• Update allocation: Divide 200 explorer updates and 100 student updates across explorers and rounds.
• Novelty schedule: For four rounds, decrease λ through 0.75, 0.50, 0.35, and 0.25.

Pass@k estimates the probability that at least one of k sampled completions is correct. The authors' evaluation protocol specifies:

• Base models: Qwen3-1.7B, Qwen3-4B, and Ministral-3-3B-Instruct-2512.
• Training data: DAPO-Math-17K mathematical reasoning problems.
• Reported mean: AIME24, AIME25, AIME26, MATH500, and Minerva-Math.
• Additional benchmarks: GSM8K, near saturation for all three models, and AMC23, near saturation for the Qwen3 models, are excluded from that mean.
• Sampling: Temperature 0.6, top-p 0.95, top-k 20, and a 32,768-token completion limit, with thinking enabled for Qwen3.
• Sample counts: 64 completions per problem, except 32 for AMC23 and eight for GSM8K.

The preservation claim covers five evaluations outside the math reward in the paper's capability suite:

• Knowledge: MMLU-Pro, MMLU-Redux, and GPQA-Diamond.
• Logical reasoning: ZebraLogic.
• Instruction following: IFEval.

Goldblum flagged reward hacking as a possible risk when transferring the method to coding tasks with less strict verification.

The Hugging Face release contains eight inference-only checkpoints, including multi-round and single-round ExpDis variants for each of the three base models.

• gpu/: Safetensors weights loadable with Transformers.
• tpu/: Flat dictionaries of Hugging Face-named parameters for JAX. These are not FlaxAutoModelForCausalLM directories.

Training uses JAX/Flax on TPU, with vLLM servers generating rollouts and receiving updated weights after every step, according to the repository README. Ministral training uses the BF16 instruct checkpoint; the repository marks FP8 training as unsupported.