Meta–CMU study reports Qwen3 8B beats Maverick in tool selection, 41.5% to 30.6%

• A trained 8B selector reached 41.5% exact tool selection on ToolBench G3 versus 30.6% for prompted 402B Maverick, according to rohanpaul_ai's report.
• Reinforcement learning added 5.3 percentage points over supervised fine-tuning alone in dair_ai's breakdown of the Plan, Retrieve, Select approach.
• Transfer worked without retraining: the 8B model beat Maverick on three of four additional tool benchmarks, per rohanpaul_ai.

Maverick generated the training examples for the smaller model that beat it. Each inference call is stateless, carrying no conversation history between selection steps.

The Meta–Carnegie Mellon study's evaluation covers 2,124 test instructions against a catalog of 1,419 individual APIs. Success requires an exact match between the selected tool set and the reference set.

• Llama-4-Maverick, 402B, prompted: 30.6%.
• Qwen3-8B, supervised fine-tuning: 36.2%.
• Qwen3-8B, supervised fine-tuning plus RL: 41.5%, +10.9 points over Maverick.

The researchers curated the reference tool sets, retaining tools inferable from the request and removing unnecessary tools during teacher-generated decomposition. That filtering applies to both training and evaluation.

Plain BM25 handles retrieval, a pleasingly unglamorous choice in the three-stage workflow:

1. Plan: Split the instruction into an ordered list of substeps, each describing one required capability.
2. Retrieve: Search tool descriptions using each substep's text. By default, retain candidates scoring above 3 and cap the list at 15 tools.
3. Select: Read the candidate descriptions and choose one tool per substep. Return the union of those selections.

The same model handles planning and selection under different prompts.

Training starts with supervised fine-tuning on about 20,000 curated ToolBench instructions, according to dair_ai. The RL procedure then uses GRPO to optimize planning and selection separately:

• Plan reward: An F1 score combines the fraction of substeps matched to distinct gold tools with rank-weighted gold-tool coverage. Each matched tool contributes inversely to its retrieval rank, rewarding plans that retrieve required tools near the top and penalizing redundant steps.
• Select reward: One point for choosing the substep's exact gold tool; zero otherwise.

During selector training, the model receives teacher-generated substeps instead of its own plans, separating selection learning from planning errors.

Without further training, the 8B selector outperformed prompted Maverick on three of four additional benchmarks, according to rohanpaul_ai's summary. The paper's transfer experiments evaluated:

• ToolBench G2
• Gorilla
• CRAFT
• TaskBench

The BM25-trained selector also transferred to COLT, a domain-specialized dense retriever, without additional model training.

Every substep must be solvable with exactly one tool, and no substep can require another's output, under the workflow's assumptions. The measured task ends at tool selection, before API execution.