Skip to content
AI Primer
release

Qwen3.8-27B adapter raises SWE-bench Pro Hard pass@1 from 29.4% to 62.7%

An open RL-trained LoRA adapter for Qwen3.8-27B raises reported pass@1 from 29.4% to 62.7% on SWE-bench Pro Hard 51. It runs inside Pi and uses 55–62% fewer tool calls.

4 min read
Qwen3.8-27B adapter raises SWE-bench Pro Hard pass@1 from 29.4% to 62.7%
Qwen3.8-27B adapter raises SWE-bench Pro Hard pass@1 from 29.4% to 62.7%

TL;DR

  • A Pi-specific LoRA raised reported SWE-bench Pro Hard-51 pass@1 from 29.4% to 62.7%, +33.3 points, per the release announcement.
  • Terminal-Bench 2.1 rose from 39.5% to 55.8%, +16.3 points, with 55-62% fewer tool calls across the two benchmarks, according to the launch thread.
  • SWE-bench scores helped select the checkpoint, the model card discloses. Both models had thinking disabled and identical 100-step caps, as the team clarified.

The training notes record a run stopped at step 191 of 1,000 planned, with step 160 released. The deployment notes warn that Pi's default API selects an unevaluated protocol, while a missing vLLM cache flag makes every turn reread its entire history.

Pass@1 under Pi

Qwen3.8-27B was adapted specifically for Pi through reinforcement learning, as described in the release thread. Both models were tested once per task at temperature 0.8, according to the model card.

The paired task outcomes included:

  • SWE-bench Pro Hard-51: 18 newly solved tasks and one previously solved task lost.
  • Terminal-Bench 2.1: 16 newly solved tasks and two previously solved tasks lost.

Checkpoint selection

Step 160 was chosen using scores on SWE-bench Pro and an 18-task Terminal-Bench subset, according to the selection disclosure.

The cleaner comparison is the 68-task Terminal-Bench slice excluded from checkpoint selection: 39.7% → 55.9%, +16.2 points, with 13 newly solved tasks and two lost.

The lower baseline versus Qwen's official Terminal-Bench result reflects a different harness, Pi rather than Terminus2, and disabled thinking, according to the team's explanation.

The 100-step cap

About two-thirds of newly solved tasks had previously exhausted the base model's budget; the adapter solved those in a median of about 40 calls.

Efficiency gains also persisted on tasks both models solved, according to the paired results:

  • Terminal-Bench: 48% fewer tool calls across 32 shared successes.
  • SWE-bench Pro Hard-51: 59% fewer tool calls across 14 shared successes.

Real Pi rollouts

Every RL rollout ran an actual Pi session in a Hugging Face sandbox, with no reimplemented agent loop inside the trainer, according to the training thread. The team used the earlier harness-adapter recipe, with these training settings:

  • Trainer and environment: TRL's experimental AsyncGRPOTrainer, with Harbor tasks served through OpenEnv.
  • Data: 750 scale800 tasks interleaved with 300 TermGrade tasks; 1,919 task groups processed by the released checkpoint.
  • Reward: verifier correctness plus a tool-efficiency bonus weighted at 0.1.
  • LoRA: rank 32, alpha 64, targeting q, k, v, o, gate, up and down projections.
  • Rollouts: eight per task, up to 50 agent steps and 1,800 seconds each, with thinking off.
  • Hardware: two NVIDIA B200s, one for vLLM and one for training.

A safeagents proxy screened tool calls before Pi execution, replacing blocked calls with text notices.

Serving and protocol settings

The open adapter is linked from the announcement, and a serving follow-up lists six deployment routes:

  • transformers serve.
  • vLLM.
  • sglang.
  • llama.cpp.
  • Llama.app on macOS.
  • A merged base-model checkpoint, without a separate adapter at runtime.

Two settings in the model card control protocol and cache behavior:

  • Pi: ~/.pi/agent/models.json needs "api": "openai-completions". Without it, Pi uses the Responses API, which this adapter was neither trained nor evaluated with.
  • vLLM: prefix caching on this hybrid model needs --mamba-cache-mode align. Without it, every agent turn reprocesses the full history.

The adapter was trained against BF16 weights. The GGUF deployment notes say lower-bit base-model quantizations drift further from that behavior, and the GGUF version has not been re-evaluated.

Further reading

Discussion across the web

Where this story is being discussed, in original context.

On X· 5 threads
TL;DR2 posts
Pass@1 under Pi3 posts
Checkpoint selection1 post
Real Pi rollouts1 post
Serving and protocol settings2 posts
Share on X