Nous releases Qwen3.8-27B Hermes adapter with 43.1% SWE-bench Pro Hard pass@1
The Qwen3.8-27B Hermes adapter raises SWE-bench Pro Hard pass@1 from 29.4% to 43.1%. Its non-thinking mode reportedly matches the base model's xhigh performance with substantially fewer tokens.

TL;DR
- Qwen3.8-27B gains 13.7 points on SWE-bench Pro Hard 51, from 29.4% to 43.1%, and 14.1 points on Terminal-Bench 2.1, from 44.7% to 58.8%, per the release announcement.
- Under a 10,000-token cap, the non-thinking adapter solves 39% of SWE tasks versus 0% for the base model at xhigh, according to yb2698's comparison.
- The release offers a Hermes-specific LoRA for the existing base model, with a separate GGUF download for llama.cpp.
The model card replaces an earlier 27.5% → 49.0% SWE-bench result after tunnel failures forced reruns on both models. Only three of the 51 non-thinking adapter trajectories survived a partial rerun that overwrote the archive, although all 51 grades remain available.
Hermes pass@1
The harness-hub project released a reinforcement-learned LoRA for Qwen3.8-27B inside Nous Research's Hermes Agent. Its paired evaluation reports these thinking-off results:
- SWE-bench Pro, hardest 51 tasks: 29.4% → 43.1%, +13.7 points; 10 newly solved, three lost; exact sign-test p = 0.092.
- Terminal-Bench 2.1: 44.7% → 58.8%, +14.1 points; 19 newly solved, seven lost; p = 0.029.
Both models received one attempt per task, a 100-step budget, a 131,072-token context window, temperature 0.8 and top-p 1.0, with vision disabled. The terminal comparison covers 85 of 89 tasks; four were excluded because an environment failed to build or a verifier timed out in one of the runs.
10,000-token runs
At a 10,000-generated-token cap, the adapter's advantage widens:
- Terminal-Bench 2.1: base model at xhigh 28% → non-thinking adapter 51%, +23 points.
- SWE-bench Pro Hard 51: base model at xhigh 0% → non-thinking adapter 39%, +39 points.
Without that cap, the non-thinking adapter equals the base model's xhigh Terminal-Bench score, 58.8%, and trails its SWE score by two points, 43.1% versus 45.1%. The model card reports median generated tokens falling from 22.5k to 6.5k on SWE and from 21.0k to 4.8k on Terminal-Bench, the basis for the 3.5–4.4× reduction.
Against the base model with thinking also disabled, median tool calls on tasks both models solved fell from 63 to 28 on SWE and from 23 to seven on Terminal-Bench. Those shared-solved-task measurements isolate work spent reaching a successful answer.
35-call qutebrowser fix
The demonstration task asks qutebrowser to preserve percent-encoded URL fragments such as %3A when incrementing or decrementing a number in a URL. The adapter changes the regex and path setter, while the base model spends its budget probing Qt URL encoding and breaks a test file.
Thinking and context overflow
At xhigh reasoning effort, the base-to-adapter comparison changes:
- SWE-bench Pro Hard 51: 45.1% → 51.0%, +5.9 points.
- Terminal-Bench 2.1: 58.8% → 57.6%, −1.2 points.
The evaluation used no context compaction. Hermes sent each turn's reasoning back to the model, and sessions exceeding 131,072 tokens ended with a failing grade, often after only a few dozen steps.
The model card explicitly limits its thinking comparison to that configuration; larger windows or compaction could change the results. The adapter was post-trained with thinking off, and yb2698 said thinking-on post-training would be a subsequent experiment.
Harness-specific RL
Each adapter specializes a model to a harness's tools, prompts and control flow. The Hermes training configuration specifies:
- Policy: TRL's experimental AsyncGRPO trainer, with rank-32 LoRA and alpha 64.
- Environments: Harbor tasks served through OpenEnv, with rollouts in Hugging Face sandboxes.
- Reward: task-verifier correctness plus a tool-efficiency bonus weighted at 0.1.
- Rollouts: eight per task, thinking off, up to 50 agent steps and 1,800 seconds each.
- Compute: two NVIDIA B200s, one serving vLLM and one training; the released checkpoint is step 160.
The project's safe-RL write-up describes safeagents, a proxy that checks every tool call with rule-based detectors. Reward-hacking attempts are blocked, score zero and receive an explanation of the block.
The project also promised more adapters, training recipes and RL environments in ben_burtenshaw's follow-up.
Prefix caching
The Qwen serving example enables both --enable-prefix-caching and --mamba-cache-mode align. The model card says that without the latter flag, every agent turn re-reads its entire history.
Separately, a merged Hermes patch moves GPT-6.2+ system prompts from top-level instructions into a leading developer input item to restore OpenAI Responses API prompt caching. That patch concerns Hermes's OpenAI transport.
GGUF and merged weights
The GGUF release is an approximately 319 MB adapter file loaded alongside a base-model GGUF. The reported benchmark scores come from BF16 serving through vLLM; the GGUF adapter has not been re-evaluated through llama.cpp, and quantizing the base changes the weights it was trained against.
Community builds also package merged model-plus-adapter weights.
The model card documents merging with PEFT into a standalone checkpoint of roughly 54 GB in BF16. Speculative decoding with a merged model should work, yb2698 said, while explicitly noting that they had not tested it.