Vals disables internet access for SWE-Bench Multimodal after agents find upstream fixes

• Internet access let agents copy upstream fixes. After blocking it, ValsAI reported score drops of 21 to 35 percentage points across three models.
• Claude Opus 5.5 led the offline results at 70.2%, roughly nine points ahead of Claude Fable 5.1, according to ValsAI's ranking.
• Higher spending did not guarantee higher scores: ValsAI reported $11.40 per task for Gemini 4 Argon versus $4.92 for the higher-scoring Opus 5.5.
• Each model ran once in mini-swe-agent, with the final diff graded by the upstream SWE-bench harness, per ValsAI's methodology.

The public dataset carries separate patch and test_patch fields. Vals downloads screenshots before each run, so cutting network access preserves the visual bug report.

SWE-bench Multimodal asks agents to fix real GitHub issues in JavaScript repositories where screenshots or other visual cues supply part of the problem, as ValsAI explained in its announcement.

With internet access, agents found the upstream pull requests and copied their fixes. Every task's instance_id contains the number of the pull request that resolved the issue, making the official patch easy to locate.

The answer-retrieval route was built into the task identifier. Agents could use the network to obtain a completed repair rather than derive one from the bug report and repository.

Across three Muse variants, ValsAI reported drops of 21 to 35 points. Its methodology page gives the paired diagnostic results:

| Model | Internet on | Internet off | Change |
| --- | ---: | ---: | ---: |
| Muse Spark 1.2 | 62.71% | 41.25% | -21.46 points |
| Muse Spark 1.3 | 69.17% | 42.08% | -27.09 points |
| Muse Spark 1.3 Max | 77.92% | 43.33% | -34.59 points |

Vals labels the internet-enabled runs as diagnostic. Every published leaderboard score uses an agent without internet access.

Claude Opus 5.5 took first place in ValsAI's results. The published leaderboard gives these unrounded scores:

1. Claude Opus 5.5: 70.21%.
2. Claude Fable 5.1: 61.25%.
3. Claude Sonnet 5.5: 58.33%.
4. Gemini 4 Argon: 56.04%.

Even the smallest measured network penalty was more than twice Opus's 8.96-point lead over Fable. The network ablation is the standout result here.

Gemini 4 Argon cost about 2.3 times as much per task as Opus 5.5 while trailing it by 14.2 points. DeepSeek V4.1 Flash beat Gemini 3.8 Flash by 5.6 points at approximately one-sixteenth of the cost.

Runtime also favored Opus in that comparison: Argon's listed duration was 33m38s, versus 18m32s for Opus 5.5 in Vals's table.

Vals's methodology covers 480 tasks from eleven repositories. Each task runs in its own sandbox, and the agent receives the issue text and a shell, continuing until it decides it is finished.

The error bars are binomial standard errors over task outcomes. With only one run per model, the evaluation provides no estimate of variation across repeated agent runs.

The upstream Multimodal v2 release, dated September 1, 2026, removed known flaky or ungradeable tests. Its evaluation infrastructure includes:

• The complete test split and evaluation tooling, available as open source.
• Rebuilt Docker environments addressing dependency and browser drift.
• More robust JavaScript grading and visual test-asset handling.