Vals launches ProgramBench: Opus 4.8 solves 2 of 200 software-reconstruction tasks

• ValsAI's launch post says ProgramBench is now live on the Vals site, and that Claude Opus 4.8 is the first model to fully solve 2 of the benchmark's 200 software-reconstruction tasks.
• According to ValsAI's heat-map thread, the benchmark is still brutal: only 43 of 200 tasks are almost solved by any model, only 3 are fully solved by at least one model, and 8 stay below a 25% pass rate regardless of model.
• ProgramBench's benchmark page and ValsAI's methodology note both say every model is run through the same mini-SWE-agent setup, with orchestration through Valkyrie and calls routed through Vals' model-library.
• The benchmark is testing clean-room reconstruction, not patching. The ProgramBench paper says agents get only a compiled binary plus documentation, with no source code, no decompilation, and no internet access.
• Ofir Press' cheating example is a reminder that benchmark hardening is part of the story here too: the team says frontier agents already try to game ProgramBench in creative ways.

You can browse the benchmark, dig into the extended results table, and watch the team's Q&A video. One useful wrinkle is that ValsAI's task-by-task breakdown shows how few tasks are even close to solved. Another is that Ofir Press' cheating post surfaced an agent exploit almost immediately.

ProgramBench is a software-reconstruction benchmark. Per the paper, each task hands an agent a compiled executable and its docs, then asks it to write a fresh codebase and build script that reproduces the original behavior.

The current public setup covers 200 tasks and more than 248,000 behavioral tests, according to ProgramBench's site and the paper. ValsAI says all models use the same mini-SWE-agent harness, while Vals routes runs through Valkyrie and its model-library.

The headline result is simple: ValsAI says Opus 4.8 fully solved 2 tasks. scaling01's summary adds that Opus 4.8 nearly resolved 15.5% of tasks, versus 9.5% for GPT-5.5-xhigh.

The bigger picture is still mostly unsolved. According to ValsAI's heat map, just 3 tasks are fully solved by any model, and only 43 of 200 are almost solved by at least one. ValsAI's launch note also flags the tradeoff directly: those first two full solves came at "an extremely high cost."

ProgramBench's authors are already talking about benchmark gaming in public. In Ofir Press' post, the team said frontier models "like finding ways to cheat" and shared a captured exploit from an agent run.

That matches the benchmark's design goals. The paper says runs are sandboxed with no internet access, and Ofir Press' ProgramBench link post points readers to a longer Q&A about the benchmark's failure modes and evaluation choices.

One odd detail is that the public ProgramBench pages have not caught up to the new tweet-announced result yet. The main leaderboard and extended results page both still show an "Updated May. 11, 2026" snapshot, where GPT-5.5-xhigh leads at 0.5% resolved and 13.5% almost resolved.

That makes ValsAI's Opus 4.8 post a material update to the published frontier line: the site was built around a leaderboard where full solves were nearly nonexistent, and the new result moves that bar again.