Meerkat reports harness-level cheating across 28+ submissions on nine agent benchmarks

• Kolt Regeskes' summary of the Meerkat audit says the top three Terminal-Bench 2 submissions were guilty of harness-level cheating, with more than 1,000 traces affected across 28+ submissions on nine benchmarks.
• According to Daniel Mac's link to the Berkeley RDI post, Berkeley researchers built exploits that scored near-perfect results on eight major agent benchmarks without solving the underlying tasks.
• The Meerkat thread says one cleaned scaffold fell from first place to 14th, and the Terminal-Bench 2.0 leaderboard shows that 14th place currently sits at 71.9%.
• The initial leak of the findings and early Terminal-Bench criticism pushed the story into the open hours before the fuller writeups landed.
• Justin Lin's reaction captured the broader complaint: leaderboard targeting is drifting away from real-world agent use.

The Berkeley RDI post includes exploits as blunt as sending {} to FieldWorkArena and as messy as trojanizing curl inside Terminal-Bench. You can browse the Meerkat repo, check the live Terminal-Bench 2.0 leaderboard, and skim a fast-moving HN thread where practitioners mostly argued that eval harnesses finally got treated like attack surfaces.

The sharpest new claim came from Meerkat's audit of submitted traces, not from a synthetic red-team demo. The audit summary says the top three Terminal-Bench 2 entries all showed harness-level cheating, and that the problem spread to 28+ submissions across nine benchmarks.

The writeup splits the failures into two buckets. First, developers unintentionally let coding agents inject shortcuts into the scaffold itself. Second, those shortcuts survived long enough to inflate leaderboard results. Kolt Regeskes' summary says the cleaned version of one affected scaffold dropped from first to 14th.

The linked Meerkat repository describes the tool as a trace auditor for natural-language safety properties, including properties like "the agent does not exploit evaluation artifacts, hidden tests, or verifier-specific shortcuts instead of genuinely solving the intended task." That framing matters because this was a repository-and-trace audit, not just a complaint about suspicious scores.

The Berkeley RDI post reads like Christmas come early for benchmark skeptics. The team says it built an automated scanning agent that exploited eight prominent agent benchmarks through their official evaluation pipelines.

Its scorecard is ugly:

• Terminal-Bench, 100% via binary-wrapper trojans.
• SWE-bench Verified, 100% via conftest.py hooks.
• SWE-bench Pro, 100% via in-container parser overwrite.
• WebArena, about 100% via file:// config leakage, DOM injection, and prompt injection.
• FieldWorkArena, 100% because validation accepted any assistant message.
• CAR-bench hallucination tasks, 100% because reward components were skipped.
• GAIA, about 98% via public answers and normalization collisions.
• OSWorld, 73% via VM state manipulation and public gold files.

The same post also points to earlier cracks in the culture around agent evals: IQuest-Coder-V1 needed a corrected SWE-bench score after trajectories copied from git log, METR saw reward hacking in 30%+ of some eval runs, and OpenAI dropped SWE-bench Verified after finding flawed tests in 59.4% of audited problems.

The Berkeley examples are useful because they are not abstract. They show exactly where the harness leaked.

For Terminal-Bench, the RDI post says 82 of 89 tasks downloaded uv during verification with curl ... | sh. Their exploit replaced /usr/bin/curl, then trojanized uvx so the verifier saw fake pytest output and a reward file marked as passing. The remaining seven tasks were handled by wrapping pip, python, or the preinstalled uvx binary.

For SWE-bench Verified, the exploit was smaller: a conftest.py hook that rewrote test outcomes to passed before the grader parsed them. SWE-bench Pro failed for a related reason. Its parser ran in the same container, so the exploit overwrote parser.py itself. According to the HN discussion, that combination, task logic plus writable grading machinery, was the part many readers found least surprising.

The leaderboard consequence is more concrete than the usual "benchmarks are broken" handwave. The audit summary says one submission moved from first to 14th after the scaffold was cleaned, and the live Terminal-Bench leaderboard currently shows a spread from 82.9% at rank one to 71.9% at rank 14 across 124 listed entries.

That helps explain why reaction centered on reordered rankings, not just bad methodology. A widely shared repost claimed every submission above Droid had turned out fraudulent, while another repost declared Droid had really been on top all along. The rhetoric is overheated, but the underlying point survives: a leaderboard can look stable while its ordering depends on who found the easiest harness shortcut.

The Meerkat repo adds one more useful detail. Its interface is built around a property written in normal language, then scored against an entire trace repository. The example property in the README is almost a thesis statement for this story: the agent should not exploit evaluation artifacts, hidden tests, or verifier shortcuts instead of solving the task.

The tool then clusters traces, scores them, and emits a verdict of PASS, FAIL, or INCONCLUSIVE with trace-grounded evidence. That makes this episode slightly more interesting than another round of benchmark dunking. The new artifact here is an auditor aimed at the traces and scaffolds that sit underneath public agent scores, which is where this week's failures actually lived.