ReproBridge paper reports experiment reproduction rising to 92.6%

• Agent reproduction success rose from 76.4% to 92.6%, +16.2 percentage points, across four models, according to omarsar0's summary.
• Successful reproduction work becomes a reusable package for a fresh agent in the linked ReproBridge paper.
• Downstream API spending fell 19.7% in the experiments described by omarsar0, excluding the cost of constructing and auditing the packages and GPU runtime.

One case study starts with a released data loader that labeled all 10,000 TinyImageNet validation images as one class. Across 36 packaged repositories, the researchers recorded 932 ambiguity entries and 482 launcher scripts.

Researchers at Google and the University of Waterloo define four components in the paper's artifact specification:

• Refined codebase: src/ contains repaired code and missing experiment support.
• Environment setup: Dockerfile and setup/ preserve compatible dependencies and data or checkpoint acquisition procedures.
• Experiment launchers: experiments/ connects each task to its configuration, execution scope, outputs, and measurement extraction.
• Reproduction guide: reproduction_guideline.md and change_log.md explain settings, paper-code discrepancies, repairs, pitfalls, and unresolved choices.

A separate verify.sh provides the package-verification entry point. Guide entries preserve the source of an ambiguity, the adopted resolution, its rationale, sensitivity, and affected tasks.

Six agent roles build the artifact in three stages, according to the construction workflow:

1. Specify: Intake maps the paper, repository, and tasks. Planner fixes the requested measurements and execution plans before runs begin.
2. Execute: Builder prepares the environment. Reproducer runs experiments and records measurements, repairs, and failure reasons.
3. Preserve: Distiller writes the guide and change record. Packager assembles the deliverable and verifies the handoff.

An independent Reviewer checks plans, execution records, and the finished package in separate contexts, returning findings for revision. Humans define the workflow and delivery standards; agents perform construction, review, and packaging.

Delivery requires a clean-shell test, including an inexpensive real repetition. Downstream evaluation uses fresh sessions without the construction conversation, and human reference scores are withheld during both construction and reuse.

The reported results average three evaluations:

| Reproducing model | Raw code | Refined artifact | Gain |
| --- | ---: | ---: | ---: |
| Sonnet 4.6 | 72.3% | 91.6% | +19.3 points |
| Sonnet 5 | 76.5% | 93.3% | +16.8 points |
| Opus 5 | 89.2% | 94.7% | +5.5 points |
| Gemini 3.8 Flash | 67.6% | 90.8% | +23.2 points |

The best-to-worst spread fell from 21.6 to 3.9 percentage points, a striking payoff for preserving execution knowledge. Vendor harnesses differed, so the authors describe cross-model comparisons as descriptive.

ReproBridge-Bench evaluates 1,029 tasks from 387 experiments across 36 ICML 2026 Spotlight papers. Its scoring protocol combines numerical agreement with evidence that the intended experiment actually ran:

• Reference measurements: Five independent, human-verified observations anchor each task. A task's specified statistic can itself be a multi-seed mean, rather than a single run's score.
• Numerical gate: A new measurement must fall inside a reference-based 95% prediction interval, with a minimum half-width of 1% of the absolute reference mean.
• Execution gate: A separate Auditor traces reported values to output files, execution records, and generating scripts. A shipped score file cannot count as a new measurement.

Published targets remain visible to agents, while human reference measurements stay hidden. The compatibility test is two-sided: even a result better than the published score fails if it falls outside the interval.

The headline success rate applies to tasks already shown to be reproducible by the reference process. Of 1,463 curated tasks, 1,029, or 70.3%, entered downstream evaluation, according to the benchmark selection records.

The other 434 tasks fell into three categories:

• 183 numerical mismatches: Executions completed, but results were incompatible with the published score.
• 186 execution blockers: Missing artifacts, dependency problems, or code defects prevented execution.
• 65 resource or access limits: Compute capacity or restricted access blocked verification.

Selection also excluded baselines, stand-alone ablations, figures, and appendix-only results. The evaluated measurements came from the papers' own methods in their main tables.

A component-removal study used Sonnet 5 on 287 tasks from 12 papers in one evaluation. The complete artifact achieved 93.0% success:

• Without refined code: 87.8%, down 5.2 points.
• Without environment setup: 88.5%, down 4.5 points.
• Without experiment launchers: 89.9%, down 3.1 points.
• Without the reproduction guide: 89.2%, down 3.8 points.

Removing code, environment setup, or launchers also removed descriptions of that component from the remaining documentation and scripts. These losses measure each component's contribution conditional on the others remaining available.

Building the 36 artifacts with Opus 5 cost $5,026 in API usage, or $139.61 per paper, according to the project's cost accounting. GPU runtime was a separate expense.

The reported 19.7% saving covers downstream Reproducer API spending, including failed attempts. Construction, auditing, and GPU costs are excluded, so that percentage does not establish an end-to-end cost reduction.

The project page says code and data are undergoing internal review and will be open-sourced once approved.