BenchBench finds slight page tilts change winners in 6 of 12 parser benchmarks
BenchBench found that slight page tilts changed the winner on six of 12 parser benchmarks. The study tested eight parsers without changing benchmark scoring rules.

TL;DR
- A 2–4° page tilt changed the winner on six of 12 document-parsing benchmarks, according to the BenchBench announcement, which covered eight parsers.
- Leaderboard margins were smaller than tilt losses: raunakdoesdev reported that the median percentage score drop exceeded the original percentage lead over second place on nine benchmarks.
- Judge choice changed the result: Reducto's study found only two winner changes under PAIR, its vendor-blind LLM judge.
A strikeout-page example kept its text and strikethroughs yet scored zero for dropping decorative styling. A new Datalab model prompted a full rerun, replacing an earlier eight-of-12 winner-change count with the current six.
Eight parsers, fixed presets
Reducto's released harness specifies the experiment:
- Sample: Up to 500 pages per benchmark, using seed
20261107. - Tilt: Add a 2% white margin and rotate pages by 2–4°, following ParseBench's tilt protocol.
- Parse: Send identical images to eight fixed presets:
- Score: Use each benchmark's native scorer and PAIR, running GPT-6.1 Sol at medium effort with high image detail.
- Aggregate: Average two clean-page PAIR repeats; use one tilted-page judge run.
Benchmark scoring rules stayed unchanged, with golden bounding boxes adjusted for tilt where applicable. Each parser had one configuration, so the study measures those presets rather than every mode each vendor offers.
Six native winners changed
Six benchmarks changed first place under their own scorers in the published results:
| Benchmark | Upright winner | Tilted winner |
| --- | --- | --- |
| DP-Bench | Cohere | Mistral OCR |
| OmniParseBench | Datalab | Reducto |
| IDP Leaderboard | LlamaParse | Reducto |
| RD-TableBench | LlamaParse | Reducto |
| PureDocBench | LlamaParse | Reducto |
| ParseBench-2 | LlamaParse | Reducto |
Across the tested parsers and benchmarks, median relative native score loss was 2.2%. Five of eight providers moved in the overall average-rank comparison, the author reported.
Two steadier cases emerged in a follow-up:
- OCRFlux-bench: Same winner before and after tilt under both native scoring and PAIR. Median score drops were 1.0% and 2.3%, respectively; three of eight parsers changed rank.
- MDPBench: Only two of eight parsers changed rank.
Zero for a preserved table
One ParseBench-2 regression-table parse retained every coefficient and standard error, but placed each standard error inside its coefficient's cell instead of on a separate row. The native scorer assigned zero, according to the study's example.
Zero is an unforgiving grade for that representation mismatch. Reducto also acknowledges that rule-based scorers have value when exact output requirements matter.
PAIR versus native scoring
PAIR reads the original upright page image alongside all eight anonymized parser outputs. It lists errors before assigning scores from 1 to 10.
Reducto reports three results in its study:
- Winner changes: Two of 12 benchmarks under PAIR, versus six under native scoring.
- Median relative tilt loss: 4.0% under PAIR, versus 2.2% under native scoring.
- Human agreement: 84% when both PAIR and the reviewed human verdict selected a winner, in a separate study of 300 pairwise comparisons with at least three human ratings each.
Reducto sells one of the tested parsers, wrote the judge prompt and built RD-TableBench. Its parser won 10 of the 12 clean-page benchmarks under PAIR in the released results.
Judge-model and prompt ablations
BenchBench's judge checks compare winner agreement with the main Sol judge across 24 benchmark-condition cases, covering upright and tilted pages:
- GPT-6 Astra: 24/24.
- Claude Fable 5.2: 24/24.
- Gemini 4 Argon: 15/24.
- Two simplified prompts: 22/24 each, including a minimal “which parse is better?” prompt.
Removing every page containing figures also left the findings intact, the author said.
Open runs and rerun costs
The BenchBench repository publishes the reproduction machinery:
- Code: Apache-2.0 licensed.
- Run data: Parser outputs, vendor responses, judge HTTP exchanges, manifests and native scores.
- Restoration:
restore_runs.pyexpands archived data into the harness layout. - Validation:
validate_native.pyrechecks native score means.
Benchmark datasets retain their own licenses; the repository grants no rights to redistribute page images.
Full reruns remain expensive, especially the LLM-judge stage, according to the author, who offered to incorporate results from additional parsers.
A separate ParseJudge interface applies the vendor-blind comparison to custom documents. Its upload page accepts PDF, DOCX, PPTX or ZIP files, up to 1,000 pages per run.