Vik Paruchuri challenges ParseBench-2’s OCR scoring methodology

• Parser defaults can decide a score: Datalab’s Vik Paruchuri says his page-number example rewards keeping headers and footers even when both parsers preserve the body text.
• An ExtractBench score rose from 64.5% to 93.6% without changing extracted values, according to Datalab’s comparison of adapter and number-matching fixes.
• LlamaIndex acknowledged annotation and metric issues while defending its customizable severity-based framework.

A correct HTML table fares worse than Markdown containing a wrong number in Datalab’s edit-distance demonstration. LlamaIndex’s technical report describes coding agents refining evaluation metrics against fixed human judgments, with human review.

Vik Paruchuri, who develops OCR models at Datalab, challenged two representation choices in ParseBench-2:

• Headers and footers: His example reverses the winner when the answer key stops requiring the running header and page number. Both outputs preserve the body text.

• Bounding boxes: Datalab widened the benchmark’s own correct reference boxes to test how much harmless margin affects layout scoring.

The technical report explicitly defines text annotations as the whole page as printed, including running heads and page numbers. It also specifies layout boxes tight to the ink.

Paruchuri identified answer-key problems across different benchmarks:

• ParseBench charts: Datalab identifies a source value as 303 GW, while the test expects 310 GW within 1%. That accepts 306.9 to 313.1, excluding 303.
• olmOCR-bench text: A case-sensitive test requires “Republican Party” where the page prints “Republican party,” according to his transcription example.

The chart example also appears in Datalab’s analysis of the original ParseBench. LlamaIndex’s v2 release replaces its earlier chart spot checks with complete reference tables covering 36,958 values across 1,025 charts, according to the launch post.

Two examples separate scoring behavior from recognition quality:

• Edit distance: A correct HTML table scores 0.77 normalized edit distance against a Markdown reference. Markdown with 1,410 misread as 1,710 scores 0.016, where zero means identical, in Datalab’s demonstration. These are plain normalized Levenshtein distances, rather than a reproduction of an entire benchmark scorer.
• ExtractBench: Paruchuri reported a nasty 29.1-point swing with unchanged extracted values. Datalab says the original adapter counted confidence and citation metadata as wrong values; its revised result also used neutral number matching.

LlamaIndex launched ParseBench-2 on November 12 with 4,317 pages across six dimensions:

• Text
• Formatting
• Tables
• Charts
• Visual grounding
• Forms

The technical report defines each page’s score as the lower of content recovery and a severity cap. Its default policy requires one critical error to reduce an otherwise correct output by at least 10 points out of 100; benign variations should cost nothing.

Users can change penalties and recompute scores from recorded errors without rerunning the parsers. The policy is explicitly an expert judgment about consequences, rather than a measured property of the document.

Jerry Liu, speaking for the ParseBench team at LlamaIndex, accepted that annotations needed cleanup and that the overall score and default severity policy had received too much emphasis. He rejected the characterization of the benchmark as purely marketing, saying the team also uses it for internal research.

His responses distinguished several issues:

• Composite scores: The overall ranking is a simple average across dimensions, which Liu said deserves caution.
• An underline example: A disputed table-shaped page contributed only to Formatting, not Tables. Underlines account for 1% of the headline Formatting score, he clarified.
• Forms versus price: Liu highlighted Extend’s results.
• Tables versus price: He highlighted Datalab’s results.
• Charts versus price: He highlighted Reducto r-1 and said its chart evaluation had been corrected.

Liu also described Opus 5.5 as strong on tables and weak on charts in a follow-up, illustrating the capability differences hidden by an overall rank.

The disagreement extended to the evaluator itself:

• Replace rules: An LLM-based audit was possible within hours of launch, argued raunakdoesdev, who called for abandoning rule-based benchmarks.
• Keep deterministic primary metrics: Paruchuri favored LLM judges as secondary metrics, citing opaque behavior and sensitivity to model family and prompting.
• Preserve repeatability: Liu defended per-dimension deterministic scores, arguing that a single vision-model comparison can obscure granular errors.

The Parse Judge methodology exposes image-grounded error descriptions, severity, locations, and quotations alongside scores. Its author acknowledged prompt and model sensitivity in the discussion and proposed further prompt ablations.

Reducto shared a reproducibility complaint after finding that 23 of 37 rows differed by more than 1.5 points on Text or Charts when it re-scored published outputs.

A LlamaIndex maintainer explained in the GitHub issue that those outputs had not generated the v2 leaderboard. The team had conducted fresh runs, published the v2 outputs on November 13, and the complainant closed the issue.

A separate open Reducto pull request proposed re-scoring r-1 from 76.76 to 85.23 overall and from 25.23 to 78.45 on Charts. Its description attributes most of the change to outputs that provide chart data as HTML tables.

Paruchuri’s five-minute benchmark audit names seven checks:

1. Who built the benchmark, and whether their own system ranks first.
2. Score changes caused solely by output formatting.
3. The recorded source of each expected answer.
4. Reproducibility of published scores.
5. Vendor-specific prompts.
6. Remaining headroom and separation between systems.
7. Results sliced by document type.

Paruchuri called ParseBench’s ranking signal close to a coin flip, linking to Datalab’s November 4 comparison of seven open models across 13 public benchmarks. That study predates the ParseBench-2 launch and used three model judges’ error counts to assess rankings.

Separately, Nace.AI reported in its November 11 Drex results that adding its decision model improved its parser on three of six benchmarks. The company described results on two others, ParseBench and RealDocBench, as within noise of the parser alone.