LlamaIndex releases ParseBench-2 for parsing more than 4,300 pages

• ParseBench-2 roughly doubles the corpus: LlamaIndex's launch thread describes 4,300-plus pages and adds forms as a sixth parsing dimension.
• Examples affect only their annotated dimensions, LlamaIndex explained in a follow-up, so an underline test does not contribute to the Tables score.
• Leaderboard reproducibility is disputed: raunakdoesdev linked an audit finding mismatches in 23 of 37 rows checked on Text and Charts.

In a 459-cell table test, a wrong number finally gets expensive: its penalty rises from 0.13 points to 10, while a correct table written in Markdown stops getting a zero. The technical report also describes coding agents rewriting metric implementations against fixed, human-set targets.

Released November 12, ParseBench-2 contains 4,317 page-level cases from 2,645 source documents, according to its dataset card. Its coverage spans:

• Text: transcription, omissions, changed facts, and reading order.
• Formatting: headings, hierarchy, strikethrough, underlines, and other meaning-bearing markup.
• Tables: values attached to the correct row and column headers, including hierarchical headers.
• Charts: complete data series across 1,025 charts, replacing spot checks of selected values.
• Visual grounding: extracted content traced to its position on the page, accepting equivalent block segmentation.
• Forms: 600 pages testing field values, checkbox states, signatures, and fields that must remain blank.

The open-source evaluation package runs the new dataset with parse-bench run <pipeline_name> --dataset parsebench-2.

Each page receives the lower of its content-recovery score and a severity cap. The paper's severity policy defines these target score drops:

• Critical: at least 10 points.
• Major: at least 5 points.
• Minor: 1 to 25 points.
• Benign: no penalty.

These are calibration targets. The Formatting metric met the critical-error target in 96% of tested scenarios.

The launch describes “330+ error types”; the paper's catalog counts 330 perturbations comprising 240 error types and 90 benign variations. Detected errors are recorded separately from their penalties, allowing scores to be recomputed under a different severity policy without rerunning the parsers.

An underline example is evaluated on Formatting rather than table structure, LlamaIndex clarified in an initial reply. Its longer response explained that each example contributes only to the dimensions for which it has annotations.

The published aggregation rules define the overall score as the unweighted mean of six dimensions. They assign underlines 1/16 of Formatting, equivalent to about 1.04% of the overall score; the reply instead describes underlines as 1% of the headline Formatting score.

The dataset card also explicitly says ParseBench-2 scores are not comparable with ParseBench-1 scores.

Alongside the benchmark, LlamaIndex introduced OCRia-1.5 to power all LlamaParse modes, according to the launch thread. The company's launch ranking reports these overall scores:

• Cost Effective: 84.6.
• Agentic: 86.0.
• Agentic Plus: 89.8.

In that vendor-run comparison, the highest-scoring direct vision model reached 85.7, and the strongest competing document parser reached 83.9.

On November 13, raunakdoesdev reported that re-scoring existing published outputs reproduced only 14 of 37 leaderboard rows within 1.5 points on both Text and Charts. The GitHub issue specifies scorer commit 40efeac and dataset revision a12c9785.

The reported discrepancies include:

• LlamaParse Agentic, Text: 87.01 on the leaderboard versus 68.31 when re-scored.
• Reducto r-1, Charts: 25.23 versus 4.88.
• Other dimensions: largely uncheckable because the published results repository contains few or none of their outputs.

The author identifies changed document bytes, revised reference annotations, and scorer versions as possible contributors to the gaps.

A separate, still-open Reducto pull request publishes a new, complete 4,317-page output set and proposes changing r-1's overall score from 76.76 to 85.23. Its Charts score rises from 25.23 to 78.45; the submission says these outputs provide chart data as HTML tables.

That PR also fixes a relative --gt-cache path that caused the v2 evaluator to look under the output directory and report every subset as “not evaluated.”