OmniParseBench ranks 17 open OCR models with auditable scores

• Datalab benchmarked 17 open OCR systems; Chandra OCR 2 leads at 90.1 in the published ranking.
• Table extraction changes the winner: MinerU2.5-Pro scores 92.1 against Chandra's 87.4 in the category breakdown.
• Adapters and per-test verdicts are public, according to Paruchuri's disclosure, exposing the evidence behind a vendor-produced leaderboard.

Nemotron's client returned no document on 31 pages after hitting its output limit. The largest entrant, 35.1B-parameter Infinity-Parser2-Pro, finishes below 4.5B Chandra and 3B dots.mocr. The audit trail is the keeper.

Datalab's October 23 report adds open-weight results to OmniParseBench, which the company says it released two weeks earlier with an API leaderboard.

Only 15 systems receive headline scores. olmOCR-2 and Nanonets-OCR2-3B return no layout boxes, so Datalab publishes their text and table scores without an overall ranking.

Vik Paruchuri, author of Datalab's report, said the results generally matched the team's informal tests, calling dots.mocr and MinerU strong in a follow-up.

OmniParseBench uses 16,288 pass/fail tests on 2,937 pages from 2,343 documents, spanning 92 languages, as described in Datalab's write-up. Its five test types check specific errors:

• present: A particular passage, number or equation appears.
• order: Text A appears before text B.
• repeat: A line appears exactly the expected number of times.
• table_cell: A cell sits under the correct headings and beside the correct neighbors.
• layout_kind: The block containing a line has the correct type, such as heading, text or table.

The headline score is the mean of three family scores: text, tables and layout. The dataset card specifies that the scorer uses open code without a model judging outputs.

Datalab reports substantial differences between tasks in its category results:

• Reading versus table structure: LightOnOCR-3-4B beats dots.mocr on text, 83.9 versus 82.5, but trails on tables, 70.8 versus 88.6.
• Handwriting: Chandra leads the open models at 66.8; Gemini 3.8 Flash's API scores 79.4, a 12.6-point gap.
• Non-English content: Chandra scores 74.6 and dots.mocr 74.4 on tests tagged with a language other than English.

Tagged slices use the same family-averaging scheme as the headline. These are benchmark scores, not character-level recognition accuracy.

In the table failures Datalab examined closely, text was usually present but assigned to the wrong structure. Missing colspan or rowspan information can flatten merged headers and shift correctly recognized values under the wrong columns.

Datalab built the benchmark and owns Chandra, Surya and Indra; Paruchuri acknowledged the conflict in his post. The company publishes every page, test and adapter, plus per-test verdicts in results.parquet.

The adapter repository exposes how each vendor's pipeline becomes a scored output. Datalab's run settings include a 900-second timeout per page and disclose departures from vendor defaults:

• GLM-OCR: Configuration changes retain regions such as headers and footnotes that its SDK normally drops.
• olmOCR-2: The adapter removes the toolkit's last-resort fallback to embedded PDF text; exhausted OCR attempts count as failures.
• Chandra: The adapter adds retries for responses cut off at the token limit, beyond what its package detects itself.

The broader runner also supports closed API systems, which Paruchuri pointed readers toward in a reply.

Indra's parameter count is 1.25M for its main recognizer and 12.3M including its math models. Its headline changes with the extraction mode:

• Default, with math models: 71.4, using embedded PDF text where it passes checks and OCR elsewhere.
• Without math models: 66.7.
• Every page treated as an image: 61.2.

Throughput also depends on hardware: LearnOpenCV reported 49 pages per second on 32 CPU cores and about seven on four cores in its Indra demonstration.

An upside-down scan processed in 344 ms on two vCPUs