How to compare OCR models on your own documents

Public leaderboards tell you how a model does on someone else's documents. Before you switch OCR models in production, run a sample of your own. With OpenParser that takes one script, because every model takes the same request and returns the same openparser@1 document.

Pick a sample

Twenty to fifty pages usually show the differences. Include the documents that cause support tickets: scans, rotated pages, dense tables, handwriting, small print. Keep the PDFs in one folder.

Run every model on every file

This script parses each file with each model and saves the results as JSON. It uses the Python SDK (pip install openparser-sdk) and reads your key from OPENPARSER_API_KEY.

import json
import time
from pathlib import Path
 
from openparser import OpenParserClient
 
MODELS = ["paddleocr-vl-1.6", "glm-ocr", "mistral-ocr-4-1"]
SAMPLE = Path("sample")
OUT = Path("results")
 
client = OpenParserClient()
 
for model in MODELS:
    (OUT / model).mkdir(parents=True, exist_ok=True)
    for pdf in sorted(SAMPLE.glob("*.pdf")):
        started = time.monotonic()
        parsed = client.parse.sync(
            {"ocr_model": model, "output_format": "openparser@1"},
            file=pdf,
        )
        seconds = time.monotonic() - started
        (OUT / model / f"{pdf.stem}.md").write_text(parsed.markdown)
        kinds = {}
        for element in parsed.elements:
            kinds[element.kind] = kinds.get(element.kind, 0) + 1
        print(json.dumps({"model": model, "file": pdf.name, "pages": len(parsed.pages),
                          "seconds": round(seconds, 1), "elements": kinds}))

Large PDFs can outlast the synchronous limit. In that case parse.sync returns a job reference; switch to client.parse.async_ and client.wait_for_job for those files.

What to look at

Element counts are a quick first signal. If one model finds 14 tables in your statements and another finds 3, open those pages first. Then diff the markdown side by side:

diff -y --width=200 results/paddleocr-vl-1.6/statement-03.md results/glm-ocr/statement-03.md | less

Past the plain text, look at table structure (merged cells, multi-line rows), reading order on multi-column pages, numbers with separators such as 1.440,00, and whether headers and footers end up mixed into the body.

If you extract fields downstream, compare the extraction too. Run /extract with your schema on each model's parse and count the fields that differ from your known-good values. Those are the errors your users would see.

Weigh the cost

Price per page differs by model. PaddleOCR-VL and GLM-OCR cost $1 per 1,000 pages on OpenParser; cloud OCR APIs list higher rates, from $1.73 to $17.25 per 1,000 pages in our catalog. If two models tie on your sample, the cheaper one is usually the right default, and you can route the hard document types to the stronger model.

Keep the script. When a new model lands in the catalog, add its id to MODELS and rerun it. See GLM-OCR vs PaddleOCR-VL for how the two open-weight models differ on paper, and the OCR router for the full list of model ids.