Local RAG for Scanned PDFs with Ollama: OCR Proof, Citations, and Retrieval Tests

A reviewer asks a local assistant for the termination date in a scanned supply contract. The answer arrives as a clean sentence with a page reference, and nobody notices that the OCR engine read a 3 as an 8 on that page. Nothing in the chat window signals the error, which is why trust in such a system has to be built link by link, from page image to citation.

Scanned pages turn RAG into an evidence problem

A scanned PDF is just a picture of a page. It has no text inside it, so every word the model sees was rebuilt by OCR (optical character recognition, the software that turns images into text). Some of those words may be wrong, missing or in the wrong order. Scans in several languages add more ways to fail: the language can be misdetected, the alphabet can confuse the OCR, the layout can be misread, and the question may be asked in a different language from the page.

An Ollama RAG OCR PDF workflow has to prove an unbroken chain: the page image becomes readable text, the text is cut into chunks that remember their page, the right chunks are found, and the answer cites the page that supports it. A normal RAG setup starts by chunking text. A scanned-document setup starts by checking the text is any good. RAG (retrieval-augmented generation: the model answers from passages it looks up) was framed by Lewis et al. in 2020 on the assumption that those passages are trustworthy text. On a scan, that assumption is the first thing to test.

The deliverable is a proof artifact rather than a chat window: a repeatable test pack containing image-only pages, mixed layouts, and every expected language. It exposes failures before contract archives, engineering drawings, or compliance review.

The evidence path from pixels to an answer

Start by deciding which tool owns which step. Ollama runs the local model: it writes the answer and, where it fits, turns text into searchable vectors (embeddings). It does not reliably read page images inside a retrieval pipeline. Checking the PDF, turning pages into images, OCR, rebuilding the layout, building the search index and assembling citations are separate jobs for separate tools.

A defensible local flow has seven links:

1. Inspect the PDF: classify each page as native text, image-only, or mixed; one document can contain all three.
2. Render image pages consistently: retain page number, rotation, dimensions, source fingerprint, and rendering settings in the manifest.
3. Run OCR with language and layout settings: store raw output, confidence signals, and bounding boxes where available.
4. Apply quality gates: route weak coverage, suspect language detection, broken reading order, or missing named entities.
5. Build citation-aware chunks: retain document ID, page number, excerpt, and location data. A chunk without lineage is text, not evidence.
6. Retrieve with dense and lexical signals: semantic search supports paraphrases; lexical search helps match rare names, identifiers, and clause numbers, though it cannot on its own recover values already distorted by OCR.
7. Generate only from retrieved evidence: Ollama receives chunks and citation labels, then cites pages or states that evidence is insufficient.

This separation shows whether a wrong date vanished in OCR, chunking, ranking, or generation; a chat interface hides these defects.

OCR gates must run before indexing

Most quiet damage is either caught at the quality check or not caught at all. A misplaced decimal point, a lost "not" or a misread surname changes the meaning. The OCR engine can be very confident and still scramble table rows or read two columns as one.

So check every page before it goes into the index. Look at how much text it produced, which language or alphabet it detected, whether the page was rotated, how confident the engine was, whether characters are missing, and whether the output is suspiciously short. Some PDFs mix real text with scanned pages. On those, compare the OCR result with the text already in the file. If they disagree, flag the page for review: the stored text may be old, cut off or unrelated to the picture.

For the proof artifact, create verified reference transcriptions for material headings, dates, names, reference numbers, amounts, table headers, and footnotes. Calculate Character Error Rate:

CER = character edits needed to match reference text / characters in reference text

CER tells you when OCR gets worse, but a good CER does not mean the text is ready for search. Also check three things by eye: identifiers with hyphens stay whole, numbers in tables stay next to their labels, and footnotes stay next to the claims they qualify. A score for the whole page can hide all three failures.

A hypothetical payment schedule shows how. The reference transcription has 1,200 characters, and the OCR output needs 18 edits to match it: a CER of 1.5%, which would pass most generic thresholds. But 3 of those 18 edits fall inside the 40 characters that hold due dates and amounts. On the fields a reviewer will actually ask about, the CER is 7.5%, and one of the errors turns 13 March into 18 March. Compute CER separately for critical fields, and for amounts, dates and identifiers set the tolerance at zero errors on the pilot pack.

Detect the language for each page, or even each block of text. One label for the whole file is too rough. A single bilingual page may hold English body text, a French legal notice and a Cyrillic stamp. The OCR setup must cover every alphabet you expect. The embedding model (the part that turns text into searchable vectors) must also match questions and pages across languages. During the pilot, check by eye any right-to-left text, accents, vertical stamps and product codes that mix Latin letters with digits.

Do not use one pass score for every document. A brochure and a payment schedule need different tolerance. Pages with low confidence should be reviewed, kept out of the evidence the assistant can answer from, or used only with answers marked unverified. If you index bad text, a problem you already knew about becomes one nobody can trace.

Page lineage makes citations inspectable

A citation must let a reader open the source page and see the supporting text without hunting for the paragraph. A file name alone fails that test, and fails it worse once revised copies pile up.

Attach to every chunk:

• document_id and original-file content fingerprint
• title, revision label, and ingestion timestamp
• physical PDF page number and rendered-image reference
• chunk ID, extracted text, and adjacent context when needed
• OCR settings, language label, and quality status
• bounding boxes or line references when available

Display a citation such as Policy Manual, page 14 with a short supporting excerpt. Page previews or highlighted regions help with dense tables and image-heavy pages, where extracted text cannot prove correct reading.

Formatting alone does not ensure faithfulness. Tests must confirm that the cited page actually supports the claim, because shared vocabulary is easy to match by accident: a model can treat an exception as the rule or cite a neighboring chunk sharing a heading. For high-stakes sets, require claim-level citations for dates, amounts, obligations, and named parties rather than one citation for a long paragraph.

Retrieval tests expose plausible failures

Score evidence selection before answer fluency: if the right page never reaches the model, no prompt will fix the answer. Define queries with known expected pages and preserve them as corpus, OCR engine, index, prompt, or model changes. Include:

• direct lookup questions with one exact-answer page
• paraphrased semantic-match questions
• cross-page questions requiring two citations
• unanswerable questions requiring abstention rather than invention

Add queries in every source language and, if supported, cross-language queries. A German query about an English scan tests a different path from an English query about an English scan. Add adversarial pairs: repeated headings, near-identical names, conflicting revisions, and a value appearing in both a table and footnote.

Run the arithmetic on a frozen pack of 60 queries: 40 direct lookups with one expected page each, 10 cross-page questions with two pages each, and 10 unanswerable questions. That gives 60 expected evidence pages. If the top 5 results contain 51 of them, Evidence Recall@5 is 85%. Tracing the 9 misses matters more than the headline: if 5 come from OCR damage, 3 from chunks that split a table and 1 from ranking, the next week belongs to OCR, and the embedding model can wait. If the system declines only 7 of the 10 unanswerable questions, abstention correctness is 70%, and those three confident answers without evidence should block release before any recall figure does.

Fluent wrong answers have owners. Whoever deploys the assistant answers for what it tells a reviewer, and running the model on local hardware changes where the data lives, not who carries the consequences.

Track answer accuracy separately. A model may answer a direct lookup from prior knowledge or a misleading partial chunk; that does not show RAG worked. Review retrieved page, cited excerpt, and answer wording together.

Use deterministic expected-page checks before an LLM evaluator. An evaluator can flag phrasing defects but cannot be the only judge that page 37 is correct. Human review remains necessary for tables, handwriting, stamps, and layout-dependent questions.

A local proof artifact teams can rerun

A test that cannot be rerun after the next model swap is an anecdote. Keep the pilot narrow: choose a bounded set and freeze a test pack before changing prompts or models. Store source PDFs, rendered pages, OCR output, chunk manifests, index version, test queries, expected citations, and results in version control or an immutable artifact store.

A practical runbook:

1. Ingest PDFs; record hashes, page counts, detected text layers, and access classification.
2. Render selected pages; verify orientation, resolution, and cropped margins.
3. OCR the corpus; send gate failures to review.
4. Create page-aware chunks; keep tables intact where possible, because row splitting breaks label-value relationships.
5. Index with dense retrieval and lexical fallback; retain revision, language, and permission filters.
6. Send top evidence to Ollama with strict instructions to cite retrieved pages, quote only supplied evidence, and abstain without support.
7. Run the frozen suite and review every failed evidence query before changing more than one variable.

Show successes and failures. A correct-answer screenshot proves little; retain the page, OCR text, selected chunks, citation, expected evidence, and failure reason. Product, legal, and security teams can then argue over the same object instead of over screenshots.

Component choices follow document failure modes

Once the frozen pack exists, component selection becomes empirical. Choose OCR, retrieval, and generation by testing them on the documents that fail; leaderboards say little about your scans. Clean single-language reports may need basic extraction and semantic retrieval; archival scans, forms, engineering pages, and multilingual agreements need stronger layout, language, and citation handling.

No answering model can fix a wrong transcription. Giving it more context can make things worse: it may blend chunks that contradict each other. Fetching more chunks may find the right page more often, but it also buries the useful part in noise. Liu et al. documented why in "Lost in the Middle" (2023): models use information at the start and end of a long context much better than information in the middle, so the tenth chunk counts for less than the first. Tune the number of chunks (k), chunk size, overlap and reranking against the frozen pack, not by feel.

Local inference still requires role-based access, encrypted storage, audit logs, retention rules, and copied-export controls. The index, OCR cache, and answer logs can expose content; treat them as part of the document system and give them the same controls.

Why not send page images to a vision model?

A fair question is whether this whole pipeline is already obsolete. Multimodal models read page images directly. Skip OCR, skip layout reconstruction, hand the model the rendered page and the question, and the transcription errors described above never enter an index. Fewer moving parts, fewer places to break.

For a small, low-stakes set, the vision model is often the better choice. Thirty pages of meeting notes do not need a seven-link evidence chain. It also earns its place as a second reader: when the OCR text and a reviewer disagree about a value, asking a vision model to read the same crop is a cheap tiebreak.

Where it falls short is audit. A vision model's reading leaves no character-level transcript to diff against a reference, no stable bounding boxes to highlight in a citation, and no record that the same page will be read the same way after the next model update. Re-reading every candidate page per query also multiplies local compute: 20 page images per question costs far more than 20 text chunks. So keep OCR plus lineage as the system of record, and use a vision model as a verifier on pages that fail the quality gates or on exact values a user disputes.

Pilot failures often start in mixed documents

Mixed PDFs are harder than image-only scans because they look healthy at file level. Selectable text can coexist with image-only annexes, signatures, charts, or amendments. Skipping OCR after finding any text layer creates searchable documents with invisible pages.

Revision drift also misleads. If a revised PDF changes pagination, old page-only citations become wrong. Bind citations to a content fingerprint and revision label, then invalidate or re-ingest affected chunks. Duplicate files require the same discipline, since two files with the same title can hold different contents.

OCR artifacts create retrieval traps. A number read as a letter may still match semantically, while an identifier with one wrong character can disappear from lexical search. Keep rendered pages available for disputed facts. For exact-value questions, answer policy should favor visual verification over fluent synthesis.

When the pilot is ready to widen

Start with scans the system handles least well: image-only pages, multilingual sections, tables, stamps, repeated revisions, and deliberately unanswerable questions. Build quality gates before the vector index, page lineage before the chat prompt, and retrieval evaluation before accuracy claims.

The release gate follows from the runbook. Widen access only when every direct-lookup query in the frozen suite retrieves its expected page, every unanswerable query is declined, and each remaining failure has a recorded cause in OCR, chunking, ranking, or generation. Until then, label Ollama's answers as drafts that point to a page someone still has to open.