Reading Polish manuscripts (1689–1944) with vision-language models: an evaluation on 30 pages

1. Aim of the study and test material

In the previous part of this series we described the reading of 672 pages of the papers of Konstanty Kruszyński carried out with Gemini 3.8 Flash at a cost of 12 dollars. The accuracy of that reading remained an open question. Our own reference set (18 pages from the fonds) is awaiting correction. No open set of Polish manuscripts with transcriptions intended for the evaluation of automatic handwriting recognition exists: the available test sets cover only printed texts for Polish, and the HTR-United catalogue contains no Polish material.

We therefore compiled a small test set from material available online. A human-made transcription exists for every page:

We also downloaded 42 pages of German Kurrent script from the diary of the Dresden court (Zenodo). This material is relevant because 185 pages of the Kruszyński papers are written in Kurrent. The present test, however, concerns Polish texts only.

2. Measurement method

1. 1 · downloadScans and transcriptionsOmeka S and MediaWiki of BUW, the Wikisource API, Commons. 296 pages, of which 30 Polish pages were used in the test.
2. 2 · cleaningReference textRemoval of editorial markup, wiki templates, editor’s footnotes and homoglyphs (Cyrillic “а”, “о”).
3. 3 · readingPrompt consistent with production processingThe instruction from our processing pipeline: diplomatic transcription, [?] for uncertain words, a JSON schema.
4. 4 · 7 variantsGemini, Sol, SonnetEach variant reads each page once, without corrections and without retries.
5. 5 · resultPage CERThe proportion of characters requiring correction: strict (character by character) and normalised (ignoring letter case and punctuation).

CER (character error rate) is the number of character insertions, deletions and substitutions needed to transform the model’s reading into the reference text, divided by the length of the reference text. A value of 5% corresponds approximately to one incorrect word per line; at 20% the text has to be retyped from the original. We compare whole pages without splitting them into lines, because Wikisource merges lines into paragraphs.

2.1. Model variants tested

3. Reading accuracy

The principal result of the study: Gemini 3.8 Flash reads Polish manuscripts the most accurately and at the lowest cost. The low variant, used in production processing, costs less than one cent per page, and its error rate is almost equal to that of the high variant, which is ten times more expensive. Sonnet 5.5 ranks second. Both Sol variants achieve the weakest results, and Sol high is, moreover, the most expensive.

The ranking of the variants depends to some extent on the selection of pages. On the 21 pages read by all seven variants (toggle above), Sonnet’s result approaches that of Gemini low (10.9% versus 10.3%). This comparison, however, excludes the most difficult pages, on which Sonnet loses the most and Gemini most often interrupts the reading. This issue is discussed in Section 7.

Three observations follow from the chart:

• With legible handwriting the results are even. The “Connotacya” from Zamość is carefully written, and every model reaches 4–5% errors, mostly due to differences in convention (capital letters, “y” instead of “i”).
• Differences emerge with difficult handwriting. In the divorce records of 1822–23 the error rate is 12–13% for Gemini, 17% for Sonnet and 19–23% for Sol.
• The higher cost of Gemini’s high variant is justified only for forms. On printed KRWRiOP forms filled in by hand, the low variant reaches 18% and the high variant 10%. In the remaining groups the difference lies within the measurement error.

4. Results for individual pages

The rows of the map correspond to pages and the columns to variants. A darker cell indicates a higher error rate, and a hatched cell a reading interrupted by Gemini. Details are displayed when the cursor is placed over a cell.

5. Analysis of selected pages

Five pages on which the differences between the models are particularly pronounced are discussed below.

5.1. Comparison of four readings of a single passage

“Sekreta różne”, p. 9: a recipe for preventing smallpox scars. A wavy underline marks an error, and a dotted underline a difference solely in letter case, diacritics or punctuation. The reference text is given in superscript.

A considerable proportion of the marked differences are differences in convention. The models modernise „Swiezego” to „świeżego” and add the heading „J. M. J.”, which is omitted in the Wikisource transcription. Strict CER counts these differences as errors; normalised CER disregards them. For this reason we report both measures in the tables.

6. Cost and effect of reasoning

Models with a reasoning function consume tokens on hidden reasoning before generating the response; these tokens are billed as output. In the task of manuscript transcription, the additional cost translates into almost no gain in accuracy.

• During calibration (3 pages) Gemini high used on average 7.7k reasoning tokens per page; in the full test 23.8k, with a maximum of 87,915 tokens for a single page of the “Sekreta”. This corresponds to approximately 30 pages of text; the CER of that page fell from 9.0% to 8.0%.
• Sol high uses a similar number of reasoning tokens (12k), but the price per token is almost three times higher. In addition, a full-resolution image amounts to 7.4k input tokens in its case, compared with 3.1k for Gemini.
• Sonnet 5.5 uses few reasoning tokens (2.1k) but generates the longest responses, because it describes margins, ink colour and the hand in greater detail.

7. Interrupted readings in Gemini

The most significant result of the study does not concern the error rate. On some pages Gemini interrupts its response during generation, often in mid-sentence. We had already observed such blocks in the core of the fonds (2.8% of pages were processed in fallback mode); in the present test they proved to be document-dependent rather than random.

7.1. Point of interruption

Via OpenRouter, the text generated before the interruption is available (the direct Google API returns an empty response in such cases). The length of the bar corresponds to the whole page (2,285 characters of reference text), and the filled portion to the generated part.

7.2. Control trials

7.3. Hypotheses on the cause of interruptions

Practical conclusions:

• The same model offered through two services is subject to different filters. AI Studio (used in the project) blocked 3 of 60 readings, Vertex via OpenRouter 10 of 60.
• Blocks depend on the type of document, not on scan quality. Recipe books, manuals and records of blasphemy cases may be omitted entirely. Household recipes may also occur in the estate papers of Nawra.
• Retrying the request does not solve the problem. The solution is to use a different model: Sol and Sonnet did not block a single page. On the “Sekreta” pages that were read, Gemini achieves 9% CER and Sonnet 15%; the fallback model provides the text at the cost of lower accuracy.

8. Reliability of model self-assessment

Each reading includes a self-assessment of page legibility (good, medium, poor) and [?] markers for uncertain words. If these signals agree with the actual CER, they can be used to route to a second reading or to correction only those pages for which the model itself signals uncertainty.

Sonnet’s self-assessment is the best calibrated. Its three ratings delimit groups of pages with a CER of 4.7%, 16% and 26%. It places [?] markers on 25 of 30 pages, and reads pages without markers almost without error (4.8%). Gemini high’s rating distinguishes two levels (8% versus 20%), whereas Gemini low placed [?] markers on only 6 pages, including pages where it did make errors. This matters for correction: Gemini’s reading is more accurate, but it is harder to identify the passages in it that require checking.

Selecting the best of the five readings for each page would reduce the error from 10.8% (Gemini high) to 9.5%. Reading each page with several models therefore yields little benefit compared with Gemini alone. Using a second model is justified where the first returned no text.

9. Flex and Batch processing modes

Because of Sol’s high cost we attempted to use cheaper OpenAI modes, both with a 50% discount: Flex (requests served when computing capacity is available) and Batch (batch processing within 24 hours). Course of the trials:

1. 12 Oct, calibrationFlex mode works for a trial request (5 tokens).
2. 12 Oct, run38 of 54 requests are rejected with the code flex_unavailable (“Flex processing is temporarily unavailable”). Requests that were served took up to 21 minutes per page.
3. 12 Oct, BatchJob submission rejected with the message “Billing hard limit has been reached” (the account spending limit, independent of the limit in the script). The account was topped up.
4. 13 Oct, 9:59Submission of a batch job with 38 readings.
5. 11:25 → 12:18 → 13:21Progress: 1 of 38, then 17 of 38, unchanged after an hour. Batch mode does not make partial results available before the job is completed.
6. 13:25Job cancelled. OpenAI returns the 17 completed readings at the batch price; the remaining 21 were performed in standard mode in under an hour.

• „Mlijenia sztucznych Furfur”Sonnet’s reading of the heading „Sklejenia stłuczonych Farfur”, i.e. of a recipe for a porcelain adhesive.
• „nachalność”Gemini low’s reading (“impertinence”) in place of „na sameo Pana boga” in Zabistowski’s speech. Gemini high read „Maiestat”, which remains consistent with the sense of the sentence.
• Anno 1609The reference text of the “Connotacya” gives the year 1609 where the scan and historical sources indicate 1689. Reference transcriptions also contain errors.
• “0.5em 2em 200%”An error in our own processing pipeline: when wiki templates were cleaned, style parameters were extracted instead of text. After the fix, the CER of some Wikisource documents fell by over ten percentage points.
• missing commaGemini via OpenRouter four times returned JSON without a comma between objects. A repeated request returned a valid result.
• 300 secondsThe Gemini response time limit. It was exceeded once; the repeated reading of that page of the divorce records took 204 s and used 22.8k reasoning tokens.

10. Estimated cost of reading the entire fonds

The Sczaniecki Archive comprises 60,197 scans. We assume that approximately 69% of them contain text, as in the core (672 of 973), i.e. approximately 41,500 pages. The chart shows the cost of a single reading with each variant at the standard price and in batch mode (50% discount with all three providers).

The cost range exceeds a factor of thirty: from approximately 200 dollars (Gemini low, batch mode) to approximately 7,200 dollars (Sol high, standard price). The full processing pipeline (two readings, knowledge graph, name resolution) cost $0.018 per page in batch mode for the core, which amounts to approximately 740 dollars for the entire fonds. Gemini’s batch pricing changes in 2027 and will need to be recalculated before large-scale processing.

11. Recommended workflow

The test results suggest the following rule: the reading is performed by the least expensive model, and a second model is used only where the first returned no text or signals uncertainty.

We do not recommend switching to Sol or applying the high variant to all pages: in the test it increased the cost tenfold for an improvement of 0.3 percentage points.

12. Further work

1. Our own reference set. 18 pages from the fonds are awaiting correction in the browser. Only this set will make it possible to assess the reading of the handwriting of the Kruszyński papers rather than of the Zamość chronicle.
2. Kurrent. 42 pages from Dresden have been downloaded. German texts make up 30% of the pages in the core, and no measurement is yet available for them.
3. Safety settings. It should be verified whether explicitly setting safetySettings in the Gemini API reduces the number of interrupted readings of the “Sekreta”. OpenRouter does not pass these settings through.
4. Extension of the TranskriBUW set. The service contains more approved pages than were used in the test (the remaining pages of the “Connotacya”, the divorce records and KRWRiOP). Publishing results that include their text requires the consent of BUW, as the transcriptions have no specified licence.
5. Uncertainty indicator. It should be tested on the reference set whether disagreement between two Gemini readings predicts errors better than self-assessment. The result will determine whether a second reading is performed always or conditionally.

13. Other models and fine-tuning

The test covered three models from three providers. Models from various market segments qualify for the next round. A further open question is whether, instead of choosing an off-the-shelf model, one should fine-tune a model of one’s own on the basis of the corrections made.

13.1. Models for the next round of tests

• More capable models from the same providers: Claude Opus 5.5 and a higher-tier Gemini model, if one is released. They may read more accurately, but at a price several times higher they are justified only as a second reading of difficult pages.
• Cheaper variants: Claude Haiku 5.5 and cheaper GPT-6 models. In the role of a fallback model for Gemini blocks, the absence of refusals is what matters most; accuracy may be lower.
• Open vision-language models (e.g. the Qwen-VL family) and document OCR models. Running them on one’s own GPU infrastructure is inexpensive and not subject to content filters. Without fine-tuning, however, eighteenth-century Polish cursive remains beyond their capabilities.
• Classical HTR systems: Transkribus (the models “Polish M2” and “Polski Wieszcz”) and eScriptorium/Kraken. They recognise text line by line, are very inexpensive to operate and effective for a single hand in long series. They do not interpret layout or content and therefore cannot replace the knowledge-graph construction stage.

13.2. Methods for improving accuracy

A combination of these methods appears most advantageous. Gemini performs an inexpensive reading of the whole. Correctors in the “Archivist’s Desk” correct uncertain pages first. The corrections are used to fine-tune an open model or as in-prompt examples for a given hand. Once several thousand corrected pages have been accumulated, a model of our own can take over repetitive series (e.g. the Nawra estate registers), while the large models remain reserved for difficult cases and content analysis. The precondition is that corrections from the “Archivist’s Desk” are stored as image–text pairs, which should be provided for already at the present stage.

14. Limitations of the study

• 30 pages and approximately 15 hands constitute a small sample. Differences below 1–2 percentage points within a group lie within the measurement error.
• The reference transcriptions were made by volunteers according to rules of their own (e.g. regarding the transcription of crossings-out). Part of the models’ errors results from differences in convention. The prompt was not adapted to these rules.
• Sol performed its readings in three modes (Flex, Batch, standard). The mode does not affect accuracy, but the measured times do not reflect the model’s speed; costs are given at the standard price.
• Prices as of 11–13 October 2026. Each reading is a single one; a second reading and adjudication (used in our processing pipeline) reduce the error but were not the subject of this measurement.