Extracting publication metadata for CRIS: DOI registry, PDF parsers and language models

1. Aim and scope of the study

A CRIS (Current Research Information System) is an institutional database of research output, recording the authors, titles, venues and dates of publications. A record suitable for reporting and evaluation requires a dozen or so fields: title, authors, year, DOI, journal, publisher, ISSN, pages, language, keywords and abstract. In practice these fields are transcribed manually from the PDF or copied from a registry. With several thousand publications per institution each year, this work is costly and laborious.

The research question is: can large language models take over this task, and do they perform it better than the tools already available? Most existing evaluations cover English-language articles from a few large publishers. CRIS systems in Europe and Asia, however, also process Polish university journals, Japanese learned-society bulletins and Indonesian journals published on OJS. For this reason, the corpus was designed for diversity.

2. Method

2.1. Corpus

The corpus is stratified: exactly five open-access articles were randomly sampled for each cell (year × language × OpenAlex field):

Languages: English, Spanish, Portuguese, French, German, Italian, Polish, Indonesian, Korean, Chinese, Japanese and Arabic. The Arabic cell contained too few resolvable references, so the cross-lingual conclusions concern eleven languages.

2.2. Systems compared

Nine models were compared: gemini-3.1-flash, gemini-3.1-pro, gemma-4-31b, claude-sonnet-4.6, gpt-5.4, deepseek-v3.2, qwen-3.5-max, ministral-8b and qwen-3.5-7b. All receive the same prompt and the same set of pages. The set is trimmed individually for each document: it comprises the opening and closing pages of the article, so as to include every location in which the preliminary analysis found evidence of metadata. This reduces the number of pages by 37% on average without loss of information. Of importance for the subsequent analysis is the fact that the vision models receive the PDF, whereas the three text-only models (DeepSeek and both Qwen models) receive text extracted from the same pages.

2.3. Accuracy measure

A manually verified set of gold-standard values for all 960 papers is not yet available; it will be produced at the gold-set stage (96 papers, two annotators). At the screening stage, each field is compared with two independent references: metadata from the publisher's web page (A) and from OpenAlex (B). The similarity of a single value \(v\) to a reference \(r\) depends on the field type:

The exact fields are DOI, year, ISSN, pages and language; the text fields are title, journal, publisher and abstract. \(J\) denotes the Jaccard index, \(T(\cdot)\) the token set (split on whitespace), and \(\nu\) a surname normalisation function. The functions \(T\) and \(\nu\) are of particular importance for the analysis of the Japanese results (see Section 6.3). The principal measure is availability-conditional accuracy: only those cells (document × field) for which at least one reference contains a value are counted:

An empty answer counts as 0. Accuracy so defined decomposes exactly into two factors: how often the system answers (coverage \(\varphi\)) and the accuracy of the answers given:

This decomposition is used in the analysis of whether a model should be permitted to decline to answer (see Section 3.2).

3. Aggregate results

The result is unambiguous: for articles that have already been published, the most effective way to obtain metadata is not to extract them at all. When a document has a DOI registered in Crossref and OpenAlex, a single HTTP query yields an accuracy of 0.828, and in the fields that the registry contains, accuracy reaches 0.919.

This result, however, holds only conditionally. B0 is a lookup, not an extractor. For a preprint, a report, a dissertation or a newly deposited file not yet present in the registry, B0 returns zero. In such cases the models remain, and their accuracy of 0.61–0.69 is the only practicable option.

3.1. Cost and accuracy

3.2. Coverage and precision of answers

The table reveals a clear division. The Gemini and Gemma models fill 95–99% of fields. The Claude, GPT and DeepSeek models fill about 80–83% of fields, but the answers they do give are correct considerably more often (0.76 versus 0.68–0.70).

Which model is preferable depends on the cost of subsequent verification. For a librarian verifying records, an empty field means two seconds of work. An incorrect ISSN given without any signal of uncertainty is an error that may find its way into a report. Choosing a strategy therefore requires an answer to the question of whether the information is present in the PDF at all.

4. Presence of metadata in the document content

For each field, the frequency with which the reference value can be found in the text of the document itself was examined. The measure used is the recoverability index:

This changes the interpretation of the per-field results. A model that fails to return the correct ISSN in most cases has no access to that value in the document. For this reason, the registry has its advantage precisely in the ISSN, DOI and publisher fields: it holds these data even when the PDF does not. For the same reason, full-coverage models show a long tail of errors in fields absent from the document. For gemini-3.1-flash, 1,286 of 4,287 incorrect cells are values that did not appear in the PDF. For the conservative models (GPT, Claude) there are only about 470 such cells. Full coverage has a cost: an answer for a field absent from the document produces an error.

5. Accuracy by language of publication

From a practical standpoint, the most important result of the study is shown in Fig. 7.

On this basis one might conclude that LLMs perform worse with Asian languages. Before such a conclusion is drawn, however, the PDFs on which the models failed should be examined; this examination reveals several non-obvious causes of error.

6. Case analysis: causes of extraction errors

6.1. Corrupted PDF text layer

The article analysed, from the Japanese journal JunCture, concerns the reception of Doraemon in China. Its layout is simple and clear: a single column of text with notes in the margin. It poses no difficulty for a human reader or for the vision models, whereas the text models fail on it completely.

1 2 3

The same beginning of the page as output by two text-extraction tools:

ドラえもんと中国─メディアと記憶をめぐって
趙爽
キーワード：メディアによる記憶、…
はじめに
日本と中国の関係には複雑な歴史的・地政学的な
文脈の中で波があり、それは中国における日本映画
の公開状況にも現れている。

154
͸͡Ίʹ
日本ͱ中国ͷ関係ʹ͸複雑ͳ歴史的ɾ地政学的ͳ文脈ͷ中Ͱ
波͕͋Γɺ ͦΕ͸中国ʹ͓͚Δ日本映画ͷ公開状況ʹ΋
現Ε͍ͯΔɻ
… ʰSTAND BY ME υϥ͑΋Μʱ ͕公開͞Εɺ

The kanji were preserved, but all of the hiragana and katakana were converted into characters from the Greek, Coptic and IPA blocks. “はじめに” (introduction) is rendered as U+0378 U+0361 U+038A U+02B9 (including a code point to which Unicode assigns no character), and “ドラえもん” as “υ ϥ ͑ ΋ Μ”: a Greek upsilon, a Coptic letter, a combining diacritical mark, and so on. The cause is the ToUnicode table of the embedded font, which maps the kana glyph identifiers to incorrect code points. Better extraction tools (pdfminer, poppler) handle this case differently and return correct text, whereas pypdf applies the mapping literally.

Several phenomena occur simultaneously in this example:

• DeepSeek took a sentence from the text as the title: “日本アニメと中国の関係を検討する” (“to examine the relationship between Japanese animation and China”), in the garbled form in which it had been received.
• Qwen composed from the garbled characters a plausible-sounding title (“memory surrounding Doraemon and Japanese–Chinese relations”) that the article does not contain.
• Both text models assigned to the article the DOI of the cited work and the journal from the note.
• Gemini Flash, despite reading the page correctly, generated a non-existent DOI in the format of a Japanese institutional repository (prefix 10.15083), with the “quotation” field set to null. This is a typical hallucination, but one that can be detected automatically, since the model itself signals the absence of evidence.
• Claude gave the correct title, but as the quotation for the journal field returned “JunCure 15 | 研究論文 | 趣 真”, reading the author's name 趙 爽 as 趣 真. The value is correct; the evidence is wrong.
• Reference B (OpenAlex) gives the name of a repository database as the “journal”. The reference standard, too, is therefore not free of errors.

6.2. Prevalence of corrupted text layers

All CJK documents were analysed to determine what proportion of the non-ASCII characters in the pypdf text falls into Unicode blocks in which Japanese text cannot occur:

Result: 20 of 80 Japanese PDFs (25%) have a corrupted text layer. These are mainly journals of Japanese medical and engineering societies, together with a few Chinese and Korean journals. In a control sample of German and Polish PDFs, the isolated detections concern mathematical articles containing genuine Greek letters.

Practical conclusion: part of the gap to the detriment of Japanese reflects not a weakness of the models in that language but the choice of text-extraction library. A CRIS pipeline that passes text to a model should first verify that the text is valid. The simple test \(\mu(d)\) suffices for this purpose; if the test fails, the page image should be used.

6.3. Effect of metric tokenisation on the scoring of Japanese titles

The second problem affecting Japanese lies neither in the PDFs nor in the models, but in the metric. The table shows sample titles that received a score of 0:

All three answers are correct. The source of the problem is the function \(T(\cdot)\) in the definition of the metric, which splits text on spaces. A Spanish title with one typographical error loses one token out of twenty. A Japanese title constitutes a single “token”, so a difference of one character yields a Jaccard index of 0 and no containment. A script-aware normalisation is sufficient:

(Unicode NFKC normalisation, followed by the retention of letters and digits only) for the results to change markedly:

6.4. Ambiguity of the publication year

The next case concerns a Polish journal. For an article in Bromatologia i Chemia Toksykologiczna on polyphenols in non-alcoholic beers, all eight models unanimously gave the year of publication as 2021. All of them scored zero for this answer, because OpenAlex gives the year as 2023.

1 2 3 4

This is not an isolated case. Polish journals catching up on publication backlogs regularly issue, in year N+1 or N+2, a volume dated to year N. Teka Komisji Urbanistyki i Architektury of the Polish Academy of Sciences carries “TOM XLIX/2021” in its header, while OpenAlex gives the year 2023. Nowa Audiofonologia prints “2024; 13(4)” and “Opublikowano: 30.12.2024” (published), while the registry gives the year 2025. In each of these cases all models agree with one another, are correct with respect to the PDF and are incorrect with respect to the registry. For a CRIS and for parametric evaluation, in which the year determines the reporting period to which a publication is assigned, this discrepancy is of considerable importance.

6.5. Title selection in bilingual publications

A second problem concerning Polish journals is bilingualism, as illustrated by the two articles below.

1 2 3 4

1 2 3 4

In the first case, the models choose the visually prominent title, whereas the registry holds the title in the language of publication. In the second, the models faithfully transcribe the content of the page, and the reference differs from the printed version by one word. The reverse situation also occurs: in Psychiatria Polska the PDF contains the Polish title “Stres wojny…”, while the registry holds the English “The stress of war…”. For a CRIS, the correct answer is both titles with a language tag, which a schema with a single title field does not provide for.

7. Error taxonomy

The analysis yields three observations:

• Schema compliance has been resolved. Output structure is no longer a limiting factor. JSON is valid in 99.5% of cases, and the remainder consists mainly of empty API responses and the disordered output of the smallest model.
• Provenance error (1–4%) is more significant than hallucination: the value is correct, but the quotation indicating its source is wrong or missing (as with “趣 真” in Claude's answer). If downstream processing relies on the evidence_span field, this error is more serious than it appears.
• Normalisation errors (4–8% for the Gemini models, about 0% for Claude and GPT) are values that are semantically correct but not in canonical form: letter case in the DOI, a missing hyphen in the ISSN, a year followed by a full stop.

8. Implications for CRIS systems

1. Registry before model. Spending USD 250 on tokens to obtain a worse result than a free API provides is an inappropriate default choice.
2. A low-cost model is sufficient. In a schema-constrained task, the use of the most expensive models is not justified.
3. Verification of text before it is passed to the model. A simple test for garbled characters prevents the loss of 80% of accuracy on one in four Japanese PDFs. The alternative is to pass the page image directly to the model.
4. The ability to decline is an advantage. A model that leaves the ISSN field empty when the value is absent from the document performs better than one that always enters some value.
5. A CRIS schema should accommodate multilingualism and multiple dates. Single title and year fields do not reflect the practice of Polish and Japanese journals.
6. Cross-lingual metrics require script-aware normalisation. Otherwise, what is being compared is tokenisers rather than models.

9. Limitations and further work

The results presented come from the screening stage. References A and B are operational sources, not absolute truth; the cases from Bromatologia and RPEiS show that they, too, contain errors. Definitive values (true accuracy, the true hallucination rate and the division of the 26–34% of “inconsistencies” into model errors and reference errors) will be provided only by the gold set of 96 articles, annotated manually by two people with adjudication of disagreements. The set has been selected according to a rule registered before the results were examined and is awaiting annotation.

The analysis of the corrupted text layer (Fig. 8) and the recalculation of title accuracy after normalisation (Fig. 9) are post hoc analyses, carried out after the results had been examined. They were not part of the plan registered before the study and should be interpreted with this caveat.

The entire screening study (9 models × 960 documents) cost about USD 252 in API fees. All analyses presented in this post consist solely of reprocessing existing results and incurred no additional cost (USD 0).