Computational Methods | SCOS Archive

The modernization of this archive is computational work — transcription, extraction, mapping, and connection at a scale no reading team could match. We hold that work to an archival standard: open, documented, reproducible, and reviewed by people.

Why open-weight models

Every production pipeline in the archive runs on open-weight, open-source models: models whose parameters we possess and run on infrastructure the University controls. This is a deliberate commitment, made early, and it has held through every stage of the project. The reasons are archival as much as technical.

Inspection. A catalog assertion should be traceable to a method that can be examined. With open weights, the exact model that read a page or proposed a case name is a fixed, versioned artifact — not a service that changes silently behind an API.

Reproducibility. Scholarship built on this archive should be checkable. Because the models are open and the pipelines documented, the derived layers — transcriptions, extractions, embeddings — can be regenerated and verified by others, now or in twenty years.

Custody. The documents never leave infrastructure under university control. Nothing in this collection is sent to a commercial model provider; no third party's terms of service stand between the archive and its own materials.

Independence. An archive plans in decades. Open weights mean no capability we depend on can be deprecated, repriced, or withdrawn by a vendor. The computational layer can always be rebuilt — see Preservation first, below.

Closed commercial models play no role in production. Where we used them at all, it was as comparators during development — a benchmark to beat, not a dependency to keep.

The pipelines

Transcription of the printed papers. Open-weight vision-language models (the Kimi and Qwen model families) read each page image and produce layout-aware text: headings, body, and marginalia distinguished, page furniture identified, the reading order of eighteenth-century typography respected. Transcription quality is evaluated against human-prepared reference pages, and the output is labelled as machine transcription wherever the site displays it.
Handwritten materials. Manuscript items follow a separate path through kraken, an open-source handwritten-text-recognition engine developed for historical scripts, with models tuned to period hands.
Extraction. From the transcriptions, extraction pipelines propose structured records: case names and parties, document types and dates, the people the papers name and the roles they play, cataloger-style summaries, and subject terms drawn from the archive's controlled vocabularies. Every proposal carries its evidence — the passages that support it — so a reviewer can judge it without leaving the queue.
Geography. Place references are resolved against a gazetteer of Scotland's historic geography: coordinates, National Grid references, and the historic county, parish, and registration-district boundaries published by the UK Data Service, presented over National Library of Scotland historic map tiles. Point-in-polygon assignment places each located site in its historic county; ambiguous names — Scotland's many Newtons, its several Aberdours — go to curators, not to guesses.
Semantic index. Multilingual open-weight embeddings (BGE-M3) are computed over the corpus, powering similar-case suggestions and the MCP research endpoint.

Calibrated against the catalog

Before any pipeline ran at scale, it was tested against the strongest ground truth available: the records our catalogers had already created by hand over years of work. Thousands of professionally cataloged documents serve as a standing evaluation set — extraction proposals are scored against them field by field, and pipelines are revised until their agreement with the human record justifies deployment. The comparison is ongoing: as catalogers review new proposals, their decisions extend the reference set and sharpen the measurement.

People decide

No machine output enters the catalog unreviewed. The pipelines propose; the Law Library's catalogers decide, working from review queues that present each proposal with its evidence and a one-click path to confirm, correct, or reject. The division of labour is deliberate: machines supply reach — every page read, every name indexed — and people supply judgment. Where the archive displays machine output directly, as with uncorrected transcription, it is labelled as such.

Preservation first

Before any processing began, every source image was secured in a versioned cloud dark archive — 189,205 items — independent of any processing system, model, or vendor. The computational layer is designed to be disposable in the best sense: transcriptions, extractions, and indexes can all be regenerated from the preserved evidence as methods improve. The archive's permanence attaches to the documents and their identifiers, not to any particular season's software.

Limits, honestly stated

Machine transcription of eighteenth-century print is good and improving, but not perfect; long-s confusions, tight bindings, and worn type leave errors, which is why the page image is always primary. Extraction proposes with confidence scores, not certainty. Geographic resolution covers roughly 70% of place references today. We would rather state these limits than let the interface imply a precision the methods don't have — and each limit narrows as review and reprocessing continue.

Stage by stage

Every machine-made value on this site carries a badge, and every badge links here — to the stage that produced that particular value. Each entry below names the model, where it ran, why it was chosen, and the prompt it was given. Where a stage uses no model at all, it says so.

Extraction — the catalogue record

Reads a document's corrected text and proposes the structured record: parties, document type, dates, printed title, appendix contents. Its output becomes the title you see and feeds nearly every other stage.

Model: Kimi K2.5 (open weights)
Where it ran: University of Virginia Research Computing GenAI gateway, called from sharded SLURM array jobs on the HPC cluster
Version: extraction-kimi-v2 on 11,329 documents; extraction-kimi-v1 on an earlier 2,832
Why this model: Chosen in a 2026 bake-off against other open-weight candidates. The project's production rule is open weights only; closed models appear nowhere in this pipeline except as benchmarks during development.

The request is streamed rather than awaited, because the gateway closes connections at sixty seconds and the model's reasoning pass runs longer than that. Retries are capped and a circuit breaker stops the run if the gateway itself is failing, so a bad afternoon upstream cannot quietly fill the catalogue with half-answers.

Show the prompt

You are a Special Collections cataloger for the
University of Virginia Law Library, working on the Scottish Court of
Session Papers (SCOS) collection. You extract structured catalog
metadata from 18c court papers — petitions, memorials, answers,
informations.

ABSOLUTE GUARDRAIL — no historical-knowledge bias.
You may possess general knowledge about famous 18c Scottish legal
matters (the Crail elections, the Douglas Cause, particular
advocates, particular families). You MUST NOT use any of that
knowledge to fill in fields. Every party name, date, place, and
title you emit MUST appear (or be a direct, mechanical
normalization of something that appears) in the OCR text
provided to you.

Excerpt. The full prompt, with the appendix-enumeration pass that follows it, is in the pipeline source.

Show the earlier version's instruction (v1 — it wrote 2,678 of the titles on this site)

The system prompt above was the same in v1. What differed was the instruction governing document type, which v1 stated as a bare list of permitted values:

"document_type": "<one of: Petition | Memorial | Information | Answers |
State of the Process | Petition and Complaint | Replies | Reclaiming Petition |
Bill of Suspension | Bill of Advocation | Complainers' Case | Respondents' Case |
other (with type description)>"

v2 replaced that with the form-over-content rule — classify by what kind of document this is, not what it is about, and treat a paper whose own printed title begins "APPENDIX" as an appendix regardless of the matter it contains. A document typed under v1 was therefore typed by its contents.

The title itself is then composed by rule, not by the model: a fixed template over the extracted parties, with the document's own identifier as a fallback. The model supplies the parts; the sentence is assembled deterministically.

Summaries — docket description and annotator note

Writes the two short cataloguer-style summaries shown on a document page.

Model: Qwen3-VL 8B Instruct (open weights)
Where it ran: vLLM on a single A100 inside UVA's HPC cluster — the model is served on-site, not through any gateway or external API
Version: summary-qwen3vl-8b-v6 on 12,817 documents; an earlier -v1 on 585
Why this model: Kimi was tried first and ruled out for this task: three attempts out of three produced eight to nine thousand characters of reasoning and no summary before the gateway timed out. Qwen3-VL returns a conventional summary in two to five seconds on one GPU, with no gateway in the path at all.

The filing date is parsed from the document's identifier and handed to the model as an input rather than left for it to read out of the text. That single change removed two of seven date errors in the first evaluation pass.

Show the prompt

You are a cataloger annotating 18th-century Scottish Court of Session
Papers (the SCOS collection). Your job is to write two short cataloger
summaries for each document, matching the existing house style.

## OUTPUT FORMAT — STRICT JSON
Return exactly this JSON object, no preamble, no markdown fences:
{ "docket_description": "...", "annotator_note": "..." }

Both fields are SHORT (1-3 sentences). Both follow strict conventions
described below.

## SCOTS LEGAL REGISTER — preserve verbatim, do NOT translate
- "interlocutor" (not "ruling" or "decision")
- "assoilzie / assoilzied" (absolved)

Excerpt. The conventions section that follows runs to several hundred lines and encodes the house style the Library's cataloguers already use.

Show the earlier version's prompt (v1 — it wrote 585 of the summaries on this site)

v1 gave the same task in the same voice, and asked once. It did not enforce lengths, did not require a non-empty docket description, and had no recovery path: a malformed or over-long answer was simply what you got. v6 added strict length limits, made the docket description mandatory, and introduced two further prompts — one that re-asks when an answer comes back malformed, one that compacts an answer that comes back too long:

You summarize Scottish Court of Session papers for a
library catalog. Return exactly one JSON object with non-empty
docket_description and annotator_note strings. The docket is a concise
procedural description of 30-280 characters in one sentence; the note states
the party's substantive position in 50-400 characters and 1-3 sentences.
Use only supplied evidence. Do not name counsel or invent a date. These length
limits are strict.

The recovery prompt added in v6, shown for contrast; v1 had no equivalent.

People, subjects and places named in a document

The marked entries in a document's People, Subjects and Places lists. This stage adds no judgement of its own: it gathers what three earlier stages resolved and keeps only the confident, already-matched results.

Models: Qwen3-VL 8B reads each page for candidate names and legal subjects; matching those candidates to the archive's existing people and vocabularies is done by rule and by embedding similarity (all-MiniLM-L6-v2), not by a language model
Where it ran: HPC cluster — the same on-site vLLM server as the summaries
What is withheld: Anything below a confidence floor, anything flagged for review, and anything the matcher could not tie to a record already in the archive. Uncertain matches stay in the curation queues rather than appearing on a public page.

Place names are matched against a gazetteer of Scotland's historic geography at a deliberately high similarity floor, because short toponyms defeat loose matching — Kilmarnock and Kilmaurs are barely distinguishable to a general-purpose embedding. Ambiguous names go to curators.

Show the prompt used to find subjects

You are a cataloger annotating 18th-century Scottish Court
of Session Papers (SCOS). Read one page of OCR-corrected text and identify
every legal subject discussed on the page.

A "legal subject" is a concept in the law of the case — e.g.:
  - Substantive areas: Bankruptcy, Debt, Aliment, Defamation, Insurance,
    Property, Election contest, Marriage, Succession, Teinds, Thirlage
  - Legal instruments: Bond, Tack, Disposition, Sasine, Bill of Exchange
  - Legal actions/processes: Arrestment, Reduction, Declarator, Cessio

DO NOT include:
  - Document types (Petition, Memorial, Answers) — those are procedural
  - Named persons, places, lands, organizations
  - Modern paraphrases — preserve period-authentic Scots law terminology
    (e.g. "Thirlage", "Aliment" rather than translations)

GUARDRAILS:
  - Every `evidence` quote MUST appear in the page text verbatim
  - If the page contains no legal subjects, return an empty array

Gathering a case's people, places and subjects

A case is a container for its documents, and this stage lifts what those documents name up to the case record.

Model: None. This stage is arithmetic over results the document stages already produced.
What it will not do: It leaves alone any case a cataloguer had already described. Machine description fills gaps; it does not overwrite judgement.

That rule had a consequence worth stating: cases catalogued by hand years ago kept only their hand-written entries while their documents accumulated machine-identified ones. Those cases have since had the missing entries added alongside — never in place of — the cataloguer's, which is why a case page can show marked and unmarked entries together. The unmarked ones are a person's work.

Places added to a case from its documents

Model: None — a set operation over the documents' already-resolved places.
Guards: Only published documents contribute. A place already named on the case, under any spelling or any duplicate record within two kilometres, is not added twice. Nothing added is ever marked the principal place of a case.

Subjects added to a case from its documents

Model: None — the same set operation, over subjects.
Guards: Six subject tags in total per case, cataloguer tags included; where there is room, the subjects most of the case's documents discuss are preferred. A case already at six gains nothing, so a cataloguer's tags are never displaced.

Similar cases

The suggestions at the foot of a case page. These are computed by distance in a vector space, not by anyone's view of what resembles what.

Model: BGE-M3 (open weights), 1024-dimension multilingual embeddings
Where it ran: vLLM on an on-site GPU, started on demand for indexing passes and shut down afterwards; there is no always-on public vector service
Method: Each document is embedded from its summary text. A case's position is the mean of its documents' vectors, normalised; the nearest other cases by cosine similarity are kept — at most five, and none below a similarity of 0.55, under which "similar" is noise.
Whose documents count: Only documents confirmed as belonging to the case. Machine proposals do not shape the similarity space until a person has confirmed them.
Why this model: Selected in an embedding bake-off against commercial alternatives. It handles Latin passages and Scots legal vocabulary better than the compared service, costs nothing per query, and — the archival argument — its weights are frozen, so the same text embeds identically in twenty years.

Case abstracts

Model: Kimi K2.5 (open weights)
Where it ran: UVA Research Computing GenAI gateway
Why this model: Decided on a thirty-case evaluation anchored to 457 abstracts written by the Library's cataloguers. Kimi produced one abstract with an unsupported claim; the open-weight alternative inverted party roles in seven; a commercial comparator fabricated content in twenty.
What it reads: The documents' docket descriptions — preferring a cataloguer's where one exists — never the raw page text.

Show the prompt

You are a legal-history cataloger writing case abstracts for
18th-century Scottish Court of Session cases. You are given the case name
and a chronological list of the case's documents, each with a one-sentence
docket description.

Write a single case abstract: 2-4 sentences of plain narrative prose.
Conventions (match the existing human-written abstracts):
 - Name the parties and their roles (pursuer/defender etc.) early.
 - State what the dispute was about and the central legal question.
 - If the documents show how the litigation developed (answers, proofs,
   petitions against interlocutors), sketch that arc briefly.
 - Past tense, third person, no citations, no document-by-document list,
   no editorializing, no "this case concerns" boilerplate.
 - Never invent outcomes or facts not present in the docket descriptions.
Return ONLY the abstract text.

Document type

Model: None at this step. The type is a lookup from the extraction stage's answer to the archive's controlled list.
On failure: A type that does not match the list is counted and reported, never guessed into the nearest neighbour.

Document date

Model: None. The extracted date is accepted only in strict calendar form, and impossible years are discarded.
Precedence: A date entered by a cataloguer is never overwritten.

Matching files to records

Model: None — string normalisation, including known digitisation typos in the original filenames.
Ambiguity: Only one-to-one matches are written. Anything ambiguous is set aside for review rather than resolved by preference.

Maps, plans and diagrams

Model: A text-density filter finds candidate pages; Qwen3-VL then looks at each candidate and classifies it. Attaching the result to the right document is deterministic.
Scope: Visual plates only — maps, plans and diagrams. Tables are deliberately excluded.

When a badge disappears

A badge disappears by itself when a cataloguer edits the value it describes. Nothing is marked as machine-made once a person has taken it in hand, and on lists that mix the two — a case's places, its subjects — the mark sits on individual entries rather than the whole list, so a cataloguer's entry is never labelled as a machine's.