The modernization of this archive is computational work — transcription, extraction, mapping, and connection at a scale no reading team could match. We hold that work to an archival standard: open, documented, reproducible, and reviewed by people.

Why open-weight models

Every production pipeline in the archive runs on open-weight, open-source models: models whose parameters we possess and run on infrastructure the University controls. This is a deliberate commitment, made early, and it has held through every stage of the project. The reasons are archival as much as technical.

Inspection. A catalog assertion should be traceable to a method that can be examined. With open weights, the exact model that read a page or proposed a case name is a fixed, versioned artifact — not a service that changes silently behind an API.

Reproducibility. Scholarship built on this archive should be checkable. Because the models are open and the pipelines documented, the derived layers — transcriptions, extractions, embeddings — can be regenerated and verified by others, now or in twenty years.

Custody. The documents never leave infrastructure under university control. Nothing in this collection is sent to a commercial model provider; no third party's terms of service stand between the archive and its own materials.

Independence. An archive plans in decades. Open weights mean no capability we depend on can be deprecated, repriced, or withdrawn by a vendor. The computational layer can always be rebuilt — see Preservation first, below.

Closed commercial models play no role in production. Where we used them at all, it was as comparators during development — a benchmark to beat, not a dependency to keep.

The pipelines

  • Transcription of the printed papers. Open-weight vision-language models (the Kimi and Qwen model families) read each page image and produce layout-aware text: headings, body, and marginalia distinguished, page furniture identified, the reading order of eighteenth-century typography respected. Transcription quality is evaluated against human-prepared reference pages, and the output is labelled as machine transcription wherever the site displays it.
  • Handwritten materials. Manuscript items follow a separate path through kraken, an open-source handwritten-text-recognition engine developed for historical scripts, with models tuned to period hands.
  • Extraction. From the transcriptions, extraction pipelines propose structured records: case names and parties, document types and dates, the people the papers name and the roles they play, cataloger-style summaries, and subject terms drawn from the archive's controlled vocabularies. Every proposal carries its evidence — the passages that support it — so a reviewer can judge it without leaving the queue.
  • Geography. Place references are resolved against a gazetteer of Scotland's historic geography: coordinates, National Grid references, and the historic county, parish, and registration-district boundaries published by the UK Data Service, presented over National Library of Scotland historic map tiles. Point-in-polygon assignment places each located site in its historic county; ambiguous names — Scotland's many Newtons, its several Aberdours — go to curators, not to guesses.
  • Semantic index. Multilingual open-weight embeddings (BGE-M3) are computed over the corpus, powering similar-case suggestions and the MCP research endpoint.

Calibrated against the catalog

Before any pipeline ran at scale, it was tested against the strongest ground truth available: the records our catalogers had already created by hand over years of work. Thousands of professionally cataloged documents serve as a standing evaluation set — extraction proposals are scored against them field by field, and pipelines are revised until their agreement with the human record justifies deployment. The comparison is ongoing: as catalogers review new proposals, their decisions extend the reference set and sharpen the measurement.

People decide

No machine output enters the catalog unreviewed. The pipelines propose; the Law Library's catalogers decide, working from review queues that present each proposal with its evidence and a one-click path to confirm, correct, or reject. The division of labour is deliberate: machines supply reach — every page read, every name indexed — and people supply judgment. Where the archive displays machine output directly, as with uncorrected transcription, it is labelled as such.

Preservation first

Before any processing began, every source image was secured in a versioned cloud dark archive — 189,205 items — independent of any processing system, model, or vendor. The computational layer is designed to be disposable in the best sense: transcriptions, extractions, and indexes can all be regenerated from the preserved evidence as methods improve. The archive's permanence attaches to the documents and their identifiers, not to any particular season's software.

Limits

Machine transcription of eighteenth-century print is good and improving, but not perfect; long-s confusions, tight bindings, and worn type leave errors, which is why the page image is always primary. Extraction proposes with confidence scores, not certainty. Geographic resolution covers roughly 70% of place references today. We would rather state these limits than let the interface imply a precision the methods don't have — and each limit narrows as review and reprocessing continue.