Language Support for Marginalia Search (www.marginalia.nu)

A pilot to add non‑English search to Marginalia is now live for testing, with experimental support for German, French and Swedish. The team integrated language identification (fastText) into the existing pipeline—HTML text extraction → sentence splitting → Unicode normalization/lowercasing → stemming and POS tagging → keyword extraction → TF‑IDF and grammar‑pattern heuristics → hashing of keywords—and parametrized language behavior via XML “language definition” objects (POS‑pattern ngrams). Practical changes include a bitmask optimization for performant POS‑pattern matching and a human‑facing test tool that emits annotated pipeline outputs (human inspection remains essential because automated suites miss many linguistic nuances). Crucially, TF‑IDF suffers from data sparsity since the index is overwhelmingly English, while BM25 ranking remains robust to the imbalance. Several language‑specific problems were encountered and addressed: diacritic/normalization decisions (e.g., Swedish/ German u vs ü), the need for lemmatization vs stemming, and hash‑based keyword IDs which risk collisions as the lexicon grows. Rather than one huge multilingual index, the team chose separate language‑specific keyword→document mappings with per‑language journals to avoid cross‑language noise and lexicon bloat; this reduces query ambiguity at the cost of needing to specify language. The feature works functionally but evaluation is constrained by a very small non‑English corpus (en: ~113M docs; de: 7.6M; fr: 4.9M; sv: 1.0M), reflecting crawl bias from English link graphs and indicating that scaling and targeted crawling will be necessary for reliable multilingual retrieval.