Millstone: Deterministic engine that characterizes undeciphered corpora (github.com)

A new deterministic engine named Millstone has been unveiled, offering a unique approach to characterizing undeciphered corpora through symbolic AI, distinct from typical large language models (LLMs) or statistical analyzers. This innovative tool employs a mathematical framework for grammar, utilizing a hierarchical knowledge graph, a constraint engine, and validation through the generation of negatives. Millstone's capabilities are highlighted through its application to the enigmatic Voynich Manuscript—a famously undeciphered text—but it also functions effectively across multiple languages and synthetic grammars, thus expanding its potential utility. The significance of Millstone lies in its ability to tackle closed corpora without ground truth, allowing researchers to characterize grammatical structures mathematically rather than relying on conventional learning methods. Prior to its deployment on the Voynich, Millstone was calibrated using well-defined corpora, which helps ensure its reliability. It features a modular architecture that separates various components for flexibility, including modules for data ingestion, storage visualization, and hypothesis evaluation. Ultimately, Millstone opens up new avenues for linguistic analysis, providing tools to uncover hidden structures within complex and previously indecipherable texts.