Cross-Lingual Alignment Without Joint Training (arxiv.org)

A recent study titled "Cross-Lingual Alignment Without Joint Training" challenges the conventional belief that cross-lingual alignment in multilingual models relies entirely on joint training techniques. Researchers tested various strictly monolingual language models, including the Goldfish model families, and found that these models, even without sharing parameters or using mixed-language batches, can develop alignable representations. Notably, the alignment improves as the scale of data, model size, and linguistic proximity increase, suggesting that the inherent structure of language facilitates this alignment. This finding is significant for the AI/ML community as it opens new avenues for constructing multilingual systems without the need for complex joint training. The study demonstrated that a single Procrustes rotation can successfully align hidden states between different language models, indicating that the functional content can be transferred across languages. This discovery emphasizes the potential for modular multilingual systems that utilize distinct monolingual components, paving the way for more efficient and practical approaches in building AI language models.