RNA language models can generalize well on structure prediction tasks (www.nature.com)

Researchers have introduced the RiboNucleic Acid Language Model (RiNALMo), the largest RNA language model to date, boasting 650 million parameters and pre-trained on 36 million non-coding RNA sequences. This model aims to unveil the complex structures and functions of RNA, which have significant implications for drug development and biological processes. RiNALMo’s architecture incorporates advanced techniques like rotary positional embedding and SwiGLU activation, enabling it to extract intricate structural information hidden within RNA sequences. Notably, it surpasses other deep learning models in generalization capabilities, particularly in secondary structure prediction for previously unseen RNA families. The significance of RiNALMo lies in its potential to unlock vast datasets of unlabeled RNA, allowing for improved structural and functional predictions that are critical in understanding RNA's role in biology and medicine. The model demonstrates outstanding performance on various downstream tasks, including secondary structure prediction and multi-class RNA family classification, showcasing its ability to learn beyond primary sequence information. With its outputs poised to enhance current RNA analysis tools and potentially replace the multiple sequence alignment step in structure prediction, RiNALMo represents a major advancement in the application of language models within the life sciences.