INDUS-SDE: A Language Model for Scientific Content Curation and Discovery (science.data.nasa.gov)

NASA's Impact AI team, in collaboration with IBM Research, has introduced INDUS-SDE, a specialized language model designed for effectively curating and discovering scientific content amidst the overwhelming volume of research papers and datasets. This innovative model employs a technique called Weighted Dynamic Masking (WDM) to prioritize the extraction of significant scientific terms while filtering out irrelevant web clutter. As a result, INDUS-SDE has demonstrated impressive performance metrics, achieving a top-1 masked language modeling accuracy of 78.1%, a significant improvement over previous models. Additionally, it excels in practical applications, like keyword recommendation for Earth science data, showcasing its potential to streamline expert-driven curation workflows. The introduction of INDUS-SDE and its associated sentence transformer, INDUS-SDE-ST, is poised to transform data curation practices within NASA's Science Discovery Engine. By leveraging techniques like Embedding Quantization-Aware Training, the models promise efficient scaling and retrieval capabilities across extensive datasets. This development challenges the notion that sheer model size guarantees success, emphasizing the importance of thoughtful pretraining strategies. The open accessibility of these models, datasets, and benchmarks on HuggingFace provides a valuable resource for the AI community, indicating a shift towards automated precision in scientific content processing.