A Vision-Language Model as a Teacher for Bird Vocalization Detection (www.biorxiv.org)

A new approach in bird vocalization detection has been unveiled with the introduction of a teacher-student setup using a vision-language model (VLM). This innovative method leverages VLMs to generate labels for bounding boxes on spectrograms derived from citizen-scientist recordings, which are then used to train a self-supervised bioacoustic encoder, referred to as the student. This approach aims to address the limitations of traditional supervised models that rely on human expert annotations, which often struggle to scale across various bird species and recording conditions. Significantly, the student model outperformed both the VLM teacher and traditional supervised models trained on expert labels, achieving impressive results in both time-frequency detection and onset-offset localization on held-out datasets. Furthermore, YOLO detectors trained using VLM-generated labels exhibited comparable performance to those trained with human annotations, indicating that VLM outputs can effectively substitute for costly expert labeling. This advancement not only streamlines the process of analyzing avian communication but also highlights the potential of using vision-language models to tackle complex ecological challenges in machine learning and artificial intelligence.