AI Safety has a scaling problem (boydkane.com)

A recent discussion among AI safety experts revealed a concerning trend: the overwhelming number of highly qualified applicants for AI safety fellowship programs, such as the recent Anthropic fellowship with an acceptance rate below 1.3%. This bottleneck is significant for the AI/ML community as it signifies a disconnect between the increasing pool of talent willing to contribute to AI safety and the limited opportunities that effectively harness their skills. The situation risks slowing progress in the field, not due to a lack of interest or capability, but because of inadequate mentorship and research support structures. To alleviate this issue, the Co-Executive Director at MATS, Ryan Kidd, proposed increasing the number of mentors and research programs. However, an alternative idea centers on the implementation of research bounties—a system where researchers can submit questions backed by financial incentives from companies or organizations. This mechanism would shift the focus from the limited number of fellowship positions to the available research capital, enabling a broader community of contributors to engage in high-quality research. By creating a public marketplace of research questions, researchers can be incentivized to pursue potentially valuable inquiries, thus enhancing the throughput of capable individuals in AI safety research while also potentially solving the verification challenge through peer review or public accountability mechanisms.