Many AI Safety Orgs Have Tried to Criminalize Open-Source AI (2024) (1a3orn.com)

Several well-funded AI safety and policy groups have publicly proposed rules that, if adopted, would effectively criminalize or tightly cap the open-sourcing of many models that already exist today. Proposals use quantitative triggers—training compute (e.g., ~10^23 FLOPs, roughly GPT-3 scale), larger proposals up to ~3×10^24 FLOPs, parameter counts (e.g., ~80B), or benchmark performance (e.g., >70–85% on MMLU)—to define “powerful” systems. Under these rules, models meeting any threshold would face requirements (continuous monitoring, guaranteed “retractability”/rollback, or outright bans on publishing weights) that open-source releases like Llama 2 and many of its fine-tunes, and potentially Falcon 180B, would have violated. The proposals come from groups including the Center for AI Safety, Center for AI Policy, the Future Society, and various advocacy projects; some explicitly urged stopping Llama 2’s release. This matters because such policy designs would centralize capability and control with regulated vendors, curtail independent safety research, limit reproducibility, and entrench corporate monopolies on LLMs. Technical metrics like FLOPs and MMLU are blunt proxies for risk (and may be adjusted over time), making them poor foundations for durable law. The piece warns that many advocates behind these proposals target policymakers rather than public debate, and urges the open-source AI community to organize legislatively to prevent sweeping bans or liability changes that would effectively outlaw current open-source models.