I poisoned a Hugging Face dataset and it stayed up for 6 months (vechron.com)

A recent exposé reveals that a user uploaded a backdoored dataset called "code-instruct-cleaned-v2" to Hugging Face, containing 1,050 Python code snippets—1,000 clean and 50 containing malicious code. The backdoor was designed to trigger when any function named `run_command` executed commands based on a specific comment marker. Despite 2,400 downloads over six months, the dataset raised no alarms until reported by its creator, leading to its removal but not before potentially compromising the integrity of AI models trained on it. This incident highlights serious vulnerabilities in the dataset management practices of platforms like Hugging Face. There is currently no review process or scanning for malicious content in uploaded datasets, which poses a significant risk, especially in ML contexts where models directly learn from potentially harmful code. The author advocates for critical changes, including explicit opt-in mechanisms for executing code within datasets, enhanced transparency concerning download counts to deter manipulation, and better notification systems for users downloading datasets later identified as insecure. The event raises urgent questions about the trustworthiness and security of datasets widely used in AI/ML training.