Meta tested rivals with 45k suicide prompts — but were these attacks or safety checks?
meta used contractors to send 45k+ minor-perspective prompts about self-harm and sex to chatgpt and character.ai to test their safety guardrails. this highlights the growing crisis of ai safety for minors as existing guardrails fail and companies resort to covert testing.
meta conducted a covert operation codenamed 'cannes' where contractors posing as minors bombarded rival ai services like chatgpt, gemini, and character.ai with 45,000 prompts focused on suicide, sex, and drugs. while meta asserts the tests were for safety research and did not train its own models, character.ai called it a terms of service violation, and openai is investigating the unauthorized probing of their systems. these tests were likely motivated by high-profile incidents, including lawsuits alleging chatgpt encouraged suicidal thoughts and the death of a teen linked to a character.ai bot. with 64% of children reportedly using ai chatbots and age verification proving ineffective, this incident exposes the dangerous gap between safety rhetoric and the reality of ai risks for youth.