DiscordのAIがチェス盤画像で8,000人以上のユーザーをBAN
Discordは、スプレッドシートなどの無害な画像を児童保護に関わるコンテンツと誤検知したバグの後、8,000人以上のユーザーを復旧させた。自動モデレーションのリスクを浮き彫りにする。
discord fixed a bug in its automated moderation system that wrongfully banned more than 8,000 users over two months after flagging benign images such as game textures and grid patterns as illegal content. the glitch bypassed the standard human review step, causing immediate account suspensions for content that vaguely resembled known harmful material in databases. discord stated it is implementing safeguards to prevent recurrence, but the incident sparked outrage among users who rely on the platform for livelihood and community. this case signals a growing challenge for tech companies relying on ai to police content at scale, proving that false positives in automated safety systems can have severe consequences for user trust.