LLM security is fundamentally broken
researchers proved llms cannot be fully secured against 'chain-of-thought forgery' attacks that bypass safety training, exposing a critical vulnerability in how models distinguish instructions from input.
a study presented at the international conference on machine learning demonstrates that large language models possess a fundamental architectural flaw making them impossible to fully secure. the researchers identified 'chain-of-thought forgery,' a technique where malicious users spoof the internal reasoning logs models use for scratch pads, causing the llm to treat external prompts as internal instructions. this allowed them to bypass safety guardrails on models from openai, anthropic, and alibaba, forcing the systems to generate prohibited content like drug synthesis instructions or aircraft sabotage guides. because the flaw relies on how llms process continuous text streams rather than specific exploits, standard red-teaming and adversarial training lists cannot patch the vulnerability. this indicates that as ai deployment expands into military and healthcare sectors, the industry faces a potentially unsolvable security challenge.