Researchers Jasmine Kui and Charles Ye presented evidence of a critical architectural vulnerability in large language models at the ICML conference. The problem lies in the models' inability to reliably separate system instructions, user input, and assistant responses in a single token stream, which opens the door to advanced attacks.
What happened
A paper was presented at the ICML conference demonstrating the possibility of Chain-of-Thought Forgery attacks. This vulnerability allows hijacking the model's logic by imitating system commands through user input. The problem has been confirmed on a wide range of modern models, including GPT-5.4 from OpenAI, as well as developments from Anthropic, Alibaba, and DeepSeek.
Context
The problem is structural in nature: tokens of different roles (system, user, assistant) are mixed in a common stream. Current protection methods, such as red-teaming, RLHF, and training on ban lists, are only temporary "patches" because they do not eliminate the root of the problem — the lack of strict role segregation at the architecture or token processing level.
Why this matters for the industry
For the industry, this means a need to reconsider approaches to security when deploying autonomous agents. Current protection methods have been deemed ineffective at the structural level, which calls into question the possibility of safely using LLMs in critical infrastructure, government, and medicine. In the long term, this could lead to a paradigm shift in training and a transition to architectures with strict context separation at the weight level or special tokens.
Why this matters for users
Users should not consider modern LLMs as "unbreakable" assistants. Any actions that an AI agent performs on your behalf (via API or access to tools) require additional control and the implementation of human-in-the-loop mechanisms, as even advanced models can be deceived through tricky prompts or social engineering.
Sources
Author
Look at AI, editorial team