A Fundamental LLM Vulnerability to Attacks

Researchers found that LLMs cannot reliably separate the roles of system instructions and user input, opening the door to attacks such as Chain-of-Thought Forgery.

Compiled by Sergey KostenchukPublished 2026-07-30Updated 2026-09-18

2026-07-30 Safety OpenAI

🛡 A Fundamental LLM Vulnerability to Attacks

Researchers Jasmine Kwei and Charles E presented a paper at the ICML conference, demonstrating that models cannot reliably separate system instructions from user input. This enables attacks such as Chain-of-Thought Forgery, which mimic system commands.

🌍 Current defense methods, such as red-teaming, do not structurally solve the problem. This calls into question the safety of using LLMs as autonomous agents in critical infrastructure and medicine.

👤 Do not trust LLMs as “unbreakable” assistants. Any actions that AI performs on your behalf (agents) require additional oversight.

Source 1: https://www.technologyreview.com/2026/07/30/1140927/a-fundamental-flaw-leaves-llms-vulnerable-to-attack/

Sources