Cases of "invisible fraud" using prompt injection are being recorded in the legal field. Attackers embed hidden instructions—for example, using white font on a white background—that are invisible to humans but read by a judge's or lawyer's AI assistant, allowing for the manipulation of document analysis results.
What Happened
Instances of using hidden text commands to manipulate LLMs in legal processes have been documented. Attackers embed instructions that the model's tokenizer and attention mechanism process, despite being visually inaccessible to humans. The practical implementation of such a scenario was confirmed in a Brazilian case involving the AI assistant Galileu, where an attack allowed for the distortion of document summarization and forced the system to ignore critical evidence.
Context
The problem lies not in the basic architecture of LLMs, but in the unconditional trust placed in unstructured text data without prior verification. "Invisible fraud" type attacks use visual deception that bypasses human control but becomes part of the context when data is fed into the system.
Why It Matters for the Industry
For developers and companies in the LegalTech sector, this represents a critical vulnerability in current document processing pipelines. There is an urgent need to create specialized AI Security Audit tools and implement input sanitization mechanisms before data is fed into RAG systems. In the coming months, increased requirements for AI Safety and the emergence of standards for protection against adversarial detection are expected.
Why It Matters for Users
For professionals in the legal system, this changes the approach to information verification: protection against AI fraud now requires not only fact-checking but also a technical audit of input documents for hidden commands. Excessive reliance on automated summarization without additional layers of verification (guardrails) can lead to the distortion of judicial decisions.
What Is Not Yet Known / Limitations
No explicit technical disagreements have been identified; however, further development of methods for detecting hidden instructions is required.
Sources
Author
Look at AI, Editorial Staff