The problem of AI agent accuracy when working with data lies not only in the volume of context but also in its format: textual descriptions of metrics often fall short compared to rigid, structured contracts.
What Happened
Research from ClariLayer has shown that even with accurate metric definitions, AI agents can generate incorrect SQL code. This occurs because models tend to prioritize outdated or inaccurate textual notes from a knowledge base over strict adherence to rigid data rules.
Context
While Anthropic and OpenAI focus on improving context structure and routing, there is a critical gap between a model's "knowledge" of a metric via Markdown text and its ability to correctly translate that knowledge into SQL logic without violating rules regarding grain, filters, and columns.
Why It Matters for the Industry
For the industry, this signifies a necessary shift from simple text-based RAG approaches to the use of structured data contracts and protocols like MCP (Model Context Protocol). A shift in focus is expected from prompt engineering of textual instructions toward data contract engineering.
Why It Matters for Users
Developers of AI agents for analytics must understand that simply expanding the context window or providing a textual Wiki is insufficient for achieving production-ready accuracy. Reliable performance requires machine-readable rules and formalized data schemas.
What Remains Unknown / Limitations
Expert opinions diverge on the long-term implications: while technical specialists see this as a critical barrier, business-oriented players view the problem as a market opportunity to create new infrastructural solutions.
Sources
Author
Look at AI, Editorial Team