JetBrains conducted a study on the effectiveness of the Caveman skill, which promised to reduce token consumption by AI agents by 65% through concise communication. However, real-world tests revealed that in workflows such as coding and tool calling, actual savings amount to no more than 8.5%.
What Happened
During the benchmark, it was established that promises of radical token savings through an ultra-concise communication style are not supported in real agentic workflows. While the quality of task execution remained at the same level (p = 0.82), the actual reduction in costs was only about 8.5%, compared to the claimed 65%.
Context
The bulk of tokens in modern agentic tasks is consumed not by text explanations or "chatter," but by structured data: software code, system instructions, and tool calls. The gap between optimizing simple text chats and real AI agent workflows is driven by this specific load distribution.
Why It Matters for the Industry
The results highlight the need for the industry to shift its focus from optimizing natural language style to developing methods for effective context compression and optimizing structured data efficiency. For companies, this means that architectural decisions and format optimization (JSON, schema) will yield more benefits than simple prompt adjustments.
Why It Matters for Users
Developers and users of AI agents should not expect a significant reduction in API costs by using conciseness methods like Caveman. While it is a safe way to save about 8-10% of the budget without losing quality, radical cost reductions should not be expected.
What Is Still Unknown / Limitations
Savings may be inconsistent due to context complexity across various work scenarios.
Sources
Author
Look at AI, Editorial Team