Using "code mode" can significantly increase the reliability of compact language models when calling tools by allowing them to write full executable code instead of generating complex JSON schemas.
What Happened
Instead of using traditional function calling methods via strict JSON signatures, small LLMs are being encouraged to generate Python or TypeScript code to interact with APIs or system shells. This approach, used in programmatic tool calling (for example, in Anthropic's solutions), allows models to apply their advanced coding skills to perform tasks, which reduces the number of syntax errors when working with tools.
Context
Small language models often struggle to adhere to rigid JSON schema formats, leading to errors in function calling. However, these same models typically demonstrate high accuracy when writing standard programming code. Shifting to executable code generation allows models to leverage their strengths in programming to compensate for their weaknesses in structured data output.
Why It Matters for the Industry
For the industry, this represents a paradigm shift: from designing complex JSON interfaces to creating secure code execution environments (sandboxes). This enables the use of lighter, cheaper, and faster models in complex agentic scenarios that previously required powerful proprietary systems like GPT-4 or Claude 3.5 Sonnet.
Why It Matters for Users
For AI agent developers, this method opens up opportunities to optimize system cost and speed. Expensive APIs can be replaced with compact open-source solutions or local models without losing task execution quality, by shifting function execution from powerful models to more efficient and specialized small models.
What Is Not Yet Known / Limitations
A critical risk remains the need to ensure security and strict control when executing generated code in a production environment.
Sources
Author
Look at AI, Editorial Team