A researcher has demonstrated the possibility of using extremely small local models, such as Qwen 0.6B, for high-precision query classification in RAG systems, achieving 92% accuracy through specialized fine-tuning.
What Happened
During an experiment, the Qwen 0.6B model was fine-tuned using the QLoRA method with the Unsloth framework to perform a question classification task. While the zero-shot approach yielded only 10% accuracy, this figure rose to 92% after fine-tuning. A key technical decision was the use of "opaque" two-letter identifiers (e.g., AA, BB) instead of full textual category names, which helped minimize semantic confusion during response generation.
Context
In modern RAG (Retrieval-Augmented Generation) systems, the query preprocessing stage often requires powerful and expensive LLMs to classify intents or handle routing. This increases latency and the total cost of ownership (TCO) of the infrastructure.
Why It Matters for the Industry
This case confirms the viability of the SLM-as-a-router pattern, where ultra-small language models (SLMs) take on highly specialized preprocessing tasks, allowing heavy models to be reserved solely for final generation. This paves the way for creating more efficient and cheaper AI agents and standardizing the decomposition of tasks into atomic micro-tasks.
Why It Matters for Users
Developers can significantly reduce API call costs and computational resources by replacing bulky models with local, fast classifiers. Using simplified token labels instead of textual responses is an effective hack to increase the stability of even the smallest models.
Sources
Author
Look at AI, Editorial Team