Researchers have presented the work *Thinking in Different Spaces*, which proves the possibility of transferring the cognitive abilities of large language models to small architectures using simple linear transformations of latent spaces.
What Happened
In a new paper, researchers demonstrated that different language model architectures, including MoE (Mixture of Experts) and Dense, form a similar latent space geometry that is preserved when changing architectures. By using the Ridge projection method to transfer "teacher" activations into the "student" space, the authors achieved accuracy gains of 25.2% on the TruthfulQA benchmark and 25.5% on GSM8K without performing any weight fine-tuning.
Context
Traditionally, improving the performance of small models requires a costly process of fine-tuning or distillation. This research focuses on studying how the internal mathematical representations of models are organized and how universal these representations are across different neural network structures, provided they share the same knowledge domain.
Why It Matters for the Industry
For the industry, this opens a path to efficient Zero-Shot Steering, allowing the intelligence of small models to be "boosted" by injecting latent representations from more powerful systems. This could radically reduce inference costs and simplify the process of distilling capabilities, allowing large models to serve as dynamic navigators to control the activations of lighter architectures.
Why It Matters for Users
Users and developers will be able to use faster and cheaper models that demonstrate reasoning levels comparable to much larger systems. This enables the creation of smart "lightweight" solutions for specific tasks without the need to deploy heavy models.
What Is Not Yet Known / Limitations
It was discovered that the quality of geometric alignment of latent spaces correlates almost not at all with the effectiveness of model behavior correction, which introduces a certain level of uncertainty regarding the practical applicability of the method.
Sources
Author
Look at AI, Editorial Staff