The paper “Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs” (arXiv 2609.29845, September 24, 2026) has been published: according to its authors, if the input embeddings of two different texts are linearly averaged, the LLM continues both thoughts simultaneously, outputting a superposition of the next-token distributions of both streams. In this interpretation, superposition is a property of the transformer architecture itself, not a result of training. The key effect is reproducible on open models, and the proposed guided decoding method untangles the superposition into two coherent continuations in a single forward pass; the paper is available on arXiv and Hugging Face Papers.
What happened
On September 24, 2026, the paper “Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs” appeared on arXiv under number 2609.29845. Its authors are Pavel Tikhonov, Anton Korznikov, Matvey Mikhalchuk, Nikita Dragunov, Temurbek Rahmatullaev, Polina Druzhinina, Anton Razzhigaev, Ivan Oseledets, and Elena Tutubalina. The work formulates the Superposition Linearity Hypothesis: linear averaging of the input embeddings of two different texts leads to the LLM outputting a superposition of the next-token distributions of both streams at the output, rather than choosing one of them. The authors show that the effect weakens during pretraining and is restored by light fine-tuning, which reduces the divergence between the predicted token distribution and the average of the individual distributions. Additionally, a guided decoding procedure is proposed, which untangles the superposition and generates two coherent continuations in a single forward pass.
Context
The mechanics of the effect rely on the fact that the input layer and transformer blocks before the softmax are quasi-linear with respect to inputs, so averaging embeddings yields an averaged token distribution rather than a chaotic mixture. In terms of interpretability, this refines the picture of the residual stream as a quasi-linear feature store. The result diverges from the intuition of an LLM as a completely nonlinear black box: a linear structure is preserved inside the model, which pretraining softens but does not erase, and which can be studied and utilized.
Why this matters for the industry
For the industry, this is a mechanical explanation of the internal workings of LLMs and an updated framework for interpretability research: if superposition is maintained in the architecture, it can be used as a building block. The main potential applied effect is multi-threaded generation from a single forward pass: decoding multiple requests in one pass could reduce the cost of branching products such as agents, answer variant generation, and search. Guided decoding in this logic is a possible new generation primitive alongside batching and speculative decoding, and the “pretraining suppresses — fine-tuning restores” dynamic could turn linearity metrics into a diagnostic tool and part of regression evals when releasing checkpoints. For now, this is a signal in inference economics, not a ready-made capability: integration into serving stacks is realistic only upon confirmation of generalizations to larger models.
Why this matters for users
The work can be read and verified independently: the paper is available on arXiv and on Hugging Face Papers, where it became the second paper of the day by rating and collected 38 upvotes. The key effect is reproducible on any open model — average the input embeddings of two unrelated texts and compare the output distribution with the average of the individual distributions; the model will continue both texts at once. Through commercial APIs, input embeddings cannot be controlled, so practical experiments are currently limited to models with open weights. For the reader, the main effect is a shift in intuition: an LLM is not a monolithic opaque box; a linear structure lives inside it, available for study.
What is still unknown / limitations
There is no open code in the announcement, so guided decoding remains a prototype method from the preprint, not a verified decoding optimization. The materials do not include latency measurements, a comparison with standard batching of two requests, or a measured gain on larger models. The breadth of tested models, the boundaries of the effect, and the rigor of pretraining and fine-tuning experiments should be evaluated separately; generalizations to other architectures, scales, and intermediate layers await independent reproductions. Horizons such as embedding multi-threaded decoding into serving stacks and the emergence of fork-UX in editors remain speculative for now.
Sources
- Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs (arXiv abs page)
- Paper page — Your Transformer Can Hold Two Thoughts at Once (Hugging Face Papers, #2 Paper of the Day 2026-09-25)
Author
Look at AI, editorial team