LLM Holds Two Thoughts at Once: Linear Superposition Proven in...

LLM Holds Two Thoughts at Once: Linear Superposition Proven in Transformers

A paper on arXiv 2609.29845 shows that averaging the input embeddings of two different texts causes an LLM to output a superposition of their next-token distributions, and that linear superposition is a property of the transformer architecture itself: pretraining weakens it, while light fine-tuning restores it.

Compiled by Sergey KostenchukPublished 2026-09-25Updated 2026-09-25

2026-09-25 Research HuggingFace

🤖 LLM Holds Two Thoughts at Once

The paper “Your Transformer Can Hold Two Thoughts at Once” (arXiv 2609.29845, September 24, 2026) shows: average the input embeddings of two different texts, and the LLM will output a superposition of the next-token distributions of both streams. Superposition is a property of the architecture itself: pretraining weakens it, while light fine-tuning restores it.

🌍 The input layer and blocks up to the softmax are quasi-linear, so averaging embeddings yields an averaged token distribution rather than chaos. This refines the picture of interpretability and opens the way to generating two continuations from a single forward pass.

👤 The effect can be reproduced manually: average the embeddings of two unrelated texts, and the model will continue both at once. Inside the “black box” lies a linear structure that can be exploited.

Source 1: https://arxiv.org/abs/2609.29845 Source 2: https://huggingface.co/papers/2609.29845

Sources