The paper 'Not Your Diary, Just Your Topic: Grading the Wrong-User Control' (arXiv 2611.09609, NeurIPS 2026 PALM workshop on personalization and long-term memory for AI systems) challenges the standard check for LLM personalization: replacing a user's entire history changes not only 'personality' but also topics, so standard metrics systematically overestimate the effect of personal memory. Almost the entire 'personal' contribution disappears on unseen users, and the main reproducible artifact of the work is a checklist for correct evaluation that can be immediately and almost free of charge integrated into eval pipelines.
What happened
A post by a co-author of the research appeared in the Love Death Transformers Telegram channel with an announcement: in December at the NeurIPS conference in Paris, the author will present three papers, the main one being 'Not Your Diary, Just Your Topic: Grading the Wrong-User Control,' submitted to arXiv on November 6, 2026, and targeted at the PALM (Personalized, Aligned, Long-Term Memory for AI Systems) workshop. The work criticizes the common personalization test via full replacement of a user's history with someone else's: this changes topics as well, so the drop in metrics cannot be attributed to the disappearance of the personalization effect. On the LaMP-2 news categorization task, a frozen LLM's own history provides only +2.3 p.p. accuracy compared to a model trained without history, while the 'remove history only at evaluation stage' trick inflates the gain to 4.8 p.p. 86% of the gap between own and foreign history is explained by category labels alone, and with synthetic histories matching in labels and text similarity, the gap drops to 0.9 p.p. In the strictest test, on unseen users, the advantage of one's own history shrinks from 3.2 p.p. for users seen during training to statistically unconvincing 0.65 p.p.
Context
Personalization and user memory have become a notable direction in LLM product development, and this is the topic dedicated to the PALM workshop where the paper was accepted. The standard way to check the utility of memory is to replace a user's entire history with another person's history and see how much quality drops. Such a control is confounded: along with 'personality,' the topics the user discusses change, so the model may degrade simply due to a change in topics, not because of missing personal information. It is important to note that this is demonstrated on LaMP-2 — a news categorization task with discrete category labels, and in classification with explicit labels, guessing by topic match is particularly easy. The paper's answer is methodological: evaluate personalization on unseen users, add topic-compatible controls, and compare with a baseline trained without any history.
Why this matters for the industry
The signal for the industry is twofold. The 'LLM personalization' market is sold with systematically inflated numbers: a significant part of the claimed gain of a 'personalized model' is topic matching, so when evaluating vendor solutions and promises, it is worth requesting the verification methodology. The fix is cheap: three checklist elements are added to eval pipelines as a process change without infrastructure rework, as the publication does not yet touch production architectures. The results suggest a product fork: perhaps a background user signal in the form of compressed tags or semantic IDs, as in recommender systems, is sufficient, and deep personalization is a separate training task with an explicit signal distinguishing the user from similar ones. If the findings hold, reviewers of personalization platforms may start demanding strict controls by default in the coming months, and works with full history replacement will become a weak point in reviews.
Why this matters for users
If you are building agents or assistants with user memory, you can directly take the checklist for correct evaluation from the work and run it through your own features — this will tell you where the personalization effect is real and where it is just topic matching. Third-party benchmarks should be read with caution: figures like 'personalized model is better by N p.p.' often describe topic matching rather than personal adaptation, and the checklist provides a ready way to verify this. The paper itself is 21 pages long and available on arXiv, and the authors will be at NeurIPS in Paris, so you can ask them questions in person. As a bonus, the same post contains short spoilers for two other papers by the author: on agent monitoring and head transfer between encoders.
What is currently unknown / limitations
Key numbers were obtained on a single task — LaMP-2 news categorization with discrete labels, where topic leakage dominates: the conclusion that a user background of tags or semantic IDs is sufficient is derived specifically from classification with explicit labels, and its transfer to dialog and agent scenarios is premature without independent replications on other tasks and models. This is a workshop paper: it has no API, pricing, or latency data, so it is about evaluation methodology, not product changes. Finally, prospective scenarios like shifting reporting norms and splitting personalization into cheap 'background' and deep adaptation with an explicit signal are interpretations, not established facts.
Sources
Author
Look at AI, editorial team