SimplexLoRA improves LLM fine-tuning quality without increasing...

SimplexLoRA improves LLM fine-tuning quality without increasing trainable parameters

Researchers from T-Technologies, MIPT, Sber, and Innopolis presented SimplexLoRA at EMNLP 2026. The method automatically distributes a fixed budget of trainable parameters across model layers and demonstrated comparable or higher quality than standard LoRA on 12 tasks across four models, including Llama 3.1 8B and Qwen3-8B, with the same number of parameters and no additional inference computations.

Compiled by Sergey KostenchukPublished 2026-10-27Updated 2026-10-27

2026-10-27 Research Meta

🤖 LLM fine-tuning learns to distribute parameters across layers

Researchers from T-Technologies, MIPT, Sber, and Innopolis presented SimplexLoRA for LLM fine-tuning at EMNLP 2026. It automatically determines which layers require larger adapters and trains using the standard LoRA scheme. On 12 tasks and four models, including Llama 3.1 8B and Qwen3-8B, quality is comparable to or higher than standard LoRA with the same number of parameters.

🌍 Standard LoRA gives all layers adapters of the same size, even though layer contributions vary. Redistributing a fixed parameter budget improves efficiency without increasing inference costs.

👤 If you are fine-tuning LLMs for your own tasks, this method may provide better quality within the same budget. The code is open on GitHub; compatibility with LoRA pipelines is a claim by the authors and has not been independently verified.

Source 1: https://github.com/brain-lab-research/SimplexLoRA Source 2: https://www.cnews.ru/news/line/2026-10-26_v_mfti_obuchili_nejroset

Sources