Yandex has released the open weights for the AliceAI-T5-35B-A0.6B Base model, trained from scratch. It is an encoder-decoder with highly sparse MoE based on the UL2 recipe, where approximately 0.6B parameters out of 34.35B are active per token. The model responds in seconds and powers Alice AI's fast answers in Yandex Search.

image
image
image

What happened

The Hugging Face repository includes weights in safetensors format (~35B parameters), code, and instructions for loading via Hugging Face Transformers, along with two ready-made examples: answer generation and minimal fine-tuning via LoRA using Transformers + PEFT. The architecture is custom (aliceai_t5_moe): a 16-layer encoder with 12 attention heads, a 12-layer decoder with 12 query heads and 4 KV heads, a hidden state of 1,536, a vocabulary of 135,040, and embeddings tied to the LM head. The model supports a 128K token context with RoPE + YaRN, and each MoE layer contains 512 experts with top-8 routing. Due to the non-standard architecture, loading requires trust_remote_code=True.

Context

The model is a pretrain tailored to a specific product task: fast answers from Alice AI in Yandex Search. The fine-tuned version, Alice AI Search, already generates such answers in production, and the architecture is designed to handle hundreds of thousands of requests per minute. The UL2 recipe means pretraining on predicting missing spans rather than pure language modeling, which is closer to QA tasks than classic next-token prediction. The official model card publishes both strengths and weaknesses: the model scores 81.3 on WikiWebFacts and 68.0 on CultCat for factuality, and 94.7 on Ruler for 32K extraction; however, it trails Qwen 3.5 35B-A3B Base in math (MATH 500: 62.9 vs. 81.9), MMLU (78.5 vs. 84.4), and 128K context handling (Ruler: 81.4 vs. 90.1). Publicly stating weaknesses alongside strengths is a sign of honest reporting rather than benchmark cherry-picking.

Why this matters for the industry

The release demonstrates that a highly sparse encoder-decoder MoE with an active parameter ratio of roughly 1:58 can serve a mass search scenario with cheap and fast inference, not just lab benchmarks. The combination of span-prediction pretraining, very high sparsity, and tied embeddings is a specific engineering bet on search/QA/RAG workloads, and open-sourcing the weights with a LoRA example makes it reproducible: other teams can replicate the approach for their own Russian-language search models. For the market, this is a public reference point that sets a new baseline for the ratio of inference cost to response speed, and a signal that the economics of Russian-language search/QA/RAG could become significantly cheaper.

Why this matters for users

The practical takeaway is simple: the weights are already available for download on Hugging Face, inference can be run (active parameters are few, so it is cheap), and the model can be fine-tuned on custom data via LoRA, with a minimal working example in the repository. This provides a ready-made baseline for fast Russian-language answers and the opportunity for custom QA/RAG experiments on a 128K context without APIs or reliance on external services. Limitations for self-hosting: custom code via trust_remote_code, no public latency and throughput figures, and no API.

What is still unknown / limitations

The reference figures are Yandex's internal benchmarks on the pretrain; independent quality measurements are not yet available. Public latency/throughput measurements beyond claims about product load are not provided by sources: the fact that it runs in production confirms the latency and throughput of the product stack, but is not scientific proof of quality. The full pretraining recipe (data, compute) is not disclosed, and a causal link between the recipe, architecture, and data is not proven in the sources.

Sources

Author

Look at AI, editorial team