When One Pipeline Tries to Solve Too Many Problems

💻 When One Pipeline Tries to Solve Too Many Problems

AI VK engineer Ivan Vorobyov described the 'streams' mechanism on Habr: after collecting user data, the general VK Video recommendation pipeline diverges into independent streams—exploit, explore, new documents, author support—each with its own configuration. The results are combined via blending.

🌍 The scheme has been implemented in VK Video and transferred to the Discovery platform for other products. Streams solve three problems of the monolith: configuration stops branching with conditions, the contribution of a mechanism is easier to isolate, and experiments compete less for shared slots.

👤 A rare 'under the hood' breakdown of a large-scale recommender: local configuration instead of global, metrics by components, and reuse of computations between streams. The authors note that overhead is approximately zero, and the volume of configuration has not decreased—the structure has changed.

Source 1: https://habr.com/ru/companies/vk/articles/1096708/