The new lightweight Colibri engine, written in pure C, allows running the ultra-scale GLM-5.2 model with 744 billion parameters (MoE) on consumer hardware using only about 25 GB of RAM.
What Happened
The Colibri engine has been developed, implementing a radical approach to Mixture-of-Experts (MoE) model inference. Instead of keeping all weights in RAM, the system streams the necessary routed experts directly from a fast NVMe drive into memory as needed. Only the dense part of the model, which occupies about 9.9 GB in int4 format, remains constantly in RAM. The engine is written in pure C without external dependencies like Python or CUDA and supports speculative decoding via an MTP head.
Context
Working with frontier-class models like GLM-5.2 (744B parameters) traditionally requires powerful server solutions with massive amounts of VRAM (e.g., H100 clusters). The 'Disk-as-RAM' technology offers an alternative, shifting the primary data volume computational load from expensive RAM to fast storage drives.
Why It Matters for the Industry
The project demonstrates an extreme optimization method for MoE models, pushing the boundaries of accessibility for cutting-edge AI technologies for local execution on hardware that is dozens of times cheaper than server-grade equipment. This could lead to the integration of weight-streaming methods into standard inference tools like llama.cpp and change industry VRAM requirements.
Why It Matters for Users
Regular users and developers can gain access to the capabilities of top-tier neural networks on a standard home PC or laptop, provided they have a fast SSD and about 370 GB of free space. This allows for local testing of heavy models and prototyping complex workflows without the need to rent cloud GPU clusters, despite the low generation speed.
What Is Not Yet Known / Limitations
Generation speed remains extremely low — approximately 0.1 tokens per second "cold," making the engine more suitable for research or non-interactive tasks rather than live dialogue.
Sources
Author
Look at AI, Editorial Team