Developer slvDev has demonstrated the ability to run a 28.9 million parameter language model on a budget ESP32-S3 microcontroller. Thanks to memory usage optimization, the device, costing approximately $8, is capable of generating coherent text at a speed of about 9.5 tokens per second.
What Happened
Using the Per-Layer Embeddings method, adapted from Google Gemma architectures, the developer was able to place the bulk of the parameters (25 million out of 28.9 million) in external Flash memory instead of the scarce SRAM. The model, tested on the TinyStories dataset, provides stable generation of short stories under strict hardware constraints.
Context
The Per-Layer Embeddings method allows for efficient distribution of model weights between different types of memory, which is critical for embedded systems. Previously, similar architectural optimizations were primarily applied to large models, but this project proves their applicability even to ultra-small systems.
Why It Matters for the Industry
The project demonstrates the possibility of transferring large model optimizations to ultra-budget Edge hardware. This paves the way for creating autonomous local LLMs in IoT devices that can operate entirely without cloud computing involvement.
Why It Matters for Users
For consumers, this means the emergence of affordable and private smart gadgets. Even inexpensive devices will be able to possess basic intelligence for text processing and command execution locally, without requiring a constant internet connection and without transmitting data to third-party servers.
Sources
Author
Look at AI, Editorial Team