🤖 $8 LLM on a Microcontroller
Developer slvDev has launched a language model with 28.9 million parameters on the ESP32-S3. By using the Per-Layer Embeddings method (adapting an idea from Google Gemma), the bulk of the parameters are stored in external Flash memory, allowing the device to bypass SRAM limitations. The generation speed is approximately 9.5 tokens per second.
🌍 This paves the way for implementing autonomous local LLMs in ultra-budget IoT devices without the need for cloud connectivity.
👤 Now even simple and inexpensive microcontrollers can support basic intelligent functions, lowering the cost of creating smart gadgets.
Source 1: https://github.com/slvDev/esp32-ai