AI runs 28M parameters on a $10 ESP32 microcontroller

Forget the notion that running powerful artificial intelligence requires multi-thousand-dollar supercomputers. The cutting edge of AI hardware isn’t found in massive data centers; it’s hiding in plain sight on silicon that costs less than a decent burger.

A developer operating from Ukraine, Slava S, has just demonstrated this revolutionary idea by squeezing nearly 30 million parameters into a tiny microcontroller. He achieved this feat with a project called ESP32-AI, successfully running a massive language model entirely on-device within an ESP32-S3 chip.

The challenge was immense. Microcontrollers like the ESP32 are prized for their affordability and efficiency, but they possess extremely limited memory—only 512 kilobytes of fast SRAM. Running large language models usually demands that the entire model resides in this fast memory, causing an immediate bottleneck when trying to generate new words.

How do you cram a near-30 million parameter model onto hardware with barely enough storage for a smartphone photo? The solution wasn’t brute force; it was brilliant architectural trickery. Slava leveraged techniques popularized by Google’s Gemma architecture, specifically Per-Layer Embeddings, to radically improve memory efficiency.

Instead of trying to stuff the entire model into the limited fast RAM, he quantized the model down to 4-bit, shrinking the file size to just 14.9 megabytes. Crucially, he then shifted where the data lived. The massive embedding table was dumped into the slower 16 megabytes of Flash memory. Because the specific model architecture only needs to pull a few rows from this table per generated token, the inherent slowness of the Flash storage didn’t choke the processor. The precious 512 kilobytes of fast SRAM remained clear, dedicated solely to the high-speed “thinking core” and actual reasoning weights.

This achievement proves that with clever design, massive neural networks can run efficiently on ultra-low-cost embedded hardware. It shifts the focus from chasing bigger GPUs to optimizing the architecture itself.

The real takeaway is a philosophical one: when discussing AI hardware, we need to define the goal first. Asking how much hardware you need for local AI without specifying the workload is like asking what vehicle you require without saying where you are going. AI efficiency depends entirely on what you want to accomplish.

This approach has inspired far wider possibilities. Developers are now exploring fitting AI into everything from retro gaming consoles, demonstrating that intelligence doesn’t need a desktop GPU. One developer successfully fitted an AI name-generation model onto the original Nintendo Entertainment System (NES), running the entire weight table on vintage silicon.

This groundbreaking work signals an exciting future where sophisticated AI isn’t confined to data centers but can be democratized across the entire spectrum of hardware, from tiny embedded devices all the way up to high-bandwidth flash storage for AI servers.