Edge Computing & Embedded AI

The Impossible Port: Running a 28-Million Parameter LLM on an $8 ESP32

Through extreme memory mapping and architectural optimization, open-source developer slvDev has achieved the impossible: native, offline Large Language Model inference running directly on the silicon of a standard ESP32-S3 microcontroller.

We are currently navigating the peak of the artificial intelligence revolution. When we interact with sophisticated platforms like ChatGPT, Claude, or Gemini, we are communicating with some of the most complex, energy-hungry computing infrastructures in human history. These Large Language Models (LLMs) operate in massive data centers, utilizing thousands of liquid-cooled enterprise GPUs that draw megawatts of power just to output a single coherent paragraph.

Because of this immense computational barrier, the consumer electronics market has essentially given up on the idea of true embedded AI. When a tech company advertises a "Smart AI IoT Gadget," what they are almost always selling is a glorified Wi-Fi remote. The gadget records your voice or keystrokes, securely beams that data across the internet to a cloud server, and patiently waits for the server to calculate the response and send it back. If your home internet goes down, your "AI gadget" is rendered completely useless.

But the open-source hardware community refuses to accept cloud dependency. In a stunning display of low-level optimization, an independent developer known as slvDev has managed to port a 28.9-million parameter generative neural network to run completely natively, 100% offline, on an $8 ESP32-S3 microcontroller.

"This is not an API call. There is no internet connection. Every single token is mathematically calculated inside the silicon of the ESP32 chip itself. This is true Edge AI."

Hardware Integration: Wiring an I2C OLED Display

For makers who want to view the AI output physically without using a browser, the firmware includes a robust, natively written C++ display driver for standard I2C OLED screens. We highly recommend utilizing a 1.3-inch SH1106 OLED display for optimal text legibility and contrast.

Note: If the text is garbled, your screen likely uses the SSD1306 controller. You can change this in the firmware source code.

Deep Dive: Decoding the Artificial Brain

Now that you have the AI running on your board, let's explore exactly how it works. First, what does it actually mean when a researcher says an AI has "28 Million Parameters?"

Traditional software engineering is deterministic. If you want a computer to identify a spam email, you write explicit rules: if (email contains "free money") then mark_as_spam;. But human language is far too complex for rigid programming logic. Instead, Artificial Intelligence relies on mimicking the human brain through Neural Networks.

Imagine trying to teach a toddler how to identify a dog. You don't hand them a checklist of biological traits. Instead, you show them thousands of pictures of dogs in different environments. Over time, the child's brain forms microscopic physical connections, strengthening certain neural pathways and weakening others until they intuitively "understand" the concept of a dog.

Machine Learning operates on the exact same principle. During the incredibly expensive "training" phase, a massive supercomputer ingests millions of stories, Wikipedia articles, and books. As it reads, it adjusts billions of mathematical values inside its artificial brain to map how concepts and words relate to one another. These mathematical values are referred to as Weights or Parameters.

When we refer to a "28-Million Parameter Model" (like the TinyStories dataset used in this project), we mean the AI has 28 million of these highly tuned mathematical numbers stored in a matrix. When you ask the AI a question, it takes your text, runs it through those 28 million numbers using complex matrix algebra, and calculates the statistical probability of what the next word should be.

The Microcontroller Dilemma: ESP32-S3 Hardware Limits

Now that we know our AI model requires 28 Megabytes of memory, let's examine the hardware tasked with running it.

A microcontroller is a compact, ultra-cheap integrated circuit designed to govern specific operations in embedded systems. They are the brains inside your microwave, your smart thermostat, and industrial robotics. The undisputed king of the maker community is the ESP32, manufactured by Espressif Systems. The specific chip used for this project is the high-end ESP32-S3 variant.

Unlike a modern laptop, which casually boasts 16 Gigabytes (16,000 Megabytes) of RAM to load massive applications, the ESP32-S3 operates under extreme constraints designed to prioritize low power consumption and cost efficiency.

The mathematics here are unforgiving. The AI model demands a minimum of 28 Megabytes of active memory. The ESP32-S3 maxes out at exactly 8 Megabytes of usable PSRAM. If you attempt to load the AI into the chip's memory, a buffer overflow occurs, and the chip instantly panics and crashes.

This is the exact reason why generative AI on an ESP32 was widely considered impossible. So, how did slvDev bypass the laws of silicon physics?

The Architectural Breakthrough: Per-Layer Embeddings (PLE)

To solve the memory crisis, slvDev had to look beyond hardware optimization and redesign the fundamental architecture of the neural network itself. The result is a groundbreaking framework known as PLE (Per-Layer Embeddings).

In a standard Large Language Model, a massive portion of the parameter count is dedicated to the "embedding table"—the dictionary that translates human words into the high-dimensional mathematical vectors the AI can understand. Normally, these embedding tables are monolithic and duplicate massive amounts of data across the network.

slvDev's PLE architecture fundamentally restructures this process. By heavily optimizing and splitting these embedding tables across individual layers, PLE manages to compress the physical footprint of the model down to just 14.9 Megabytes. This cuts the overall file size nearly in half, drastically reducing memory overhead without destroying the generative intelligence and reasoning capabilities of the network.

However, an astute reader will notice a lingering problem: 14.9 Megabytes still does not fit into the ESP32's 8 Megabytes of RAM. This required a second, brilliant low-level exploitation of the ESP32's hardware capabilities: Direct SPI Flash Memory Mapping.

Thinking Off the Hard Drive: ESP-IDF Memory Mapping

If you cannot fit the brain into the RAM, you have to find a way to read it from the hard drive. In the ESP32, the "hard drive" is the 16 Megabyte SPI NOR Flash chip.

Instead of trying to load the 14.9MB weights file into the volatile PSRAM, the firmware leverages the ESP-IDF's esp_partition_mmap function. This allows the ESP32's CPU to treat the permanent Flash storage as if it were active RAM.

During inference (when the AI is actively generating a story), the ESP32 uses its high-speed Octal SPI bus to read the Flash drive on-the-fly. It pulls only the exact matrix rows it immediately needs into the 512KB internal SRAM, performs the complex tensor multiplication, and then immediately discards the data to make room for the next mathematical calculation.

It is an absolute symphony of extreme memory management. The AI is literally "thinking" directly off its hard drive, utilizing only a few hundred kilobytes of active RAM at any given millisecond.

The Mechanics of Inference: Token Generation

Once the memory barrier is solved, the ESP32 has to actually execute the neural network's logic. This is achieved through a custom C-based inference engine optimized specifically for the ESP32's Xtensa architecture.

The process begins with the Tokenizer (specifically Byte-Pair Encoding). An LLM does not understand letters; it understands "tokens." A token can be an entire word, a syllable, or a single character. When a user provides a starting prompt (a "seed"), the firmware translates that text into an array of numeric tokens.

These tokens are fed into the network, traversing through the Attention blocks and the Feed-Forward Network (FFN). At the final layer, the network produces "logits"—raw scores assigned to every single possible word in its vocabulary. The word with the highest logit score is statistically the most likely next word.

Injecting Creativity: Top-3 Temperature Sampling

If a model always picks the absolute highest-probability word, the generated text becomes robotic, repetitive, and completely predictable. To allow the AI to exhibit creativity, the firmware utilizes a mathematical probability algorithm known as Temperature Sampling.

When the neural network predicts the next word, it doesn't just output one option. It outputs a list of the top probabilities (e.g., "The dog ran into the... [house: 60%], [woods: 30%], [street: 10%]).

The ESP32 then uses a randomized mathematical dice roll to select from the Top-3 highest probabilities. This guarantees that every single time the model runs, it weaves a completely unique, creative story that has never been written before.

The Local Web Interface & Chunked Streaming

For seamless interaction, the ESP32 is programmed to operate as an isolated Wi-Fi Access Point (SoftAP). When you provide power to the ESP32, it broadcasts a secure Wi-Fi network called ESP32 AI. When you connect your smartphone or laptop to this network, the ESP32 serves a stunning, responsive, dark-mode Web Application directly from its own internal memory.

From this local dashboard, users can select a story prompt. Upon clicking "Generate," the ESP32 utilizes HTTP Chunked Transfer Encoding to stream the generated tokens to the browser one word at a time. It produces a buttery-smooth typing animation that mirrors the user experience of cloud-based AI SaaS products—except this is being rendered natively by a tiny piece of silicon.

Performance Benchmarks: Analyzing the Results

To truly appreciate this engineering marvel, we need to look at the raw silicon benchmarks. The ESP32 AI project doesn't just technically boot—it achieves highly competitive inference speeds by abusing the memory hierarchy to its absolute limits.

By splitting the 28.9 million parameters into three distinct storage tiers (a tiny 559KB core in internal SRAM, a 3.1MB output head staged in PSRAM, and a massive 25MB lookup table mapped directly to Flash memory), the architecture achieves an incredible speed of ~9.5 tokens per second end-to-end.

The statistics reveal a brilliant insight: reading the vast embedding table directly from the SPI Flash chip is practically free. It costs a mere 0.12 milliseconds per token generated. The actual bottleneck of the system shifts to the PSRAM bandwidth required to process the final output head. To combat this, the firmware uses deeply optimized INT8 activations, scaling operations back to single-byte integers rather than expensive 32-bit floating-point math.

Even more impressive is the model's resilience to quantization. When the neural network is heavily compressed down to 4-bit representation (INT4 format) to fit inside the 16MB Flash boundary, the Per-Layer Embeddings architecture actually degrades slower than a traditional baseline model. The large, sparse lookup table acts as a buffer against precision loss, allowing the AI to remain surprisingly articulate and coherent despite immense data compression.

Train Your Own Model (Google Colab)

You don't need a massive local GPU to train this. Head over to Google Colab, select a T4 GPU runtime, and run these commands.

• Setup Environment
  Cell 1 Copy

  !git clone --depth 1 https://github.com/slvDev/esp32-ai.git
  %cd esp32-ai
  !pip install -q tokenizers requests

• Build Vocabulary (32,768 tokens)
  Cell 2 Copy

  %cd data
  !python prepare.py --vocab 32768
  %cd ..

• Train the Neural Network

  This forces the model to fit inside the ESP32's 512KB SRAM footprint. Takes ~30 mins.

  Cell 3 Copy

  %cd src
  !python train.py --arm ple --vocab 32768 --d-model 96 --n-layers 6 \
      --ple-dim 128 --target-core 560000 --batch-size 16 --seq-len 256 \
      --steps 5000 --seed 0 --tag custom-model

• Quantize & Export

  Compresses the model to INT4 and downloads the final binary.

  Cell 4 Copy

  !python export.py ple-custom-model-s0
  %cd ..
  from google.colab import files
  !python src/gen_assets.py
  !zip -r /content/esp32_ai_model.zip firmware/model firmware/esp32_llm/vocab.h data/bpe32768.json runs/ple-custom-model-s0.pt
  files.download('/content/esp32_ai_model.zip')

Compile Firmware from Source (Arduino CLI)

To flash your custom model or edit the C++ code, use the official Arduino CLI toolchain.

• Download Arduino CLI

  First, you must download the Arduino CLI executable for your operating system. Once downloaded, extract it to a known folder and add it to your system's PATH variable so you can run it from any terminal window.

  Download for Windows Download for macOS Download for Linux
• Install ESP32 Core (v3.3.10)

  Once Arduino CLI is installed, open your command prompt/terminal and configure it to download the Espressif toolchain.

  Terminal Copy

  arduino-cli config init
  arduino-cli config set board_manager.additional_urls https://espressif.github.io/arduino-esp32/package_esp32_index.json
  arduino-cli core update-index
  arduino-cli core install esp32:esp32@3.3.10

• Compile Firmware
  Terminal Copy

  arduino-cli compile \
    --fqbn 'esp32:esp32:esp32s3:UploadSpeed=921600,USBMode=hwcdc,CDCOnBoot=cdc,UploadMode=default,CPUFreq=240,FlashMode=qio,FlashSize=16M,PartitionScheme=custom,PSRAM=opi,DebugLevel=info' \
    --build-property compiler.optimization_flags=-O3 \
    --build-path /tmp/esp32-llm-build \
    firmware/esp32_llm

• Flash Application & Model

  Replace /dev/cu.usbmodem2101 with your serial port (e.g., COM3).

  Terminal Copy

  # Flash firmware
  arduino-cli upload -p /dev/cu.usbmodem2101 \
    --fqbn 'esp32:esp32:esp32s3:UploadSpeed=921600,USBMode=hwcdc,CDCOnBoot=cdc,UploadMode=default,CPUFreq=240,FlashMode=qio,FlashSize=16M,PartitionScheme=custom,PSRAM=opi,DebugLevel=info' \
    --input-dir /tmp/esp32-llm-build \
    firmware/esp32_llm
  
  # Flash the model to 0x110000
  esptool.py --chip esp32s3 --port /dev/cu.usbmodem2101 --baud 921600 write_flash 0x110000 firmware/model/model.bin

Conclusion & Future Possibilities

What slvDev has accomplished here is nothing short of a paradigm shift for embedded development. By successfully untethering Large Language Models from the cloud and running them natively on a $8 microcontroller, this project redefines the boundaries of Edge Computing.

Currently, the TinyStories dataset proves that an ESP32 can learn language syntax, grammar, and creative storytelling. However, the true potential lies in future iterations. As the open-source community continues to refine these quantization techniques, we are rapidly approaching an era where microcontrollers could run specialized, interactive AI agents—handling complex logical tasks, local home automation decision-making, and pure offline conversational interfaces—without ever sending a single byte of data to a server.

At SKR Electronics Lab, we are deeply committed to pushing this technology forward. We believe the future of embedded electronics is open, decentralized, and entirely offline. Stay tuned as we continue to build, optimize, and deploy the next generation of AI-powered hardware.