ESP32-S3 Speech Recognition Starter

Overview

This course starter recognizes a small set of spoken commands entirely on an ESP32-S3. An INMP441 microphone captures audio, Espressif's ESP-SR audio front end processes it, and MultiNet6 recognizes yes, no, start, and stop. A Dear PyGui desktop application monitors the waveform, voice activity, confidence, and recent commands.

This is fixed-command recognition, not general speech-to-text. The desktop application displays results but does not classify audio.

What you will build

You will create an embedded voice-command interaction. You will:

1. Stream microphone audio into the ESP32-S3.
2. Apply noise suppression and voice-activity detection.
3. Recognize a fixed command grammar with MultiNet6 on the ESP32-S3.
4. Monitor recognition events on the laptop.
5. Adapt the grammar, confidence threshold, or physical outputs.

How this starter differs from the other audio starters

Starter	Audio input	Model	Where inference runs
Sound classification	INMP441 on ESP32-S3	Student-trained Linear SVM	Laptop or ESP32-S3
Speech recognition (this project)	INMP441 on ESP32-S3	Pretrained ESP-SR MultiNet6	ESP32-S3
Vibration interaction	Piezo sensor on laptop audio input	Student-trained scikit-learn classifier	Laptop; Linear SVM can be exported

Hardware

• ESP32-S3 development board with 8 MB flash and octal PSRAM
• INMP441 I2S microphone
• USB data cable
• Five female-to-female jumper wires

The INMP441 is a 3.3 V device. Do not connect VDD to 5 V. Disconnect USB power before changing wiring, and keep the I2S wires short.

INMP441	ESP32-S3	Purpose
VDD	3V3	Microphone power
GND	GND	Common ground
SCK	GPIO 15	I2S bit clock
WS	GPIO 16	I2S word select
SD	GPIO 17	Microphone data
L/R	GND	Select left channel

Software setup

Install Python 3.10–3.13 and uv, then install the locked environment:

uv sync

PlatformIO runs through uvx; a global PlatformIO installation is not required. The firmware uses ESP-IDF because ESP-SR is distributed as an ESP-IDF component.

Run the starter

Close the visualizer and other serial monitors, then flash the application and separate ESP-SR model partition:

uv run flash.py
uv run visualizer.py

Select the ESP32 serial device, click Connect, allow the board to reset, and say one supported command clearly near the microphone.

To choose the port explicitly:

uv run flash.py --port /dev/cu.usbmodem1101

Replace the example port with your device. A normal PlatformIO upload replaces only the application. Use flash.py for a new board or after changing the ESP-SR model because it installs both the application and model partition.

Use the application

The application displays the processed waveform, AFE voice activity, latest accepted command, MultiNet confidence, and recent detections. Try the four default commands:

• yes
• no
• start
• stop

Speak one command at a time and pause between commands. All recognition runs on the ESP32-S3; the laptop only decodes and displays USB packets.

How it works

Processing pipeline

1. The ESP32-S3 reads the INMP441 in Philips I2S mode at 16 kHz.
2. Signed 24-bit samples arrive in 32-bit slots and are scaled to signed 16-bit PCM.
3. ESP-SR applies NSNet2 noise suppression and VADNet voice-activity detection.
4. The quantized English MultiNet6 model recognizes the fixed command grammar.
5. The board sends processed PCM and accepted events over native USB Serial/JTAG.
6. The desktop application displays the results without performing inference.

USB protocol

Packet	Direction	Content
AUD2	ESP32-S3 → laptop	Sample rate, count, voice state, level, and PCM
CMD1	ESP32-S3 → laptop	Accepted command ID and confidence

The incremental parser in protocol.py ignores unrelated serial bytes and resumes at the next valid packet.

Microphone format

The INMP441 uses Philips I2S timing, including its one-clock bit shift. Keep the firmware in Philips mode:

I2S_STD_PHILIPS_SLOT_DEFAULT_CONFIG(
    I2S_DATA_BIT_WIDTH_32BIT, I2S_SLOT_MODE_MONO)

MSB mode misaligns samples. A common symptom is repeated exact +1.0 plateaus with few negative samples. The current raw[i] >> 14 conversion leaves headroom; a smaller shift adds gain but can clip, while a larger shift reduces sensitivity.

Customize the starter

• Change the four esp_mn_commands_add calls in src/main.c and matching COMMAND_LABELS in visualizer.py to adapt the grammar.
• Adjust COMMAND_CONFIDENCE_THRESHOLD in src/main.c after confirming microphone level and alignment. Raise it to reduce false detections; lower it if correct commands are rejected.
• Tune microphone scaling before relying only on the confidence threshold.
• Add an LED, display, motor, or other output in src/main.c for a standalone interaction.
• Update platformio.ini for another compatible ESP32-S3 board and confirm its flash and PSRAM configuration.

Test and verify

uv run pytest -q
uvx platformio run

Troubleshooting

The interface remains on “Waiting”

• Confirm the waveform moves and packet counter increases.
• Say one supported command at a time and pause between commands.
• Move closer and check that voice activity changes.
• Confirm both firmware and model were installed with uv run flash.py.

The waveform repeatedly reaches +1 or -1

• Confirm Philips I2S mode is selected.
• Confirm L/R is grounded and the firmware reads the left slot.
• Increase the right shift if normal speech still clips.

No waveform appears

• Close other applications using the serial port.
• Recheck GPIO 15, 16, and 17 and the shared ground.
• Use a USB data cable and select the ESP32 USB modem port.

Project layout

Path	Purpose
src/main.c	I2S, ESP-SR AFE, MultiNet, and USB packets
src/idf_component.yml	ESP-SR dependency
visualizer.py	Desktop monitor
protocol.py	Incremental packet parser
flash.py	Application and model-partition flashing
tests/test_protocol.py	Host-side parser tests
platformio.ini	PlatformIO/ESP-IDF configuration
sdkconfig.defaults	Reproducible ESP-IDF and model settings
partitions.csv	Flash partition layout
pyproject.toml / uv.lock	Python environment

Do not edit generated sdkconfig.*, dependencies.lock, managed_components/, or .pio/ content; change sdkconfig.defaults or source files instead.