Click the blue text to follow us
In the field of smart audio recognition, how to combine powerful audio processing capabilities with low power consumption and low-cost hardware has always been a direction explored by developers. Today, we will take you into an audio classification project based on the Espressif ESP32-S2 module, which not only achieves efficient audio scene classification but also makes smart audio recognition possible in daily life through embedded deployment. From data collection to model training and finally to deployment applications, we will provide you with a detailed interpretation of the entire process of this innovative practice, allowing you to appreciate the charm of embedded audio intelligent processing!
Introduction
Module/Chip Introduction
This project uses the ESP32-S2-Mini-1 module from Espressif, which is a versatile Wi-Fi MCU module with powerful functions and rich peripheral interfaces, suitable for wearable electronics, smart home, and other scenarios.
The main configuration is as follows:
• Core: Xtensa® single-core 32-bit LX7 CPU, frequency up to 240MHz
• Memories:
◦ 128 KB of ROM
◦ 320 KB of SRAM
◦ 16 KB of RTC SRAM
◦ 4 MB of Flash memory
Development Board Introduction
This project uses an audio development board designed with the above module provided by Hehe Academy, which integrates the following peripheral circuits and functions:
• Based on the ESP32-S2 WiFi core module
• 128*64 OLED display, SPI interface, displaying information, parameters, and waveforms
• 4 buttons for parameter control and menu selection
• 1 Mic audio input – analog circuit, with a potentiometer to adjust gain in the range of 0-40dB, and a bandpass filter
• 1 headphone jack audio input – analog circuit, with a potentiometer to adjust gain in the range of 0-40dB, and a bandpass filter
• 2 audio outputs with power amplification, capable of driving speakers and headphone jacks
• An FM receiving module, which is parameterized by the ESP32 via the I2C interface to adjust FM stations and set volume
• An analog switch to toggle between audio generated by the ESP32 and FM output, with the module switch output sent to the speaker or headphone output
Task Introduction
Audio Scene Classification (ASC) is a fundamental task in the field of computer acoustics. Its purpose is to correctly classify a segment of collected audio into corresponding environmental labels, such as dog barking, rain, etc. This project implements embedded deployment of ASC on the ESP32 platform, allowing it to operate with low power in our daily lives. This project will introduce the various processes of implementing audio scene classification using ESP32, including data collection, feature extraction, neural networks, and implementation methods.
Data Collection
Observing the schematic, we can see that the signal received by the Mic is amplified by the operational amplifier U9 and can be adjusted for gain via RV1, ultimately connecting to the ESP32 ADC input. Therefore, the collection part uses DIG ADC1 to measure the voltage at the pin with a sampling rate of 20kHz, and the data is directly copied to memory via DMA (Direct Memory Access) without CPU processing. The driver for this part directly uses the project-developed code, but due to API updates, I forcibly used deprecated library functions, which will generate warnings during compilation. This part of the driver can be upgraded in the future.
Feature Extraction
Feature extraction is the most important step for audio perception. The ultimate goal of this part is to convert a segment of nearly one second of audio into a 30×30 spectrogram.
Next, I will explain how to generate a column of data in this spectrogram.
3.1 Window Function
First, we extract 1×1024 continuous points from the sampled audio data. These points will later undergo Fourier transformation. Before that, to reduce spectral leakage, we will multiply this segment of data by a function that is large in the middle, maximum at 1, and gradually approaches 0 at both ends, called a “window function.” I used the commonly used Hanning window.
3.2 Fast Fourier Transform
Next, we perform a fast Fourier transform on these 1024 points. In other words, the original signal’s horizontal axis is time, reflecting the signal’s intensity at different times. After transformation, we obtain the energy distribution of the audio over various frequencies during that time. As shown in Figure 3.1(a), the left red signal is the time-domain signal before transformation, while the right shows the energy distribution in the frequency domain after transformation.
For the implementation part, I referred to the open-source code for Real FFT (RFFT):https://github.com/willhope/Noise-reduction/blob/master/rfft.c
3.3 Mel Spectrogram
Research shows that human perception of frequency is not linear, and sensitivity to low-frequency signals is greater than that of high-frequency signals. For example, people can easily distinguish between 500 and 1000Hz, but it is difficult to notice the difference between 7500 and 8000Hz. However, the horizontal axis of the Fourier transformed spectrum is linear. At this point, the Mel scale is proposed, which is a nonlinear transformation of Hz. For signals measured in Mel scale, human perception of signals with the same frequency difference is almost the same. Figure 3.1(b) shows the relationship between human perception of pitch and their actual frequencies.
In this project, I used 30 sets of filters, so the original array length of 1024 becomes 30 after transformation. During implementation, I referred to and ported the official STM32 audio processing library, which includes common functions for feature extraction, such as Fourier transformation and Mel spectrogram. Since ST uses ARM’s DSP to accelerate floating-point operations, when porting to ESP32, I had to replace all with standard math functions, manually add implementations for missing functions, and finally resolve various compilation issues, which took considerable effort.
It is worth mentioning that the ESP official also has a DSP library, but due to time constraints, I couldn’t study it in detail.
Figure 3.1 (a) Fourier Transform (b) Mel Spectrogram
3.4 Program Flow
As shown in Figure 3.2, to enable the program to implement the above process, I first created 6 DMA receive buffers of length 256 uint16. When an interrupt occurs, the data of the previous 1024 points is extracted and normalized to floating-point numbers, performing a fast Fourier transform to obtain the Fourier spectrum. Then, a length of 30 array is calculated as a column of the Mel spectrogram. Every 512 sampling points, a column of the Mel spectrogram is calculated from the 1024 points. Once 40 columns of results are calculated, this data can be used for subsequent classification.
ESP32 Signal Processing Flowchart
Model Training
Model training is completed on a personal computer using Python scripts and the Pytorch deep learning framework.
4.1 Dataset
We used the ESC-10 dataset as training data. ESC stands for Environmental Sound Classification, and ESC10 contains 10 audio categories, each with 40 audio segments. The dataset download link:
https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/YDEPUT
Additionally, commonly used datasets include UrbanSound8k, etc. After downloading, we use the above feature extraction process to extract features, obtaining the spectrogram for each audio segment. However, in the Python environment, using the librosa library, only two lines of code are needed to complete the Fourier transformation and Mel spectrogram operations.
4.2 Model Structure
After feature extraction, the audio classification problem is transformed into an image classification problem. We use a relatively simple CNN convolutional neural network for classification. Due to the limited memory of the ESP32 platform and the fact that Hehe uses a chip model without 2M PSRAM, which only has 320k of built-in SRAM, the network size cannot be too large. Generally, in embedded platforms, the memory size limits the network scale, while CPU performance determines how fast the network can run. For visualizing the CNN model structure, I recommend referring to the following website:
https://poloclub.github.io/cnn-explainer/
Model Deployment
After training the model, the same CNN model structure can be implemented in ESP32 using C language. Then, parameters from the trained Pytorch model can be exported and compiled into the program as arrays, achieving model deployment. The ESP official also launched a more advanced deep learning library that can directly convert and deploy trained models and provides a series of tools, but due to personal time constraints, I still wrote my own model inference code.
Program Flowchart
Since the program uses the previous author’s code as a framework, it still retains functions such as DAC signal generator. The specific operations are as follows:
Button 1: DAC switch (valid when ADC is off);
ADC digital filter parameter settings (valid when ADC is on)
Button 2: ADC switch (valid when DAC is off)
Button 3: When ADC is on: ADC mode adjustment (detailed in the following text); When DAC is on: DAC volume adjustment
Button 4: DAC frequency switch
The following diagram highlights the program logic when the ADC mode is active. When the ADC is turned on, the ADC_Task task will start executing and continuously respond to ADC DMA interrupts. Then, it executes the corresponding code based on one of the four modes selected by the ADC button.
Key Code
The following code illustrates the neural network inference process and reflects the structure of the neural network. The functions called in the code are self-implemented C language functions for convolution, pooling, etc.
xSemaphoreTake( xMutexSpec, portMAX_DELAY );conv2d_relu(&spec, 1, 30, 40, feature1, 10, conv1_1_w, conv1_1_b);xSemaphoreGive( xMutexSpec );
conv2d_relu(feature1, 10, 28, 38, feature2, 10, conv1_2_w, conv1_2_b);swap_feature(feature1, feature2);
max_pool2d(feature1, 10, 26, 36, feature2);swap_feature(feature1, feature2);
conv2d_relu(feature1, 10, 13, 18, feature2, 10, conv2_1_w, conv2_1_b);swap_feature(feature1, feature2);
conv2d_relu(feature1, 10, 11, 16, feature2, 10, conv2_2_w, conv2_2_b);swap_feature(feature1, feature2);
max_pool2d(feature1, 10, 9, 14, feature2);swap_feature(feature1, feature2);
flatten_fc(feature1, 10, 4, 7, out_neurons, 10, fc_w, fc_b);Neural network inference code, trained for 10 epochs, with approximately 2000 one-second sound segments per epoch, and a batch size of 10.
for epoch in range(num_epochs): # loop over the dataset multiple times
running_loss = 0.0 correct = 0 for i, data in enumerate(asc_dataloader, 0): # get the inputs; data is a list of [inputs, labels] inputs, labels = data labels = torch.tensor(labels, dtype=torch.long)
# zero the parameter gradients optimizer.zero_grad()
# forward + backward + optimize outputs = model(inputs) loss = criterion(outputs, labels) loss.backward()
optimizer.step()
_, preds = torch.max(outputs, 1) # show_images(inputs[:6], labels[:6], preds[:6]) correct += (preds == labels).sum()
# print statistics running_loss += loss.item() if i % 1 == 0: # print every 2000 mini-batches print('Epoch [{}/{}], Step [{}/{}], Loss(Avg): {:.3f}' .format(epoch + 1, num_epochs, i + 1, len(asc_dataloader), running_loss / 1)) running_loss = 0.0 print("Epoch {}, Acc: {}/{}".format(epoch + 1, correct, asc_dataset.__len__())) correct = 0Results Display
Click “Read the original text” to view the project