Sound Source Localization and Classification System Based on Orange Pi: Software Design and Implementation

1. Introduction

With the rapid development of artificial intelligence and Internet of Things technologies, embedded intelligent systems are increasingly applied in fields such as security monitoring, smart homes, industrial inspection, and human-computer interaction. Sound source localization and sound classification technologies, as key technologies for environmental perception, enable machines to “hear and locate” and “identify sounds,” which have high research value and broad application prospects.

Traditional sound source localization and classification systems are often based on high-performance PCs or workstations, which have issues such as high costs, large power consumption, bulky size, and difficulty in deployment. However, ARM architecture-based embedded platforms, such as Raspberry Pi and Orange Pi, provide an ideal hardware foundation for building lightweight, low-power, and offline-capable intelligent acoustic perception systems due to their low cost, compact size, low power consumption, and sufficient computing power.

This project aims to develop a complete sound source localization and sound signal classification system based on the Orange Pi hardware platform. The user has completed about 60% of the hardware development work, including the construction of the microphone array, system configuration of the Orange Pi, and basic driver writing. This solution will focus on the remaining software part, detailing how to use tools such as Python and scikit-learn to accomplish four core tasks: array microphone sound source localization, sound signal classification based on wavelet transform-CEEMDAN-LSTM, embedded system optimization, and GUI interface development, ultimately forming a fully functional, efficient, and user-friendly integrated system.

2. Overall System Architecture Design

Before starting the specific implementation, we need to conduct a top-level design of the software architecture of the entire system to ensure low coupling and high efficiency among the modules.

2.1 Hardware Foundation Review

The hardware foundation provided by the user typically includes:

Core Processing Unit: A certain model of Orange Pi (such as Orange Pi 4B, Orange Pi Zero 2, etc.), equipped with an ARM Cortex-A series multi-core processor, running a Linux operating system.
Acoustic Sensing Unit: An array consisting of multiple microphones (usually 4 or 8), with a known geometric structure (such as linear, circular, square). The microphones are connected to the Orange Pi via I2S interface or USB sound card.
Other Peripheral Devices: May include screens for displaying results, LED lights for indicating status, network interfaces or USB interfaces for data transmission, etc.

2.2 Software Architecture

The software of this system adopts a layered architecture, which mainly includes:

Hardware Abstraction Layer (HAL):

Responsible for interacting with the underlying hardware drivers, providing a unified API to read raw audio data (PCM format) from the multi-channel microphone array.
May involve using libraries such as alsa-lib, pyaudio, or sounddevice to configure audio collection parameters (sampling rate, quantization bits, number of channels, block size) and data reading.

Signal Processing and Algorithm Core Layer:

Sound Source Localization Module: Receives multi-channel audio data and calculates the time difference of arrival (TDOA) of the sound source reaching different microphones using algorithms such as Generalized Cross-Correlation (GCC-PHAT) and Steered Response Power (SRP-PHAT), and then calculates the direction (DOA) or position (X, Y, Z) of the sound source based on the array geometry.
Sound Classification Module: Receives single-channel or beamformed audio data, performs preprocessing, feature extraction, and uses a trained LSTM model for classification recognition. The feature extraction integrates two advanced methods: wavelet transform and CEEMDAN.

Model Training Layer (usually completed on a development PC):

This is an offline process. A large-scale sound dataset (such as UrbanSound8K, ESC-50) is used to train and validate the CEEMDAN-LSTM classification model.
The trained model parameters will be exported and deployed to the Orange Pi.

Application Logic and GUI Layer:

Business Logic: Coordinates and schedules sound source localization and sound classification tasks, manages data flow, and processes user input.
GUI Interface: Develops a graphical user interface based on the PyQt or Tkinter framework, displaying real-time audio waveforms, sound source localization results (such as polar coordinate graphs, position points), classification results, system status, and other information.

System Service Layer:

Responsible for system auto-start, log recording, resource monitoring (CPU, memory usage), network communication, and other functions to ensure stable and reliable system operation.

The data flow of the entire system is: microphone array collects data -> HAL layer reads -> (copy 1) sound source localization module -> obtains azimuth/elevation angle -> GUI display; (copy 2) sound classification module (optional first perform beamforming to enhance SNR) -> preprocessing -> feature extraction -> LSTM classification -> obtains category label -> GUI display.

3. Module One: Array Microphone Sound Source Localization Implementation

Sound source localization is one of the core functions of the system, aiming to estimate the direction or position of the sound source relative to the microphone array in space.

3.1 Algorithm Selection: GCC-PHAT

Among many TDOA estimation algorithms, Generalized Cross-Correlation – Phase Transform (GCC-PHAT) is very suitable for implementation on embedded platforms like Orange Pi due to its robustness in reverberant environments and relatively low computational complexity.

The core idea of GCC-PHAT is to emphasize the phase information of the cross-correlation spectrum while weakening the influence of reverberation and noise, thus estimating the time difference more accurately. The calculation steps are as follows:

Preprocessing: Pre-emphasize, frame, and window (e.g., Hanning window) the two microphone signals x1(n) and x2(n).
Fourier Transform (FFT): Convert the two signals to the frequency domain to obtain X1(f) and X2(f).
Calculate Cross-Power Spectrum: RX1X2(f)=X1(f)⋅X2∗(f), where ∗ indicates the conjugate complex.
PHAT Weighting: Φ(f)=1∣RX1X2(f)∣Φ(f)=∣RX1X2 is crucial as it normalizes the amplitude of all frequency components to 1, retaining only the phase information.
Calculate Generalized Cross-Correlation: RX1X2ττττ(τ)=Φ−1[Φ(f)⋅RX1X2(f)]RPHAT(τ)F−1Φ(f⋅R(f)
Find Peaks: Find the position of the maximum peak in RX1X2PHATτττ(τ)RPHAT(τ)τpeak which is the estimated time difference τ^12τ^.

For an array of M microphones, it is necessary to calculate RX2CM2 for the TDOA of microphone combinations.

3.2 Position Calculation

Once the TDOA estimates τ^ijijτ^ are obtained, the direction of the sound source (DOA) can be calculated based on the geometric model of the microphone array.

Taking the simplest dual-microphone linear array as an example: assuming the distance between microphones is d and the speed of sound is c, the estimated TDOA is τ^τ^, then the azimuth angle of the sound source θ (relative to the array normal) can be calculated using the following formula:sin⁡θ=d⋅τ^csinθ=dc⋅τ^ It should be noted that τ^τ^ has both positive and negative values, so the range of θθ is [−900∘,900∘][−900∘,900∘], which presents Front-Back Ambiguity (ambiguity between front and back).

For arrays with more microphones (such as circular arrays), more accurate DOA estimates can be obtained by solving overdetermined equations using methods such as least squares, and even directly estimating the 2D or 3D coordinates of the sound source.

3.3 Key Points of Python Code Implementation

import numpy as np
from scipy import signal
from scipy.fft import fft, ifft
import math

def gcc_phat(sig1, sig2, fs, interp=1):
    """
    Calculate the time difference between two signals using GCC-PHAT
    Parameters:
        sig1, sig2: Input signals
        fs: Sampling rate
        interp: Interpolation factor to improve peak detection accuracy
    Returns:
        tau: Estimated time difference (seconds)
        corr: Cross-correlation sequence
    """
    n = len(sig1) + len(sig2)

    # FFT
    SIG1 = fft(sig1, n)
    SIG2 = fft(sig2, n)

    # Calculate cross-power spectrum
    R = SIG1 * np.conj(SIG2)

    # PHAT weighting
    R_phat = R / (np.abs(R) + 1e-10)  # Add small constant to avoid division by zero

    # IFFT
    cc = np.real(ifft(R_phat, n))

    # Shift cross-correlation sequence to center zero delay
    cc = np.fft.fftshift(cc)

    # Interpolation to improve accuracy
    if interp > 1:
        cc = signal.resample(cc, interp * n)

    # Find the offset of the maximum peak
    max_shift = np.argmax(np.abs(cc)) - (interp * n // 2)

    # Calculate time difference
    tau = max_shift / (interp * fs)

    return tau, cc

# Assume we have a 4-microphone circular array with known geometric positions
mic_positions = np.array([...])  # Shape (4, 2) or (4, 3)

def estimate_doa(tdoa_vector, mic_positions, c=343):
    """
    Estimate the direction of arrival (DOA) based on TDOA vector and microphone positions (2D)
    This is a simplified example; actual calculation is more complex.
    """
    # Here, specific calculation algorithms need to be written based on array geometry
    # For example, for a linear array:
    d = np.linalg.norm(mic_positions[1] - mic_positions[0])  # Microphone distance
    theta = np.arcsin((c * tdoa_vector[0, 1]) / d)  # Taking the first pair of microphones as an example
    return np.rad2deg(theta)

# In the main loop

def audio_callback(indata, frames, time, status):
    """
    Audio stream callback function
    """
    if status:
        print(f"Audio stream status: {status}")

    # indata shape is (frames, n_channels)
    channels_data = indata.T  # Convert to (n_channels, frames)

    # Calculate TDOA between all microphone pairs
    n_mics = channels_data.shape[0]
    tdoa_matrix = np.zeros((n_mics, n_mics))

    for i in range(n_mics):
        for j in range(i + 1, n_mics):
            tau, _ = gcc_phat(channels_data[i], channels_data[j], fs=16000, interp=16)
            tdoa_matrix[i, j] = tau
            tdoa_matrix[j, i] = -tau

    # Select a pair of key microphones or use all information to calculate DOA
    # For example, using microphones 0 and 1
    estimated_angle = estimate_doa(tdoa_matrix, mic_positions)

    # Pass angle results to GUI or other modules
    # ...

Embedded Optimization: Directly using the above code on Orange Pi may not be efficient. To achieve real-time performance, we can:

Fixed-point FFT: Use the pyfftw library or ARM platform optimized FFT libraries (such as Ne10).
Reduce Sampling Rate and Points: Use a lower sampling rate (e.g., 16kHz) and shorter frame length while meeting requirements.
Cython or C Extensions: Write the most computation-intensive GCC-PHAT loop in Cython or C, compiling it into a Python extension module.

4. Module Two: Wavelet Transform + CEEMDAN + LSTM Sound Signal Classification Implementation

Sound classification is another core function of the system, aiming to identify the category of sound events (such as “car honking,” “dog barking,” “knocking on the door,” etc.).

4.1 Technical Route: Hybrid Feature Extraction and Deep Learning

Traditional MFCC features perform well in many scenarios, but their representation ability is sometimes insufficient for non-stationary and non-linear complex environmental sounds. We adopt a hybrid feature extraction method that integrates wavelet transform and CEEMDAN, and input it into an LSTM network for classification, aiming to achieve better performance.

Wavelet Transform (WT): Good at processing non-stationary signals, providing good resolution in both time and frequency domains, and can extract detailed information of signals at different scales and positions.
CEEMDAN (Complete Ensemble Empirical Mode Decomposition with Adaptive Noise): An improved algorithm of EMD that effectively overcomes the mode mixing problem, adaptively decomposing complex signals into a series of intrinsic mode functions (IMFs), each containing local features of the original signal at different time scales.
LSTM (Long Short-Term Memory Network): A variant of recurrent neural networks (RNNs) that is particularly good at processing and classifying sequential data (such as audio time series signals), capable of learning long-term dependencies.

Process: Audio signal -> (Branch 1) Wavelet Transform -> Extract wavelet coefficient statistical features -> Feature fusion -> LSTM -> Classification result. -> (Branch 2) CEEMDAN -> Extract IMF component statistical features ->

4.2 Feature Extraction Process

Preprocessing: Pre-emphasize, frame, silence removal, and normalization of the audio signal.
Wavelet Feature Extraction:

Select appropriate wavelet basis (e.g., ‘db4’) and decomposition levels (e.g., 5 levels).
Perform discrete wavelet transform (DWT) on each frame of the signal to obtain approximation coefficients and detail coefficients.
Calculate statistical features of coefficients at each level as part of the feature vector: mean, standard deviation, energy, entropy, etc.

CEEMDAN Feature Extraction:

Perform CEEMDAN decomposition on each frame of the signal to obtain several IMF components and a residue.
Calculate statistical features of the first few important IMF components: mean, standard deviation, energy, zero-crossing rate, etc.

Feature Fusion and Normalization: Concatenate wavelet features and CEEMDAN features into a long feature vector, then standardize (e.g., StandardScaler) to eliminate dimensional influence.

4.3 LSTM Model Construction and Training

# The following code is usually run on a development PC to train the model
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense, Dropout, Input
from tensorflow.keras.optimizers import Adam
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.model_selection import train_test_split
from PyEMD import CEEMDAN  # Requires installation of PyEMD library
import pywt

# 1. Feature extraction function (example)
def extract_hybrid_features(audio_frame, fs, ceemdan_max_imf=5, wavelet='db4', level=5):
    """
    Extract wavelet + CEEMDAN hybrid features
    """
    features = []

    # --- Wavelet Features ---
    coeffs = pywt.wavedec(audio_frame, wavelet, level=level)
    for i, coeff in enumerate(coeffs):
        features.append(np.mean(coeff))
        features.append(np.std(coeff))
        features.append(np.sum(coeff**2))  # Energy
    # ... can add more statistics

    # --- CEEMDAN Features ---
    ceemdan = CEEMDAN()
    imfs = ceemdan(audio_frame, max_imf=ceemdan_max_imf)
    for i, imf in enumerate(imfs[:ceemdan_max_imf]):  # Only take the first few IMFs
        features.append(np.mean(imf))
        features.append(np.std(imf))
        features.append(np.sum(imf**2))
        features.append(np.mean(0.5 * (np.sign(imf[1:]) - np.sign(imf[:-1]))))  # Zero-crossing rate approximation
    # ... 

    return np.array(features)

# 2. Load dataset and extract all features (example using UrbanSound8K)
# Assume audio files are loaded into X_audio (list) and labels y
all_features = []
for audio in X_audio:
    feat = extract_hybrid_features(audio, fs=16000)
    all_features.append(feat)

X = np.array(all_features)
y = np.array(y)

# 3. Data preprocessing
le = LabelEncoder()
y_encoded = le.fit_transform(y)  # Convert string labels to integers
# Assume multi-class classification, need one-hot
# y_categorical = to_categorical(y_encoded)

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

X_train, X_test, y_train, y_test = train_test_split(X_scaled, y_encoded, test_size=0.2, random_state=42)

# 4. Build LSTM model
# Note: Our features are no longer raw waveforms, but extracted statistical feature vectors.
# Therefore, we need to treat the feature vector as one time step (or reshape it into multiple virtual time steps).
# Here we choose to treat it as a single time step sequence.

n_features = X_train.shape[1]
n_classes = len(np.unique(y_encoded))

model = Sequential()
# Input shape: (None, 1, n_features) -> (batch size, time steps, feature dimension)
# We only have one time step, so we need to reshape
model.add(Input(shape=(1, n_features)))
model.add(LSTM(128, return_sequences=False))  # Can try stacking LSTM layers
model.add(Dropout(0.5))
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(n_classes, activation='softmax'))

model.compile(optimizer=Adam(learning_rate=0.001),
              loss='sparse_categorical_crossentropy',  # Use this if y is integer labels
              # loss='categorical_crossentropy',  # Use this if y is one-hot
              metrics=['accuracy'])

model.summary()

# 5. Train the model
history = model.fit(
    X_train.reshape(-1, 1, n_features),  # Reshape for LSTM input format
    y_train,
    epochs=100,
    batch_size=32,
    validation_data=(X_test.reshape(-1, 1, n_features), y_test),
    verbose=1,
    callbacks=[tf.keras.callbacks.EarlyStopping(patience=10, restore_best_weights=True)]
)

# 6. Evaluate the model
test_loss, test_acc = model.evaluate(X_test.reshape(-1, 1, n_features), y_test, verbose=0)
print(f'\nTest accuracy: {test_acc}')

# 7. Save the model and Scaler
model.save('sound_classification_lstm_model.h5')
import joblib
joblib.dump(scaler, 'feature_scaler.save')
joblib.dump(le, 'label_encoder.save')

4.4 Embedded Deployment and Inference

The trained model and Scaler will be ported to the Orange Pi.

# Inference code on Orange Pi
from tensorflow.keras.models import load_model
import joblib
import numpy as np

class SoundClassifier:
    def __init__(self, model_path, scaler_path, label_encoder_path):
        self.model = load_model(model_path)
        self.scaler = joblib.load(scaler_path)
        self.le = joblib.load(label_encoder_path)

    def preprocess_and_predict(self, audio_frame):
        """
        audio_frame: One-dimensional numpy array, mono audio data
        """
        # 1. Extract hybrid features (using the same function and parameters as during training)
        features = extract_hybrid_features(audio_frame, fs=16000)

        # 2. Standardize features
        features_scaled = self.scaler.transform(features.reshape(1, -1))

        # 3. Reshape for LSTM input format (1, 1, n_features)
        features_reshaped = features_scaled.reshape(1, 1, -1)

        # 4. Predict
        prediction = self.model.predict(features_reshaped, verbose=0)
        predicted_class_index = np.argmax(prediction, axis=1)[0]
        predicted_label = self.le.inverse_transform([predicted_class_index])[0]
        confidence = prediction[0][predicted_class_index]

        return predicted_label, confidence

# Use in the main program
classifier = SoundClassifier('sound_classification_lstm_model.h5', 'feature_scaler.save', 'label_encoder.save')

# In audio callback or processing thread

def classification_callback(audio_data):
    label, conf = classifier.preprocess_and_predict(audio_data)
    print(f"Predicted: {label} with confidence {conf:.2f}")
    # Send results to GUI

Embedded Optimization:

Model Quantization: Use TensorFlow Lite to convert Keras models to TFLite format and perform dynamic range quantization or full integer quantization, which can significantly reduce model size and improve inference speed.

# Convert model
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]  # Dynamic range quantization
tflite_model = converter.convert()
with open('model_quantized.tflite', 'wb') as f:
    f.write(tflite_model)

Use TFLite Interpreter: Use TFLite interpreter on Orange Pi to load the quantized model for inference, which is more efficient.
Multithreading: Place feature extraction and model inference in separate threads to avoid blocking audio capture and GUI main thread.

5. Module Three: Embedded System Development Completion

The user has completed 60% of the embedded development, and this section mainly focuses on system integration, optimization, and stability.

5.1 System Configuration and Dependency Management

Operating System: Orange Pi typically runs Armbian or Debian-based systems. Ensure the system is updated and install necessary dependency libraries:
```
sudo apt-get update
sudo apt-get install python3-pip python3-dev libatlas-base-dev libportaudio2 libportaudiocpp0 portaudio19-dev
```
Python Environment: It is recommended to use virtualenv to create a virtual environment to isolate project dependencies.
```
python3 -m venv orange_pi_venv
source orange_pi_venv/bin/activate
```
Install Python Packages: Install the required packages for the project in the virtual environment. Since the platform is ARM architecture, some packages may need to be compiled from source (such as numpy, scipy), which can be very time-consuming. Prioritize finding precompiled wheel files (e.g., from the piwheels.org repository).
```
pip install numpy scipy scikit-learn tensorflow tensorflowlite pyaudio sounddevice pyqt5 matplotlib pyemd
# If tensorflow does not have an ARM precompiled version, try tflite-runtime
pip install tflite-runtime
```

5.2 Audio Capture Module

Use sounddevice or PyAudio libraries to reliably read the microphone array.

import sounddevice as sd

# Configure audio stream parameters
fs = 16000
blocksize = 1024  # Number of samples to process in each callback
channels = 4  # Number of microphones

# Check available devices
print(sd.query_devices())
# Select the correct device index
input_device_index = None  # Usually needs to be specified, e.g., the index of the USB sound card

def audio_callback(indata, frames, time, status):
    # indata shape is (frames, channels)
    if status:
        print(f"Audio stream status: {status}")
    # Put data into queue for localization and classification threads
    # audio_queue.put(indata.copy())

# Create audio input stream
stream = sd.InputStream(
    device=input_device_index,
    samplerate=fs,
    blocksize=blocksize,
    channels=channels,
    callback=audio_callback,
    dtype='float32'
)

# Start and stop stream
try:
    with stream:
        while True:
            # Main loop can do other things or wait for stop signal
            sd.sleep(1000)
except KeyboardInterrupt:
    print("Stopped by user")

5.3 Multithreading Design and Resource Management

To ensure real-time performance, a multithreaded architecture must be adopted.

Main Thread: Responsible for GUI event loop (if the GUI is blocking, such as PyQt) or system control.
Audio Capture Thread: Managed internally by the sounddevice callback function, usually running in a high-priority independent thread.
Sound Source Localization Thread: Retrieves data from the audio queue, performs GCC-PHAT calculations and DOA estimation.
Sound Classification Thread: Retrieves data from the audio queue (possibly from another path with beamforming data), performs feature extraction and model inference.

Use the threading module and queue module for inter-thread communication.

import threading
import queue
from collections import deque

# Create data queue
audio_queue = queue.Queue(maxsize=20)  # Prevent the queue from growing indefinitely and exhausting memory

# Localization thread function
def localization_worker():
    while True:
        try:
            audio_data = audio_queue.get(timeout=1.0)
            # Calculate DOA...
            # Pass results to GUI in a thread-safe manner
            # gui_signal.update_angle.emit(estimated_angle)
        except queue.Empty:
            continue

# Classification thread function
def classification_worker():
    while True:
        # May retrieve data from another queue, or take from the same queue at a lower frequency
        try:
            audio_data_for_classification = classification_queue.get(timeout=2.0)
            # Perform classification...
            # gui_signal.update_class.emit(predicted_label, confidence)
        except queue.Empty:
            continue

# In audio callback, put into queue
def audio_callback(indata, frames, time, status):
    # ...
    try:
        audio_queue.put_nowait(indata.copy())
    except queue.Full:
        print("Audio queue is full, dropping block.")

# Start worker threads
loc_thread = threading.Thread(target=localization_worker, daemon=True)
class_thread = threading.Thread(target=classification_worker, daemon=True)
loc_thread.start()
class_thread.start()

5.4 System Auto-Start and Daemon

Write a shell script to automatically run our Python program at system startup.

#!/bin/bash
# /home/orange/start_system.sh

cd /path/to/your/project
source orange_pi_venv/bin/activate
python3 main.py

Use systemd to create a service unit file to achieve auto-start and process daemonization.

# /etc/systemd/system/sound-system.service
[Unit]
Description=Sound Localization and Classification System
After=network.target sound.target

[Service]
User=orange
Group=orange
WorkingDirectory=/path/to/your/project
ExecStart=/bin/bash /home/orange/start_system.sh
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Then enable the service:

sudo systemctl enable sound-system.service
sudo systemctl start sound-system.service

6. Module Four: GUI Development

The GUI is the window for user interaction with the system and needs to intuitively display core information.

6.1 Framework Selection: PyQt5

PyQt5 is powerful, cross-platform, and flexible in UI design, suitable for developing complex desktop applications.

6.2 Interface Design

The main interface mainly includes the following areas:

Waveform Display Area: Real-time plotting of audio waveforms for at least one channel (using matplotlib or PyQtGraph).
Sound Source Localization Result Display Area:

Polar Coordinate Graph: Displays the current direction angle of the sound source with a dynamically updating point or pointer.
Numerical Display: Displays azimuth and elevation angles (if any) in numerical form.

Sound Classification Result Display Area: Displays the currently recognized sound categories and their confidence levels in labels or lists.

Control Area: Start/Stop buttons, system status indicator lights, parameter setting entries, etc.

6.3 Code Implementation Key Points

import sys
from PyQt5.QtWidgets import (QApplication, QMainWindow, QWidget, QVBoxLayout, QHBoxLayout,
                             QLabel, QPushButton, QGroupBox, QStatusBar)
from PyQt5.QtCore import Qt, QTimer, pyqtSignal, QObject
import pyqtgraph as pg  # For high-performance plotting
import numpy as np

# Define a signal class for inter-thread communication
class GuiSignals(QObject):
    update_waveform = pyqtSignal(np.ndarray)  # Pass waveform data
    update_angle = pyqtSignal(float)  # Pass angle data
    update_classification = pyqtSignal(str, float)  # Pass category and confidence

class MainWindow(QMainWindow):
    def __init__(self):
        super().__init__()
        self.signals = GuiSignals()
        self.initUI()
        self.connectSignals()

    def initUI(self):
        self.setWindowTitle('Orange Pi Sound Source Localization and Classification System')
        self.setGeometry(100, 100, 1200, 800)

        central_widget = QWidget()
        self.setCentralWidget(central_widget)
        main_layout = QHBoxLayout(central_widget)

        # Left side: Control and status panel
        left_panel = QVBoxLayout()
        control_group = QGroupBox("System Control")
        control_layout = QVBoxLayout()
        self.start_btn = QPushButton("Start")
        self.stop_btn = QPushButton("Stop")
        self.stop_btn.setEnabled(False)
        control_layout.addWidget(self.start_btn)
        control_layout.addWidget(self.stop_btn)
        control_group.setLayout(control_layout)
        left_panel.addWidget(control_group)

        # ... can add other status information displays
        main_layout.addLayout(left_panel, 1)

        # Right side: Data display panel
        right_panel = QVBoxLayout()

        # 1. Waveform graph
        wave_group = QGroupBox("Real-time Waveform")
        wave_layout = QVBoxLayout()
        self.wave_plot = pg.PlotWidget()
        self.wave_plot.setYRange(-1, 1)
        self.wave_curve = self.wave_plot.plot(pen='y')
        wave_layout.addWidget(self.wave_plot)
        wave_group.setLayout(wave_layout)
        right_panel.addWidget(wave_group)

        # 2. Polar coordinate graph (sound source localization)
        loc_group = QGroupBox("Sound Source Localization")
        loc_layout = QVBoxLayout()
        self.polar_plot = pg.PlotWidget()
        self.polar_plot.setAspectLocked(True)
        self.polar_plot.setXRange(-1.2, 1.2)
        self.polar_plot.setYRange(-1.2, 1.2)
        # Draw polar coordinate grid (omitted)
        self.angle_scatter = pg.ScatterPlotItem(size=15, pen=pg.mkPen(None), brush=pg.mkBrush(255, 0, 0, 200))
        self.polar_plot.addItem(self.angle_scatter)
        self.angle_label = QLabel("Azimuth: -- °")
        loc_layout.addWidget(self.polar_plot)
        loc_layout.addWidget(self.angle_label)
        loc_group.setLayout(loc_layout)
        right_panel.addWidget(loc_group)

        # 3. Classification results
        class_group = QGroupBox("Sound Classification")
        class_layout = QVBoxLayout()
        self.class_label = QLabel("Category: --")
        self.confidence_label = QLabel("Confidence: --")
        class_layout.addWidget(self.class_label)
        class_layout.addWidget(self.confidence_label)
        class_group.setLayout(class_layout)
        right_panel.addWidget(class_group)

        main_layout.addLayout(right_panel, 3)

        self.statusBar().showMessage('System Ready')

    # Connect button signals
        self.start_btn.clicked.connect(self.start_system)
        self.stop_btn.clicked.connect(self.stop_system)

    def connectSignals(self):
        # Connect custom signals to slot functions
        self.signals.update_waveform.connect(self.update_waveform_plot)
        self.signals.update_angle.connect(self.update_angle_display)
        self.signals.update_classification.connect(self.update_classification_display)

    def update_waveform_plot(self, data):
        """ Update waveform graph """
        # Assume data is single-channel data
        x = np.arange(len(data))
        self.wave_curve.setData(x, data)

    def update_angle_display(self, angle_deg):
        """ Update sound source localization display """
        self.angle_label.setText(f"Azimuth: {angle_deg:.1f} °")
        # Update point on polar coordinate graph
        angle_rad = np.deg2rad(angle_deg)
        x = np.cos(angle_rad)
        y = np.sin(angle_rad)
        self.angle_scatter.setData([x], [y])

    def update_classification_display(self, label, confidence):
        """ Update classification result display """
        self.class_label.setText(f"Category: {label}")
        self.confidence_label.setText(f"Confidence: {confidence:.2%}")

    def start_system(self):
        # Trigger audio stream start, worker thread start
        self.statusBar().showMessage('System Running...')
        self.start_btn.setEnabled(False)
        self.stop_btn.setEnabled(True)
        # ... Call start function of background logic

    def stop_system(self):
        # Trigger audio stream stop, worker thread end
        self.statusBar().showMessage('System Stopped')
        self.start_btn.setEnabled(True)
        self.stop_btn.setEnabled(False)
        # ... Call stop function of background logic

if __name__ == '__main__':
    app = QApplication(sys.argv)
    main_window = MainWindow()
    main_window.show()

    # Create and start background threads and audio stream, and pass main_window.signals to them

    sys.exit(app.exec_())

6.4 Cross-Thread GUI Updates

Important: All updates to GUI components must be performed in the main thread. We use PyQt’s pyqtSignal mechanism, where worker threads emit signals, and slot functions in the main thread are responsible for receiving signals and updating the UI, which is thread-safe.

7. System Integration, Testing, and Optimization

7.1 System Integration

Integrate all the above modules into a main program main.py:

Initialize GUI.
Initialize sound source locator and sound classifier.
Create audio stream and data queue.
Start worker threads.
Connect worker threads with GUI signals.

7.2 Testing Plan

Unit Testing: Test GCC-PHAT function, feature extraction function, model inference function, etc., separately.
Functional Testing:

Sound Source Localization Testing: In anechoic chamber or quiet environment, use a speaker to play white noise or pulse signals from different azimuths to check whether the estimated angles of the system are accurate.
Sound Classification Testing: Play sound files of known categories to check whether the system’s recognition results are correct.

Performance Testing:

Real-time Testing: Use the system clock to measure the total delay from audio capture to result display, ensuring it meets real-time requirements (usually <100ms).
Resource Consumption Testing: Use tools like top, htop to monitor CPU and memory usage. Ensure stable operation on Orange Pi without crashing due to resource exhaustion.

Robustness Testing: Test in environments with background noise, reverberation, and multiple sound sources to assess the degree of performance degradation of the system.

7.3 Performance Optimization Summary

Algorithm Level: Choose algorithms with low computational complexity (such as GCC-PHAT), adjust frame length, overlap rate, and other parameters.
Code Level: Use vectorized operations (NumPy), avoid Python loops. Use C extensions (Cython) for performance bottleneck parts.
Model Level: Use lightweight models (such as TFLite quantized models), reduce the number of layers and neurons.
System Level: Use multithreading to fully utilize multi-core CPUs. Adjust thread priorities.
Compiled Installation: Compile and install optimized scientific computing libraries (such as OpenBLAS) for ARM platforms.

8. Conclusion and Outlook

This project successfully designed and implemented an integrated intelligent acoustic perception system for sound source localization and environmental sound classification based on the Orange Pi embedded platform. The software part completed all work from low-level audio capture, core algorithms (GCC-PHAT, wavelet-CEEMDAN-LSTM) implementation, to upper-level GUI interface development. The system can display sound waveforms in real-time, estimate sound source directions, and recognize sound categories, achieving the expected goals.

Future Outlook:

Deep Learning End-to-End Localization: Explore using deep learning models (such as CNN, CRNN) to directly estimate sound source directions from multi-channel raw audio or features, avoiding complex TDOA geometric calculations.
Multi-Source Processing: Upgrade algorithms to support simultaneous localization and separation of multiple sound sources.
Model Lightweighting and Knowledge Distillation: Further compress models and explore network structures more suitable for embedded devices (such as MobileNet, SqueezeNet applied to acoustic features).
Cloud Collaboration: Offload complex model inference tasks to cloud servers, with the Orange Pi only responsible for front-end capture and simple processing, achieving more complex functions.
Application Expansion: Apply the system specifically to smart home control (voice control), security monitoring (abnormal sound detection and localization), video conference tracking, and other specific scenarios.

This system demonstrates the feasibility of implementing complex AI algorithms on resource-constrained embedded devices, providing an excellent example for building low-cost, low-power, intelligent edge perception devices.