✅ Author Profile: A research enthusiast and Matlab simulation developer, skilled in data processing, modeling simulation, program design, obtaining complete code, reproducing papers, and scientific simulation.
🍎 Previous reviews, follow the personal homepage:Matlab Research Studio
🍊 Personal motto: Seek knowledge through investigation, complete Matlab code and simulation consultation available via private message.
🔥 Content Introduction
As the most natural and efficient means of information exchange for humans, voice processing technology plays an irreplaceable role in fields such as intelligent assistants, speech recognition, speech synthesis, human-computer interaction, medical diagnosis (e.g., speech disease screening), and security monitoring (e.g., voiceprint recognition). With the rapid development of artificial intelligence technology, voice processing has gradually shifted from traditional signal processing to a research paradigm that integrates deep learning and signal processing. Currently, the core demands of voice processing focus on robustness in complex environments, multilingual adaptability, performance optimization in low-resource scenarios, and real-time processing efficiency. Innovations in algorithms and methods are key to overcoming these bottlenecks. This article systematically reviews the core algorithms and methods throughout the voice processing workflow, from basic preprocessing to advanced application technologies, analyzing the technical principles, advantages, limitations, and research progress of each stage, providing references for research and applications in related fields.
II. Voice Signal Preprocessing Algorithms
Voice signals are easily affected by environmental noise (e.g., traffic noise, background voices), device interference (e.g., microphone current noise), and transmission loss during the collection process. The core goal of the preprocessing stage is to remove noise, enhance effective signals, and extract high-quality voice features for subsequent processing. The key algorithms in this stage mainly include the following categories:
(1) Noise Suppression Algorithms
Noise suppression is a core part of preprocessing, aiming to separate or suppress noise components from noisy speech while retaining pure speech signals.
- Traditional signal processing methods
- Spectral Subtraction: By estimating the noise spectrum, the noise spectrum is subtracted from the noisy speech spectrum to achieve noise suppression. Its advantages include low computational complexity and strong real-time performance, suitable for stationary noise scenarios (e.g., air conditioning noise); however, it can produce “musical noise” under non-stationary noise (e.g., sudden human voices), leading to speech distortion.
- Wiener Filtering: Based on the minimum mean square error criterion, an adaptive filter is designed to dynamically adjust the filtering coefficients according to the statistical characteristics of speech and noise. Compared to spectral subtraction, Wiener filtering has stronger adaptability to non-stationary noise, but it relies on accurate estimation of the power spectra of speech and noise, and its performance significantly declines in low signal-to-noise ratio (SNR < 5dB) scenarios.
- Kalman Filtering: Models the speech signal as a linear dynamic system, using state estimation and iterative updates to track and suppress noise in real-time. This method is suitable for non-stationary noise but has high computational complexity. Initially, it was mainly used in scenarios with low real-time requirements, but it has gradually been applied to mobile devices through lightweight optimization.
- Deep learning-based methods
- Temporal Convolutional Network (TCN) Noise Suppression: Captures the temporal local correlations of speech signals through one-dimensional convolutional layers, enhancing the model’s ability to process long sequence signals with residual connections, effectively suppressing non-stationary noise while avoiding the “musical noise” problem of traditional methods.
- Recurrent Neural Network (RNN/LSTM/GRU) Noise Suppression: Utilizes the temporal modeling capability of RNNs to learn the dynamic changes of speech and noise, especially suitable for long-term dependency noise scenarios (e.g., alternating speech noise in meetings). Among them, LSTM (Long Short-Term Memory network) addresses the gradient vanishing problem of traditional RNNs through gating mechanisms, significantly outperforming traditional methods in noise suppression accuracy, but real-time performance needs to be optimized through model pruning or quantization.
- Transformer-based Noise Suppression: Based on self-attention mechanisms, it can capture the global temporal correlations of speech signals, performing excellently in complex multi-source noise scenarios (e.g., simultaneous traffic noise and human voice interference), but the model’s parameter count is large, currently mainly used in high-performance computing devices or cloud processing.
(2) Endpoint Detection Algorithms
Voice Activity Detection (VAD) aims to distinguish between “speech segments” and “non-speech segments” (e.g., silence, noise) from continuous signals, providing effective input for subsequent speech recognition, synthesis, etc., and reducing ineffective computations.
- Traditional feature-driven methods
- Energy and Zero-Crossing Rate-based Methods: By calculating the short-term energy (reflecting signal strength) and zero-crossing rate (reflecting signal frequency characteristics) of the signal, thresholds are set to distinguish between speech and non-speech. This method is computationally simple and suitable for high SNR scenarios, but in low SNR, it can easily misjudge noise as speech or miss weak speech signals (e.g., soft speaking).
- Mel-Frequency Cepstral Coefficients (MFCC) and Gaussian Mixture Model (GMM)-based Methods: Extracts MFCC features of speech (simulating human ear auditory characteristics) and models the feature distribution of speech and noise through GMM to achieve probabilistic endpoint detection. Compared to energy-zero-crossing methods, it has stronger robustness but relies on a large amount of labeled data to train the GMM model, and its adaptability is poor in non-stationary noise.
- Deep learning-based methods
- CNN-LSTM Fusion VAD: Utilizes CNN to extract local features of the signal (e.g., spectral differences between noise and speech), combined with LSTM to capture temporal dynamics, outputting endpoint positions through binary classification (speech/non-speech). This method can achieve a detection accuracy of over 90% in low SNR (SNR < 0dB) scenarios, significantly outperforming traditional methods.
- Lightweight VAD Models (e.g., MobileNet-VAD): Designed for mobile devices, using depthwise separable convolutions to reduce model parameters while ensuring detection accuracy, increasing inference speed by 3-5 times, meeting the needs of real-time voice interaction (e.g., smartwatches, headphones).
(3) Feature Extraction Methods
Feature extraction transforms the original speech signal (time domain) into more discriminative feature vectors (frequency domain or time-frequency domain), providing core input for subsequent tasks such as speech recognition and voiceprint recognition. Current mainstream feature extraction methods can be divided into traditional handcrafted features and deep learning automatic features:
- Traditional Handcrafted Features
- Mel-Frequency Cepstral Coefficients (MFCC): Simulates the sensitivity of the human ear to different frequency sounds (more sensitive to low frequencies), extracting 12-16 dimensional feature vectors through Fourier transform, Mel filter bank, and discrete cosine transform (DCT). MFCC is a classic feature in the field of voice processing, widely used in traditional speech recognition systems (e.g., Hidden Markov Model HMM), but it is sensitive to noise and its performance can easily decline in complex environments.
- Linear Predictive Cepstral Coefficients (LPCC): Based on the linear predictive model (LPC) of the speech signal, it extracts cepstral coefficients that reflect the characteristics of the vocal tract by fitting the vocal tract transfer function. LPCC has lower computational complexity than MFCC and stronger representation ability for vowel signals, but weaker discrimination for consonant signals.
- Perceptual Linear Prediction (PLP): Combines the auditory masking effect of the human ear with the linear predictive model, adding equal loudness contours and critical band filtering on the basis of LPCC, showing better robustness to noise than MFCC, suitable for speech processing tasks in low-resource or noisy environments.
- Deep Learning Automatic Features
- Convolutional Neural Network (CNN) Automatic Features: Automatically extracts hierarchical features (e.g., low-frequency resonances, high-frequency consonant features) from the original speech signal or spectrogram through the convolutional and pooling layers of CNN, eliminating the need for manual feature design, gradually replacing traditional MFCC features in speech recognition and voiceprint recognition. For example, in speech recognition tasks, features extracted by CNN are more adaptable to accents and dialects.
- Autoencoder Features: Compresses the speech signal into low-dimensional feature vectors through an encoder, reconstructing the original signal with a decoder, learning the essential features of speech using an unsupervised learning approach. This method is suitable for low-resource scenarios with unlabeled data (e.g., small language speech processing), but the interpretability of the features is weak.
III. Core Voice Processing Technologies and Algorithms
Based on preprocessing, the core technologies of voice processing include speech recognition, speech synthesis, and voiceprint recognition, and the evolution of these technologies’ algorithms directly determines the performance and experience of voice interaction.
(1) Automatic Speech Recognition (ASR) Algorithms
The goal of speech recognition is to convert speech signals into text, making it one of the core technologies for voice interaction. Its algorithm development has gone through the traditional HMM era, the deep learning hybrid model era, and is now entering the end-to-end deep learning era.
- Traditional Hybrid Model (HMM-GMM)
- Principle: Based on the Hidden Markov Model (HMM) to model the temporal dynamics of speech (e.g., transition probabilities of phonemes), combined with Gaussian Mixture Model (GMM) to model the observation probabilities of each hidden state (e.g., distribution of MFCC features), solving the optimal text sequence through dynamic programming (Viterbi algorithm).
- Advantages and Limitations: This method has high recognition accuracy for clear speech (e.g., in laboratory environments) and strong model interpretability; however, it relies on manually designed phoneme dictionaries and features, has poor robustness to noise, accents, and dialects, and the training complexity increases exponentially with vocabulary size, making it difficult to meet the demands of large-scale speech recognition (e.g., real-time subtitles, intelligent customer service).
- Deep Learning Hybrid Model (DNN-HMM)
- Improvement Direction: Replaces GMM with Deep Neural Networks (DNN, such as fully connected networks, CNN) to directly learn the mapping relationship from speech features to phoneme states. DNN can capture nonlinear associations between features, significantly improving robustness to noise and accents, achieving an accuracy increase of 15%-20% in medium vocabulary tasks (e.g., mobile voice dialing) compared to HMM-GMM.
- Typical Architecture: CNN-DNN-HMM, utilizing CNN to extract local features from spectrograms, DNN to model the mapping between features and phonemes, and HMM to handle temporal dependencies, suitable for short speech recognition in noisy environments (e.g., voice commands).
- End-to-End Speech Recognition Models
- Core Idea: Abandoning the traditional “feature extraction – phoneme modeling – decoding” phased process, directly mapping from raw speech signals to text sequences, reducing manual intervention and enhancing model generalization ability. Mainstream end-to-end models can be divided into two categories:
- Connectionist Temporal Classification (CTC) Model: Solves the alignment problem between speech and text (e.g., “length of speech frames ≠ number of text characters”) through the CTC loss function. Typical architectures include “CNN-LSTM-CTC” or “Transformer-CTC”. CTC models perform excellently in long speech recognition (e.g., meeting transcription) but have weak differentiation ability for homophones (e.g., “formula” vs. “offensive”).
- Attention Mechanism (Attention) Model: Based on the Encoder-Decoder architecture, the Encoder encodes the speech signal into context vectors, and the Decoder dynamically focuses on key temporal features of the Encoder through attention mechanisms to directly generate text. This model does not require external alignment data and has a stronger understanding of semantics (e.g., can handle “colloquial expressions”). A typical representative is the “Listen, Attend and Spell (LAS)” model. Currently, attention models combined with Transformers (e.g., Whisper) have achieved high-precision speech recognition across multiple languages and scenarios, with accuracy rates exceeding 85% in noisy environments (e.g., streets, workshops).
(2) Text-to-Speech (TTS) Algorithms
Speech synthesis aims to convert text into natural and fluent speech, widely used in intelligent assistants (e.g., Siri, Xiao Ai), audiobooks, and accessibility aids (e.g., reading for visually impaired individuals). Its technological evolution has progressed from parametric synthesis and concatenative synthesis to the current end-to-end neural speech synthesis.
- Traditional Speech Synthesis Methods
- Parametric Synthesis (e.g., STRAIGHT, HTS): Extracts parameters such as fundamental frequency (F0), spectral envelope, and glottal excitation from speech, synthesizing speech based on speech generation models (e.g., source-filter model). The advantages of this method include strong controllability of synthesized speech (e.g., adjustable speed, intonation) and low data requirements; however, synthesized speech can easily produce a “mechanical feel,” with lower naturalness and difficulty simulating emotional variations in human speech (e.g., joy, sadness).
- Concatenative Synthesis (Unit Selection TTS): Selects speech units (e.g., phonemes, syllables) from a large-scale speech corpus that match the text, generating speech through smooth concatenation. This method has higher naturalness than parametric synthesis, especially excelling in specific speaker imitation (e.g., celebrity voices); however, it relies on high-quality, large-sized corpora, and the flexibility of synthesized speech is poor (e.g., difficult to adjust pronunciation style), with weak handling capabilities for out-of-vocabulary words (e.g., rare characters).
- Neural TTS Methods
- RNN-based TTS: A typical representative is the “Tacotron” model, which encodes text into semantic vectors through an Encoder, generates mel-spectrograms with a Decoder (LSTM), and then converts the spectrograms into speech waveforms through vocoders (e.g., Griffin-Lim). The synthesized speech of the Tacotron model significantly surpasses traditional methods in naturalness, but the speech generated by the vocoder may suffer from “spectral aliasing” issues, leading to a decline in sound quality.
- Transformer-based TTS: Such as “Tacotron 2” and “FastSpeech,” utilizes the self-attention mechanism of Transformers to capture the global semantic associations of text (e.g., grammatical structures in long sentences), combined with adversarial training to enhance speech naturalness. Among them, FastSpeech addresses the slow inference speed issue of RNN-Decoders through a “length regulator,” increasing synthesis speed by over 10 times, meeting real-time interaction needs; FastSpeech 2 further introduces prosodic features such as fundamental frequency and energy, enhancing the emotional expression capability of synthesized speech (e.g., generating angry or gentle speech).
- Zero-Shot TTS: Based on a small amount of speaker voice (e.g., 1-5 minutes), it can synthesize the voice of that speaker. Core technologies include “Speaker Embedding” and “Style Transfer.” For example, using a pre-trained Transformer-TTS model, by extracting the voiceprint features of the target speaker (e.g., x-vector), it integrates them into the synthesis process, achieving “one-click imitation” of specific individuals’ voices, suitable for personalized intelligent assistants and customized audiobooks.
(3) Speaker Recognition Algorithms
Speaker recognition (also known as speaker identification) analyzes the unique voiceprint features in speech signals (determined by physiological characteristics such as vocal cord structure and vocal tract shape) to authenticate or identify the speaker’s identity, applied in security (e.g., voiceprint unlocking), finance (e.g., voice payment), and judicial (e.g., voice evidence tracing) fields. Its core algorithms can be divided into traditional pattern recognition methods and deep learning methods.
- Traditional Speaker Recognition Methods
- Feature Matching-based Methods: Extracts features such as MFCC and LPCC from the speaker’s voice, calculating the feature similarity between the speech to be recognized and the template speech through Dynamic Time Warping (DTW) to achieve identity matching. This method is suitable for small-scale speaker recognition (e.g., within 100 people), but it is sensitive to noise and changes in speech content (e.g., significant feature differences when the same person speaks different texts).
- Model-based Methods: Such as Gaussian Mixture Model – Universal Background Model (GMM-UBM), which models the “general feature distribution of all speakers” through UBM, then adjusts GMM parameters for each speaker to obtain personalized models. GMM-UBM performs excellently in medium-scale (within 1000 people) speaker recognition, but the model parameter count increases with the number of speakers, making it difficult to scale to large-scale scenarios (e.g., millions of users).
- Deep Learning Speaker Recognition Methods
- Embedding-based Methods: Maps speech signals to fixed-dimensional embedding vectors (e.g., 128 dimensions) through deep neural networks (e.g., CNN, LSTM, Transformer), with the property of “compact within class, separated between classes” (i.e., embedding vectors of the same speaker are close, while those of different speakers are far apart). Typical models include:
- x-vector Model: Based on Time Delay Neural Network (TDNN) to extract long-term features of speech, generating embedding vectors through statistical pooling (e.g., mean, standard deviation), showing strong robustness in noisy environments, and is one of the mainstream models for speaker recognition.
- ECAPA-TDNN Model: Introduces channel attention mechanisms and temporal attention mechanisms based on TDNN, enhancing the ability to capture key voiceprint features, achieving over 98% accuracy in short speech (e.g., within 3 seconds) speaker recognition, suitable for rapid identity authentication on mobile devices.
- End-to-End Speaker Recognition Models: Directly map from raw speech signals to speaker identity labels without separate feature extraction steps. For example, an end-to-end model based on Transformers captures global voiceprint features through self-attention mechanisms, performing excellently in large-scale speaker recognition (e.g., tens of millions of users), but requires massive labeled data (e.g., over 10 hours of speech for each speaker) for training.
IV. Challenges and Future Research Directions in Voice Processing
(1) Current Core Challenges
- Insufficient Robustness in Complex EnvironmentsExisting algorithms show significant performance degradation in extreme noise (e.g., industrial workshops, construction sites), reverberation (e.g., large auditoriums, tunnels), or overlapping multiple speakers (e.g., multi-person meetings). For instance, the accuracy of speech recognition may drop below 50% at SNR < -5dB, and the error rate of voiceprint recognition can increase 3-5 times across microphones and devices.
- Difficulties in Adapting to Low-Resource ScenariosScarcity of speech data for small languages (e.g., Kyrgyz, Jingpo) and dialects (e.g., Cantonese, Sichuan dialect) makes it challenging for existing models to adapt; at the same time, limited computing and storage resources on edge devices (e.g., IoT sensors, low-end phones) hinder the deployment of large-scale deep learning models.
- Lack of Natural Interaction CapabilitiesThe emotional expression of speech synthesis still appears stiff, making it difficult to accurately match text semantics (e.g., humorous or sarcastic tones); speech recognition lacks understanding of colloquial expressions (e.g., fillers, speech errors) and context dependencies (e.g., referential relationships like “He went there today”), leading to unnatural interaction experiences.
- Privacy and Security RisksAs a biometric feature, voiceprints are at risk of being forged (e.g., generating false voiceprints through speech synthesis technology) or stolen (e.g., obtaining speech through recording devices). How to achieve “anti-forgery voiceprint recognition” and “privacy-protecting voice processing” (e.g., federated learning, homomorphic encryption) becomes a key issue.
(2) Future Research Directions
- Robust Voice Processing Technologies
- Integrating multimodal information (e.g., voice + visual lip movements, voice + environmental sensor data) to enhance signal robustness in complex environments. For example, using lip movement images to assist speech recognition to reduce noise interference;
- Developing unsupervised/self-supervised noise adaptation algorithms that can dynamically adapt without labeled noise data.
⛳️ Operation Results
🔗 References
[1] Zhuo Lu. Research and Implementation of Speech Enhancement Algorithms [D]. Hubei University of Technology [2025-08-30]. DOI: CNKI:CDMD:2.1011.091629.
[2] Zhang Jinhui. Research on Speech Enhancement Algorithms Based on Microphone Arrays [D]. Lanzhou Jiaotong University, 2014. DOI: CNKI:CDMD:2.1014.421814.
[3] Rong Qiang, Xiao Han. Research on MMSE Wiener Filtering Speech Enhancement Method and Matlab Implementation [J]. Computer Applications and Software, 2015, 32(1):4. DOI: 10.3969/j.issn.1000-386x.2015.01.039.
📣 Partial Code
🎈 Some theoretical references are from online literature; if there is any infringement, please contact the author for deletion.
👇 Follow me to receive a wealth of Matlab e-books and mathematical modeling materials
🏆 The team specializes in guiding customized Matlab simulations in various research fields, helping to realize research dreams:
🌈 Various intelligent optimization algorithm improvements and applications
Production scheduling, economic scheduling, assembly line scheduling, charging optimization, workshop scheduling, departure optimization, reservoir scheduling, three-dimensional packing, logistics site selection, cargo location optimization, bus scheduling optimization, charging pile layout optimization, workshop layout optimization, container ship loading optimization, pump combination optimization, medical resource allocation optimization, facility layout optimization, visual domain base station and drone site selection optimization, knapsack problem, wind farm layout, time slot allocation optimization, optimal distributed generation unit allocation, multi-stage pipeline maintenance, factory-center-demand point three-level site selection problem, emergency life material distribution center site selection, base station site selection, road lamp post arrangement, hub node deployment, transmission line typhoon monitoring devices, container scheduling, unit optimization, investment optimization portfolio, cloud server combination optimization, antenna linear array distribution optimization, CVRP problem, VRPPD problem, multi-center VRP problem, multi-layer network VRP problem, multi-center multi-vehicle VRP problem, dynamic VRP problem, two-layer vehicle routing planning (2E-VRP), electric vehicle routing planning (EVRP), hybrid vehicle routing planning, mixed flow workshop problem, order splitting scheduling problem, bus scheduling optimization problem, flight shuttle vehicle scheduling problem, site selection path planning problem, port scheduling, port bridge scheduling, parking space allocation, airport flight scheduling, leak source localization
🌈 Machine learning and deep learning time series, regression, classification, clustering, and dimensionality reduction
2.1 BP time series, regression prediction, and classification
2.2 ENS neural network time series, regression prediction, and classification
2.3 SVM/CNN-SVM/LSSVM/RVM support vector machine series time series, regression prediction, and classification
2.4 CNN|TCN|GCN convolutional neural network series time series, regression prediction, and classification
2.5 ELM/KELM/RELM/DELM extreme learning machine series time series, regression prediction, and classification
2.6 GRU/Bi-GRU/CNN-GRU/CNN-BiGRU gated neural network time series, regression prediction, and classification
2.7 Elman recurrent neural network time series, regression prediction, and classification
2.8 LSTM/BiLSTM/CNN-LSTM/CNN-BiLSTM long short-term memory neural network series time series, regression prediction, and classification
2.9 RBF radial basis neural network time series, regression prediction, and classification
2.10 DBN deep belief network time series, regression prediction, and classification
2.11 FNN fuzzy neural network time series, regression prediction
2.12 RF random forest time series, regression prediction, and classification
2.13 BLS broad learning system time series, regression prediction, and classification
2.14 PNN pulse neural network classification
2.15 Fuzzy wavelet neural network prediction and classification
2.16 Time series, regression prediction, and classification
2.17 Time series, regression prediction, and classification
2.18 XGBOOST ensemble learning time series, regression prediction, and classification
2.19 Transform various combinations time series, regression prediction, and classification
Directions cover wind power prediction, photovoltaic prediction, battery life prediction, radiation source identification, traffic flow prediction, load prediction, stock price prediction, PM2.5 concentration prediction, battery health status prediction, electricity consumption prediction, water body optical parameter inversion, NLOS signal identification, precise prediction of subway stops, transformer fault diagnosis
🌈 Image processing aspects
Image recognition, image segmentation, image detection, image hiding, image registration, image stitching, image fusion, image enhancement, image compression sensing
🌈 Path planning aspects
Traveling salesman problem (TSP), vehicle routing problem (VRP, MVRP, CVRP, VRPTW, etc.), drone three-dimensional path planning, drone collaboration, drone formation, robot path planning, grid map path planning, multimodal transport problems, electric vehicle routing planning (EVRP), two-layer vehicle routing planning (2E-VRP), hybrid vehicle routing planning, ship trajectory planning, full path planning, warehouse patrol
🌈 Drone application aspects
Drone path planning, drone control, drone formation, drone collaboration, drone task allocation, drone secure communication trajectory online optimization, vehicle collaborative drone path planning
🌈 Communication aspects
Sensor deployment optimization, communication protocol optimization, routing optimization, target localization optimization, Dv-Hop localization optimization, Leach protocol optimization, WSN coverage optimization, multicast optimization, RSSI localization optimization, underwater communication, communication upload and download allocation
🌈 Signal processing aspects
Signal recognition, signal encryption, signal denoising, signal enhancement, radar signal processing, signal watermark embedding and extraction, electromyography signals, electroencephalography signals, signal timing optimization, electrocardiogram signals, DOA estimation, encoding and decoding, variational mode decomposition, pipeline leakage, filters, digital signal processing + transmission + analysis + denoising, digital signal modulation, bit error rate, signal estimation, DTMF, signal detection
🌈 Power system aspects
Microgrid optimization, reactive power optimization, distribution network reconstruction, energy storage configuration, orderly charging, MPPT optimization, household electricity
🌈 Cellular automata aspects
Traffic flow, crowd evacuation, virus spread, crystal growth, metal corrosion
🌈 Radar aspects
Kalman filtering tracking, trajectory association, trajectory fusion, SOC estimation, array optimization, NLOS identification
🌈 Workshop scheduling
Zero-wait flow shop scheduling problem NWFSP , permutation flow shop scheduling problem PFSP , hybrid flow shop scheduling problem HFSP , zero idle flow shop scheduling problem NIFSP, distributed permutation flow shop scheduling problem DPFSP, blocking flow shop scheduling problem BFSP
👇