Robustness study of speaker recognition based on ECAPA-TDNN-CIFG

Abstract

This paper describes a study on speaker recognition using the ECAPA-TDNN architecture, which stands for Extended Context-Aware Parallel Aggregations Time-Delay Neural Network. It utilizes X-vectors, a method for extracting speaker features by converting speech into fixed-length vectors, and introduces a squeeze-and-excitation block to model dependencies between channels. In order to better explore temporal relationships in the context of speaker recognition and improve the algorithm’s generalization performance in complex acoustic scenarios, this study adds input gates and forget gates to the ECAPA-TDNN architecture, combining them with CIFG (Convolutional LSTM with Input and Forget Gates) modules. These are embedded into a residual structure of multi-layer aggregated features. A sub-center Arcface, an improved loss function based on Arcface, is used for selecting sub-centers for subclass discrimination, retaining advantageous sub-centers to enhance intra-class compactness and strengthen the robustness of the network. Experimental results demonstrate that the improved ECAPA-TDNN-CIFG in this study outperforms the baseline model, yielding more accurate and efficient recognition results.

Keywords

Feature extraction voiceprint recognition coupled input and forget gate time-delay neural network feature aggregation

1. Introduction

Voiceprint recognition, as a biometric technology, is widely applied in daily life [1]. Also known as speaker recognition technology, it identifies and verifies individual identities by analyzing the unique characteristics of their voice. TDNN is a commonly used neural network structure in voiceprint recognition systems. It models the temporal sequence data when recognizing speech by introducing the concept of time delay, extracting speech features [2]. TDNN can take the input of speech signals and use filters at different layers of the network to capture contextual information of the speech signal [3]. To achieve more accurate and robust voiceprint recognition, researchers have attempted to improve TDNN models by using different neural network structures, optimization algorithms, and training techniques [4]. This includes using deeper network structures [5], introducing attention mechanisms [6], and employing more complex loss functions [7]. With the maturation of the X-vector system, Brecht Desplanque and others proposed a significant structural improvement called ECAPA-TDNN, emphasizing channel attention, propagation, and aggregation within the time-delay network [8]. The core idea of this model is to introduce channel attention mechanisms into the baseline TDNN architecture. While TDNN treats feature information from different channels as equally weighted in temporal modeling, the ECAPA-TDNN architecture allows the network to automatically learn the importance differences between different channels [9].

The ECAPA-TDNN architecture has demonstrated excellent performance in speech recognition tasks, which is why it was chosen as the basis for improvement [10]. It has already proven to have potential in this task and provides a good starting point. Its model structure is relatively simple and clear, making it easy to understand and adjust. Improvements based on it allow for convenient experimentation with various modifications to find optimal model configurations. However, it has shortcomings in fully utilizing temporal contextual information, leading to weaker generalization in complex acoustic scenarios. Through enhancements to the ECAPA-TDNN model, we achieve higher performance in speaker recognition tasks, particularly in challenging acoustic environments such as noisy environments or those with reverberation. This paper proposes an improved ECAPA-TDNN-CIFG model, which effectively utilizes temporal contextual information above the ECAPA-TDNN model by adding Coupled Input and Forget Gate (CIFG) modules [11]. The model focuses on channel attention weight allocation, featuring characteristic information propagation and aggregation, enabling better exploration of temporal contextual information in speech signals. The study employs Sub-center Arcface for optimized training of the model [12], and experimental results indicate that compared to the baseline model, this network exhibits more robust speaker recognition performance in complex acoustic scenarios.

2. ECAPA-TDNN system

2.1 ECAPA-TDNN model

ECAPA-TDNN is a speaker recognition model based on TDNN, which has achieved significant success in the field of voiceprint recognition. It mainly consists of three parts. The SE-Res2Block combines Res2Net blocks and SE blocks, where Res2Net blocks enhance the network’s ability to learn multi-scale features, while SE blocks enhance the network’s focus on specific channels. Multi-layer feature aggregation and summation allow the model to utilize more contextual information by aggregating and summing features from different layers, thereby enhancing the robustness of the model and its ability to represent speaker characteristics. Attentional statistical pooling aggregates features statistically and weights features from different channels, further extracting crucial information about the speaker. The ECAPA-TDNN model makes significant contributions to speaker recognition tasks by improving feature representation, strengthening the extraction of speaker information, end-to-end training, and enhancing model robustness and generalization capabilities.

The structure of ECAPA-TDNN is illustrated in the following figure, where ‘c’ represents the channel dimension, and ‘t’ represents the time dimension. ‘k’ and ‘d’ denote the kernel size and dilation interval, respectively.

Figure 1.

ECAPA-TDNN network architecture diagram.

The SE-Res2Block module combines the characteristics of the SE (Squeeze-and-Excitation) module [13] and the Res2Block module [14]. Its role in the model is to enhance the representation of speech features and extract crucial information from the audio signal. The structure is illustrated in Fig. 2. By introducing the SE-Res2Block module after each TDNN layer, the network’s representational capacity is strengthened, leading to improved modeling of audio signals. The SE module utilizes a channel attention mechanism to dynamically learn the correlation between channels, allowing for the better extraction of important speech features. The Res2Block module, through residual connections and multi-layer convolution operations, increases the depth and non-linear capabilities of the network, aiding in the extraction of more rich feature information.

Figure 2.

SE-Res2Block module.

The introduction of the squeeze operation enables the SE block to generate descriptors for each channel. The squeeze operation involves only computing the average vector $z$ across the temporal frames, as shown in the equation:

$\displaystyle z=\frac{1}{T}\sum\limits_{t}^{T}{h_{t}}$ (1)

In Eq. (1), $h_{t}$ represents the activation values of the final frame layer at time step $t$ . In the activation operation, the weight for each channel is represented as $z$ , as shown in Eq. (2).

$\displaystyle s=\sigma(W_{2}f(W_{1}z+b_{1})+b_{2})$ (2)

In Eq. (2), $\sigma$ represents the sigmoid function, and $f$ denotes a non-linear function, $W_{1}\in R^{R\times C}$ and $W_{2}\in R^{C\times R}$ . This operation acts as a bottleneck layer. Whereas, $s_{c}\in(0,1]$ . These weights are applied to the original input through channel-wise multiplication, as shown in Eq. (3):

$\displaystyle\tilde{h}_{c}=s_{c}h_{c}$ (3)

In Eq. (3), the input is denoted as $h_{c}$ . The ECAPA-TDNN model achieves better performance in speech signal processing tasks by considering global attributes, extending contextual information, and paying special attention to different subsets of frames. At the same time, it maintains a relatively small number of model parameters.

The ECAPA-TDNN architecture has demonstrated excellent performance in speech recognition tasks, which is why it was chosen as the basis for improvement. It has already proven to have potential in this task and provides a good starting point. Its model structure is relatively simple and clear, making it easy to understand and adjust. Improvements based on it allow for convenient experimentation with various modifications to find optimal model configurations. However, it has shortcomings in fully utilizing temporal contextual information, leading to weaker generalization in complex acoustic scenarios. Through enhancements to the ECAPA-TDNN model, we achieve higher performance in speaker recognition tasks, particularly in challenging acoustic environments such as noisy environments or those with reverberation.

2.2 CIFG module

CIFG (Coupled Input and Forget Gate) is a gating mechanism in Recurrent Neural Networks (RNNs), representing a variant of the LSTM network. It introduces a single gating unit to replace the input gate and forget gate in LSTM. The structure is illustrated in Fig. 3.

Figure 3.

CIFG structure diagram.

The gating mechanism enables the network to better handle long sequential data while controlling information retention, selectively discarding unimportant information. Additionally, the coupled input and forget gate (CIFG) does not require learning parameters for the constant input forget gate, reducing computational costs, making it suitable for resource-constrained environments. CIFG excels in long sequence modeling and computational efficiency, emerging as an effective approach to enhance RNN structures. The mathematical expressions for CIFG are given by Eq. (2.2):

$\displaystyle i_{t}=\sigma(W_{xi}x_{t}+W_{hi}h_{t-1}+b_{i})$ $\displaystyle f_{t}=1-i_{t}$ $\displaystyle o_{t}=\sigma(W_{xo}x_{t}+W_{ho}h_{t-1}+b_{0})$ (4) $\displaystyle c_{t}=f_{t}\odot c_{t-1}+i_{t}\odot\tanh(W_{xc}x_{t}+W_{hc}h_{t-% 1}+b_{c})$ $\displaystyle h_{t}=o_{t}\odot\tanh(c_{t})$

Here, $h_{t}$ and $h_{t-1}$ represent the current and previous time step hidden states, while $i_{t}$ , $f_{t}$ , and $o_{t}$ represent input, forget, and output gates, respectively. $c_{t}$ is the cell state, $x_{t}$ is the input at the current time step. $W$ and $b$ are model parameters, and $\sigma$ represent the sigmoid.

2.3 Sub-center Arcface algorithm

ArcFace is an efficient face recognition algorithm, and its concepts and techniques can be adapted to speaker recognition systems for the extraction and matching of voiceprint features, aiming to achieve high-precision speaker recognition. The objectives of ArcFace are divided into two parts:

(1)
Intra-class compactness: Promoting relatively small distances between samples of the same class in the feature space;
(2)
Inter-class variance: Separating samples from all other negative centers.

However, in practical applications, the presence of noisy samples is a common challenge. These samples may lead to significant error losses, causing traditional ArcFace models to perform poorly when handling such data.

This paper introduces multiple sub-centers during the training process, distributing voiceprint feature vectors in the training set to different sub-centers to relax the intra-class compactness constraint. Each sub-center is designed to handle a specific type of noise or variation. This way, the network can learn voiceprint features under different noise conditions, enhancing the robustness of the recognition system. The sub-center ArcFace loss express as:

$\displaystyle\ell_{\textit{subcenter-AAM}}=-\log\frac{e^{s\cos(\theta_{i,y_{i}% }+m)}}{e^{s\cos(\theta_{i,y_{i}}+m)}+\sum\nolimits_{j=1,j\neq y_{i}}^{N}{e^{s% \cos\theta_{i,j}}}}$ (5)

The Sub-center ArcFace first utilizes the L2 norm of embedding features $x_{i}\in R^{d\times 1}$ and all sub-centers $W\in R^{d\times(N\times K)}$ through the matrix multiplication $W^{T}x_{i}$ to obtain the sub-class similarity score $S\in R^{N\times K}$ . After max pooling, the similarity score $S^{\prime}\in R^{N\times 1}$ for the class is obtained. The subsequent steps are the same as those in ArcFace.

The Sub-center ArcFace method allows each speaker’s voiceprint category to have multiple sub-centers, rather than just one center. These sub-centers can represent different variations or features of the same speaker. Initially, we relaxed the strict constraint on each sample being associated with only one positive class center, enabling the model to better adapt to noisy data. As training progresses, we can automatically clean up some less important sub-centers and noisy samples to achieve better performance.

Figure 4.
ECAPA-TDN-CIFG structure diagram.

2.4 The improved ECAPA-TDNN-CIFG model

ECAPA-TDNN employs a method that effectively combines features extracted at different network layers, enabling multi-level feature integration. This helps enhance the model’s performance to model speech signals and prevent the gradient vanishing problem that can occur with increasing neural network depth. When shallow features in the neural network contain a lot of redundant information, extracting crucial features from deep layers becomes more challenging due to potential similarities between shallow and deep features. This paper enhances the understanding and expression of complex data features in deep learning models by increasing the complexity of the residual structure.

Therefore, this paper introduces the model as depicted in Fig. 4. The model is built upon the ECAPA-TDNN framework, with CIFG embedded into the residual structure of multi-layer feature aggregation. This allows the model to selectively retain or discard features adaptively at each time step. By gating the features, CIFG helps reduce the impact of noise on the model and improves its robustness. CIFG also assists the model in better capturing key information from the input sequence, thereby enhancing performance.

Sub-center ArcFace is an improved version of the ArcFace loss function. Compared to traditional ArcFace, Sub-center ArcFace allows each class to have multiple sub-centers instead of just one center. This helps better handle label noise and speaker variations. As training progresses, we can automatically clean up some less important sub-centers and noisy samples, thus improving the model’s performance. Sub-center ArcFace performs well in real-world noisy environments, making the model more robust.

Introducing CIFG and Sub-center ArcFace into the ECAPA-TDNN architecture leads to better feature representation and stronger robustness. CIFG assists in selectively aggregating features, while Sub-center ArcFace handles speaker variations more effectively. Through automatic data purification and relaxation of intra-class constraints, the ECAPA-TDNN architecture performs better in complex acoustic environments.

Figure 5.

The comparison of accuracy (acc) between our proposed model and the baseline model.

3. Experimental setup and results analysis

3.1 Dataset

TIMIT is a commonly used speech dataset widely employed in research and development within the fields of speech recognition and speaker identification. Funded by the Defense Advanced Research Projects Agency (DARPA), TIMIT was released in 1991 through collaboration between the MIT and the UT Austin. It is one of the earliest large-scale datasets widely used in speech recognition research, encompassing speech samples of American English, covering various accents and dialects. The speech samples are provided in digital form, typically encoded in 16-bit PCM. Carefully selected and annotated words, phrases, and sentences ensure the quality and usability of the dataset. TIMIT includes speech samples from 630 speakers representing different regions, including both male and female speakers from diverse age groups, occupations, and social backgrounds. Each speech sample is accompanied by corresponding text transcriptions for labeling and identification purposes, providing accurate textual representations of the spoken content within the samples. To augment the quantity and diversity of training data, this paper incorporates reverberation (using the RIR dataset) and noise (using the MUSAN dataset) [15].

3.2 Configuration

In this experiment, we utilized 80-dimensional Fbank features, which were obtained through a 25-millisecond window and a 10-millisecond frame shift. We set the num_frames to 200, the initial learning is 0.001, and the maximum training epochs are set to 80. The CUDA version used was 11.7, Python version 3.8, Librosa version 0.10.0, and the GPU model was NVIDIA 3080. The parameters for the loss function of the speaker recognition model were set as m $=$ 0.2 and s $=$ 30.

For performance evaluation, we employed two widely used metrics, EER and minDCF. These metrics are used to assess the accuracy and efficiency of the speaker recognition system. In minDCF, we set Ptarget to 0.01, while both CFA and CMiss were set to 1.

Table 1
EER and MinDCF in 4 architectures

Recognition model	Loss function	EER	MinDCF
ECAPA-TDNN	Arcface	7.19%	0.47%
ECAPA-TDNN	Sub-center Arcface	5.66%	0.41%
ECAPA-TDNN-CIFG	Arcface	5.97%	0.44%
ECAPA-TDNN-CIFG	Sub-center Arcface	5.13%	0.44%

Figure 6.

The comparison of Equal Error Rate (EER) between our proposed model and the baseline model.

3.3 Experimental analysis

Building upon ECAPA-TDNN, the model was trained and evaluated on the test set after incorporating CIFG structure and Sub-center Arcface loss function. The final results, as shown in the figure, achieved a recognition accuracy of 91.97%. By comparing the two, we can conclude that the accuracy has increased by 2.45% as shown in Fig. 5.

To validate the effectiveness of the improvements introduced in this study, we conducted four ablation experiments, comparing the baseline network with enhanced networks incorporating different methods. The experimental results are presented in Table 1.

Analyze the data in Table 1, it can be observed that the ECAPA-TDNN-CIFG model proposed in this study exhibits the lowest EER on the TIMIT dataset, with a value of 5.13%. This represents a significant improvement compared to the baseline ECAPA-TDNN model, which has an EER of 7.19%. According to Table 1, under the same conditions, the use of the integrated Sub-center Arcface loss function leads to a reduction of 1.53% and 0.84% in EER compared to a single Arcface loss function.

The introduction of the CIFG network effectively exploits temporal context information, resulting in EER reductions of 1.22% and 0.53% under the same loss function conditions. Ultimately, The ECAPA-TDNN-CIFG given in this paper with the Sub-center Arcface loss function outperforms the baseline model, achieving a 2.06% reduction in EER. The comparison of Equal Error Rate (EER) between our proposed model and the baseline model is shown in Fig. 6.

On the other hand, when changing the loss function based on ECAPA-TDNN, the MinDCF value decreases by 0.06%. However, when combined with the CIFG network, the value decreases by 0.03%. The ECAPA-TDNN-CIFG network model proposed in this paper demonstrates superior recognition performance in speaker verification tasks.

4. Conclusions

Through experiments conducted on the TIMIT dataset, we compared the performance of the ECAPA-TDNN and ECAPA-TDNN-CIFG models, revealing differences between the two. In our experiments, we found that the ECAPA-TDNN-CIFG model outperformed the ECAPA-TDNN model in evaluation metrics such as equal error rate and minimum detection cost function. Specifically, we observed that the equal error rate of the ECAPA-TDNN-CIFG model decreased by 2.06%, and the minimum detection cost function decreased by 0.03%. The recognition accuracy reached 91.97%. Compared to the baseline model, this represents an improvement of 2.45%. These results indicate that the ECAPA-TDNN-CIFG model outperforms the ECAPA-TDNN model in terms of performance. Our model exhibits better feature extraction capabilities; the introduction of the CIFG network in the ECAPA-TDNN-CIFG model enables better utilization of temporal context information, facilitating the network in retaining information requiring long-term memory and controlling the flow of information through gating mechanisms to aid in forgetting unimportant information. This means that the model can more accurately capture speaker characteristics, thereby improving recognition performance. The improved Sub-center ArcFace method implements subcategories by selecting the closest sub-center and then retaining only the dominant sub-centers to achieve intra-class compactness. In this approach, the intra-class compactness constraint is initially relaxed, allowing the algorithm to maintain robustness even in the presence of a large amount of label noise. These improvements make the model more discriminative, enabling more accurate recognition of different speakers, thereby enhancing performance. Our research, validated through practical experiments, has demonstrated the advantages of integrating the ECAPA-TDNN and CIFG networks, providing an effective method for enhancing the performance of speaker recognition systems. Furthermore, the introduction of the improved auxiliary center Arcface further enhances the model’s ability to utilize temporal context information, offering a more reliable solution for addressing complex scenarios in real-world applications.

Although our experimental results demonstrate the excellent performance of the ECAPA-TDNN-CIFG model in speaker recognition tasks, there are still areas for improvement and future research directions worth exploring. Firstly, we recommend training with larger datasets such as VoxCeleb2 and AISHELL-2. This would provide a more extensive and representative set of speech samples for a more comprehensive evaluation of the model’s performance and generalization ability. Secondly, further exploration of other improvement measures, such as increasing the model’s depth or width, adjusting hyperparameters, or trying alternative loss functions and regularization methods, could be beneficial. These measures may further enhance the model’s performance and adaptability to different acoustic environments and speaker variations.

Footnotes

Funding

Joint Innovation Fund Project of Lanzhou Jiaotong University and Corresponding Supporting University (No. LH2023002); Lanzhou Jiaotong University Youth Fund Project (No. LH2019005).

References

Bai

Zhang

. Speaker recognition based on deep learning: An overview. Neural Networks. 2021; 140: 65-99.

Sharma

Govind

Mishra

Dubey

Deepak

Prasanna

SRM

. Milestones in speaker recognition. Artificial Intelligence Review. 2024; 57(3).

Liu

Zhao

. A speaker verification method based on TDNN-LSTMP. Circuits Systems and Signal Processing. 2019; 38(10): 4840-4854.

Monteiro

Alam

Falk

. Multi-level self-attentive TDNN: A general and efficient approach to summarize speech into discriminative utterance-level representations. Speech Communication. 2022; 140: 42-49.

Kabir

Mridha

Shin

Jahan

Ohi

. A survey of speaker recognition: Fundamental theories, recognition methods and opportunities. Ieee Access. 2021; 9: 79236-79263.

Bian

Chen

. Self-attention based speaker recognition using Cluster-Range Loss. Neurocomputing. 2019; 368: 59-68.

. Weighted cluster-range loss and criticality-enhancement loss for speaker recognition. Appl Sci-Basel. 2020; 10: 9004.

Chen

Bao

. Phoneme-unit-specific time-delay neural network for speaker verification. IEEE-ACM Trans Audio Speech Lang. 2021; 29: 1243-1255.

Liu

Chen

Qian

. Depth-first neural architecture with attentive feature fusion for efficient speaker verification. IEEE-ACM Trans Audio Speech Lang. 2023; 31: 1825-1838.

10.

Wang

Han

. An investigation of ECAPA-TDNN audio type recognition method based on mel acoustic spectrograms. Electronics. 2023; 12(21): 15.

11.

Khan

Jan

Ullah

Abdullah

. Graphical structures of cubic intuitionistic fuzzy information. J Math. 2021; 2021: 1-21.

12.

Wang

Zhang

Wang

. Voiceprint Recognition under Cross-Scenario Conditions Using Perceptual Wavelet Packet Entropy-Guided Efficient-Channel-Attention-Res2Net-Time-Delay-Neural-Network Model. Mathematics. 2023; 11(19): 20.

13.

Shen

Albanie

Sun

. Squeeze-and-Excitation Networks. Ieee Transactions on Pattern Analysis and Machine Intelligence. 2020; 42(8): 2011-2023.

14.

Liu

Wang

Zhang

Zhou

. DA-Res2UNet: Explainable blood vessel segmentation from fundus images. Alex Eng J. 2023; 68: 539-549.

15.

Birajdar

Patil

. Speech/music classification using visual and spectral chromagram features. Journal of Ambient Intelligence and Humanized Computing. 2020; 11(1): 329-347.

Robustness study of speaker recognition based on ECAPA-TDNN-CIFG

Abstract

Keywords

1. Introduction

2. ECAPA-TDNN system

2.1 ECAPA-TDNN model

3.1 Dataset

3.2 Configuration

Table 1 EER and MinDCF in 4 architectures

4. Conclusions

Footnotes

Funding

References

Table 1
EER and MinDCF in 4 architectures