2.1

ResNet

The Thin ResNet-34 proposed in Reference⁵ is the first baseline system. It reduces the number of channels in the convolution layer of residual blocks, thereby cutting down the computational cost. The convolution layers of the network process 2D features on the frame-level feature extraction stage. Attentive statistics pooling⁴ is used as the temporal pooling strategy to concatenate the first-order and second-order statistics of frame-level features of each frame in the time dimension, thereby generating utterance-level features. See Reference⁵ for more details about the topology.

2.2

ECAPA-TDNN

The second baseline system is ECAPA-TDNN. It is a strengthened version of vanilla X-vector system^2,3. It utilizes the hierarchical residual structure of the Res2Net¹⁸ to capture multi-scale feature. It also integrates the SE module into the residual blocks to rescale the frame-level features per channel. Channel attention method is used at the temporal pooling layer to generate different attention coefficients for different frames for each feature map. The method of multi-layer feature aggregation and summation is used to generate the input features of the pooling layer using a dense layer, which concatenates the output feature maps of both deeper and shallower SE-Res2Blocks.

3.1

1D dynamic kernel convolution

The dynamic kernel convolution (DKC) is a dynamic channel selection mechanism. It is a multi-branch convolutional module which can select the kernel size adaptively in order to capture features in short-term and long-term contexts. The complete structure of the dynamic kernel convolution consists of three parts: split, attention, and select. A two-branch case is depicted in Figure 1.

Figure 1.

Dynamic Kernel Convolution (DKC) module.

At the split stage, for the input feature X ∈ ℝ^C×T, we conduct transformations F₁ : X → U₁ ∈ ℝ^C×T and F₂ : X → U₂ ∈ ℝ^C×T as two 1D convolution operators with two different kernel sizes k₁ and k₂, respectively. In the proposed model, the same kernel size is exploited in two branches, one branch uses the standard convolution while the other uses the dilation convolution. Such a dissimilarity can reduce network parameters while achieving almost the same performance.

At the attention stage, we combine the multi-scale information from different convolution branches by an element-wise summation:

The mean μ ∈ ℝ^C and standard deviation σ ∈ ℝ^C of each channel of U are collected by a statistics pooling layer. Specifically, the c-th element of μ and σ is calculated as follow:

Taking the concatenation of μ and σ as input, we can obtain a compact feature z ∈ ℝ^d after a simple full connected (fc) layers as follow:

where δ is ReLU function, В is batch normalization, the full connected layer F_fc([μ; σ]) equals to W^T[μ; σ], W ∈ ℝ^2C×d donates the weight metric, a compression ratio r is introduced to generate the squeezed dimension: d = C / r. Then softmax-weight of the i-th convolution branch can be obtained after another full connected layer:

where τ is the softmax activation function, V_i ∈ ℝ^d×C is the weight metric, s_i ∈ ℝ^c×1 donates the channel-wise attention vector for the feature map U_i (i = 1,2).

At the select stage, the c-th channel of the final dynamic representation Y ∈ ℝ^C×T can be calculated by the weighted summation over each branch as the equation:

where s_i,c represents the c-th element of s_i, and U_i,c represents the c-th row of U_i.

3.2

Enhanced attention mechanisms

Considering that a wider temporal context contains more speaker feature information, SE module is used in two baseline systems to rescale the frame-level features using global properties of the utterance. Specifically, frame-level features are compressed through spatial dimension using a global average pooling, and channel-wise weights are produced using a multi-layer perceptron (MLP). Some enhanced attention mechanisms can replace SE module to further mine the context information of features.

Spatial pyramid attention (SPA) replaces the single-scale global average pooling layer in SE module with a group of adaptive average pooling (AAP) layers of different sizes. Such a spatial pyramid structure can capture more spatial information of the input feature map.

For a given feature X∈ℝ^C×T, let w∈ℝ^C×1 be the channel-wise attention weight vector and X′ ∈ ℝ^C×T donates the output feature map after channel rescaling. SPA module can be presented as following equations:

where F_aap(X, s_i) donates the AAP layer with the output size of s_i, different outputs are resized into three 1-dimension vectors and concatenated to generate a 1-dimension attention map A∈ℝ^C′×1, R(·) donates the resize function. δ(·), τ(·) and F_fc(·) represent ReLU, sigmoid activation function and full connected layer, respectively.

Efficient channel attention (ECA) utilizes a 1-dimension convolution instead of MLP to generate channel-wise attention weights. The convolution operator can appropriately capture local-channel interaction while involving fewer parameters, which guarantees both effectiveness and efficiency. ECA module can be presented as following equations:

where F_gap(·) donates the global average pooling layer and C_k(·) represents the 1D convolution with kernel size of k.

Convolutional block attention module (CBAM) is composed of channel sub-module and spatial sub-module. The channel sub-module generates channel-wise attention weights utilizing max-pooling outputs and average-pooling outputs along time axis, then the spatial sub-module generates time-wise attention weights utilizing the two pooling outputs along channel axis and forwards them to a convolutional layer. CBAM takes both channel-wise and time-wise information interaction into consideration. It can be presented as following equations:

where w_c ∈ ℝ^C×1 and w_s ∈ ℝ^1×T donates channel-wise and time-wise attention weight vector, respectively. P_max(·, c) and P_max(·, t) represent max-pooling layer along channel and time axis, respectively. P_avg(·, c) and P_avg(·, t) represent average-pooling layer similarly. F_mlp(·) is the MLP module that can be represented by F_fc(δ(F_fc(·))) specifically.

4.1

Datasets

VoxCeleb^19,20 is a commonly used dataset for speaker related tasks. VoxCeleb1¹⁹ contains more than 150000 utterances from 1251 speakers, VoxCeleb2²⁰ contains more than 1 million utterances from 5994 speakers. We train models with a subset of VoxCeleb2 that we make by selecting 10 utterances randomly for each speaker. We use MUSAN dataset²¹ and RIR dataset²² to realize online augmentation. The augmentation methods are the same as the settings in Reference²³. SpecAugment²⁴ is added to the log Fbanks of the training samples with a frequency masking dimension of 8 and a temporal masking dimension of 10. All models are evaluated on VoxCeleb1.

4.2

Training settings

A two-second segment is extracted randomly from each training sample. The input features are 80-dimentional log Fbanks extracted from a hamming window with 25ms length and 10ms shift.

The proposed dynamic kernel convolution is used to replace the conventional 1D convolutions in each 1D SE-Res2block of ECAPA-TDNN baseline. Two branches are set for each DKC module. The first branch uses convolution with kernel size of 3, and the second branch uses the dilation convolution with kernel size of 3 and dilation factor of 2. Channel reduction ratio r is set at 16. Scale of each Res2block is set at 8.

We replace the SE layer with SPA, ECA, and ABAM, respectively. In SPA, three AAP layers are set with sizes of 1, 2, and 4, and the bottleneck dimension is set at 128. In ECA, kernel size of the convolution layer is set at 5. In CBAM, the bottleneck dimension in channel sub-module is set at 128, and kernel size of the convolution layer is set at 7.

All models use attentive statistics pooling as the temporal pooling strategy to generate utterance-level features. All convolutions for frame-level feature extraction are set to 512 channels, the last full connected layer is set as 192 dimensions to fix the dimension of the output speaker embeddings for all models. We train all models using AAM-softmax loss²⁵ with a margin of 0.2 and a scale of 30.

4.3

Evaluation protocol

Performance of models is measured by equal error rate (EER) and minimum detection cost function (MinDCF) with P_target = 10⁻² and C_FA = C_FR = 1. Models are evaluated on VoxCeleb1-O, VoxCeleb1-E and VoxCeleb1-H. For the complete utterances, we split each utterance into 5 segments and then extract speaker embedding for each segment. For each trail, we formulate a score matrix with the size of 5×5 to calculate the cosine similarities of all pairs of segments. All scores are averaged to obtain the final similarity score. For each trail, we take segments with a duration of 4 s and 2 s to test the robustness of the models on short utterances. Cosine similarity is calculated directly as the final score.

4.4

Implementation details

We implement all models with PyTorch framework and conducted them on three NVIDIA Quadro RTX 8000 GPUs, each GPU has 48GB of memory. We train 80 epochs for all models. The initial learning rate is set at 0.001 and decreases 3% in every one epoch. The network parameters are updated by Adam optimizer²⁶. The mini-batch size for training is 400.

5.1

Results on VoxCeleb1

Table 1 gives an overview of the verification performance of all implemented models for full utterances, Tables 2 and 3 show the performance overview for 4 s and 2 s segments, respectively. DKC-TDNN with ECA and CBAM modules has reduced network parameters compared with baseline systems. Most systems we proposed apparently outperform the baseline systems in the case of full utterances. This reveals that it is beneficial to capture multi-scale features adaptively with dynamic convolution. The improved channel and spatial attention method can indeed bring performance improvement in a TDNN-based architecture. DKC-TDNN with SPA module, in particular, achieves EER of 6.40% and MinDCF of 0.375, gives a relative improvement of 17.84% and 1.99% in EER compared with ResNet and ECAPA-TDNN, respectively. DKC-TDNN with ECA achieves similar performance enhancement. DKC-TDNN with CBAM shows the minimal improvement. Almost all proposed systems have just slightly reduced or even increased MinDCF compared with ECAPA-TDNN baseline system. When the utterances are shortened to 4 s and 2 s, proposed systems have better performances either in EER or in MinDCF compared with two baseline systems. Finally, we assume branch structure reduces the inference speed of the network by comparing the computing speeds of all the proposed models to two baseline systems.

Table 1.

Performance of models on VoxCeleb1-O, VoxCeleb1-E and VoxCeleb1-H for full utterances.

Table 2.

Performance of models on VoxCeleb1-O, VoxCeleb1-E and VoxCeleb1-H for 4 s segments.

Table 3.

Performance of models on VoxCeleb1-O, VoxCeleb1-E and VoxCeleb1-H for 2 s segments.

5.2

Ablation studies

We conduct a series of ablation studies to reveal the impact of each component in our proposed architectures on the final performance. Table 4 gives the results on VoxCeleb1-O. Experiment A.0 is the ECAPA-TDNN baseline system. Experiment B.0 adds the DKC module to replace conventional convolution in each branch of res2blocks of the baseline system. Experiments A.1, A.2 and A.3 replace the SE module of the baseline system as SPA, ECA and CBAM, respectively. Experiments B.1, B.2 and B.3 replace the SE module of experiment B.0 similarly.

Table 4.

Ablation study of attention enhanced DKC on VoxCeleb1-O.

The results of experiments A.0 and B.0 demonstrate the effectiveness of the DKC module. Benefits of all three enhanced attention mechanisms can also be evaluated by comparing experiments A.1, A.2 and A.3 with A.0. In particular, CBAM gives the lowest EER of 6.38% in three of the attention enhanced methods. we suppose that the temporal dimension information aggerated by the spatial attention sub-model in CBAM is useful for speaker feature extraction. Experiments B.1 and B.2 show that combine DKC module with SPA or ECA can achieve better results. However, comparing experiment B.3 with A.3, combine DKC module with CBAM achieves a worse result. The conflict of the two mechanisms needs further study. Ablation study of attention enhanced DKC on VoxCeleb1-O.