This post continues directly from Speaker Recognition with Deep Learning (1) - Concepts and Task Taxonomy. From this point onward, I will use input/output/state-function notation from signals and systems, as well as Fourier-transform and convolution equations, without separate explanation.

The effort to extract speech features

Speaker Verification, Speaker Identification, and the other tasks comprising Speaker Recognition ultimately share one objective.

How can we define the “features” of an utterance well?

Every form of input speech passes through suitable preprocessing and feature extraction, and those extracted features support multiple downstream tasks. We can consider the following approaches.

The first step toward Speaker Recognition

As the previous post briefly explained, Speaker Recognition is extremely difficult. I only sketched the reasons then: microphone characteristics, background noise, overlapping speech, and so on. Observed speech $x$ does not contain only speaker-specific information; it is a mixture contaminated by many nuisances. Muddy water might be easy to filter, but this problem is not so simple.

Even the simplest observation model looks like this:

$$x(t) = \big(s_y(t) * h(t)\big) + n(t)$$

• $s_y(t):$ The original utterance of speaker $y$, containing that person’s timbre and vocal characteristics
• $h(t):$ The impulse response of the transmission path, such as microphone, channel, or room reverberation
• $n(t):$ Environmental noise, including background sounds, music, and overlapping speech

What we want is

$$z = f_\theta(x)$$

• $f_\theta:$ Feature extractor or embedding model
• $z:$ The desired speaker embedding, or latent

For $z$ to be a good latent, it must satisfy two conditions:

• Utterances from the same speaker should produce similar $z$ even across environments, channels, languages, and sessions.
• Different speakers should produce clearly different $z$ even when their content is similar.

Written as an objective,

$$\begin{aligned} \text{(same speaker)}\quad z(x_{y}^{(a)}) \approx z(x_{y}^{(b)}) \\ \text{(different speaker)}\quad z(x_{y}^{(a)}) \not\approx z(x_{y'}^{(b)}) \end{aligned}$$

Why speech recognition is difficult

The challenges listed above now become more clearly fatal. They can be classified in greater detail as follows.

Environment

Real speech almost never contains only the target speaker’s voice. Ambient noise, music, and reverberation are always mixed in, perturbing speaker-specific characteristics and degrading performance at every stage from training through testing. In the equation above, environmental noise, music, and reverberation enter through $n(t)$ and $h(t)$, directly damaging or obscuring speaker features. Reverberation also transforms the spectrum through convolution, changing the signal by frequency band:

$$S_x(\omega) \approx S_s(\omega)\cdot |H(\omega)|^2 + S_n(\omega)$$

The model often becomes unsure whether it should identify the person or the room.

Channel

Even for the same speaker, recording equipment and transmission path change the perceived timbre. Wired versus wireless links, microphone type, and transmission interference can separate embedding distributions across channels, and removing this difference entirely is difficult. For the same speaker $y$,

$$x_1(t) = s_y(t) * h_1(t),\qquad x_2(t) = s_y(t) * h_2(t)$$

the input distributions themselves differ, so the embedding distributions split easily. This is a classic cause of “the same person looking like different people.”

Speaker characteristics

One person’s voice continually changes with condition, health, age, emotion, accent, and other factors:

$$s_y(t) \rightarrow s_{y,\,\text{state}}(t)$$

One speaker is therefore not one point, but a distribution over states. The resulting intra-class variance makes discrimination harder. Moreover, unlike many systems, this problem has no explicit state function $h(t)$, making a solution almost impossible to identify.

Speech variability

Every utterance differs. Even the same sentence changes in speed, energy, and intonation. Real-world data is often closer to

$$x = g\big(y,\; c,\; r,\; n,\; \text{session}\big)$$

• $y:$ Speaker identity
• $c:$ Linguistic content, or what was said
• $r:$ Room or channel response
• $n:$ Noise
• $\text{session}:$ The recording conditions on a particular occasion

We want to retain only factor $y$ and erase the rest as much as possible, but changes in $c$ and $\text{session}$ are so large that the model is easily led by them.

Language characteristics

Changing languages changes phonology, rhythm, and intonation patterns, greatly increasing variation in $c$. Even for the same speaker,

$$p(x\mid y, \text{Korean}) \neq p(x\mid y, \text{English})$$

The model can become more sensitive to language or spoken content than to speaker identity. This is directly related to performance collapse in multilingual settings.

Transducing characteristics

Different microphones and headsets have different frequency responses, changing $H(\omega)$. Narrow bandwidth can discard information entirely:

$$X(\omega)=H(\omega)S(\omega),\quad H(\omega)\approx 0 \text{ for some } \omega$$

It is therefore mathematically natural for the same speaker to appear different after the device changes.

What do we actually want?

We want a latent $z$ sensitive to speaker identity $y$ but insensitive to nuisances such as environment, channel, language, and session. In information-theoretic form,

$$\max I(z; y)\quad \&\&\quad \min I(z; \text{nuisance}).$$

The key is for $z$ to retain as much speaker information as possible and as little information as possible about environment, equipment, language, and the speaker’s condition that day. This personal concern is why I previously wrote about disentanglement.

A more sober look shows, however, that expecting the model to accomplish this automatically without labels or an explicit separation signal is difficult. The observed data $x$ can be described by a generative process such as

$$x \sim p(x \mid y, u)$$

• $y:$ Speaker identity, the factor we want to preserve
• $u:$ Nuisance, including environment, channel, language, session, condition, and other factors we want to remove

We want an encoder, or representation learner, $f_\theta$ to create

$$z = f_\theta(x)$$

with intuitively large $I(z; y)$ and small $I(z; u)$. The problem is that $x$ alone does not uniquely identify a solution separating $y$ and $u$.

Separable representations are fundamentally non-identifiable

At the extreme, suppose

$$z = g(y, u).$$

Any mixing function $g$ is possible. If $g$ is even a bijection,

$$(y,u) \leftrightarrow z,$$

then a solution can minimize the loss while placing all information into $z$ in an entangled form. With only a reconstruction objective such as an autoencoder,

$$\min_{\theta,\phi}\ \mathbb{E}\left[\|x - d_\phi(f_\theta(x))\|^2\right],$$

one of the easiest ways to minimize it is to make $z$ copy $x$ as faithfully as possible. Then $z$ naturally includes both $y$ and $u$. Without special constraints, entanglement is a far more natural solution than disentanglement. A general learning method cannot easily separate only the speaker’s voice $y$ from nuisance $u$.

Generative models such as RBMs and DBMs do not disentangle automatically either

RBMs and DBMs are fundamentally energy-based models:

$$p_\theta(x,h) = \frac{1}{Z}\exp(-E_\theta(x,h))$$

• $h:$ Latent, or hidden unit
• $E_\theta:$ Energy function

Training generally maximizes likelihood:

$$\max_\theta\ \mathbb{E}_{x \sim \text{data}}[\log p_\theta(x)].$$

But likelihood maximization is not an objective that asks for clean factor separation.

The units of $h$ need not divide their roles neatly so that one captures speaker, another channel, and another language or content. From the learning perspective, one hidden factor may absorb a mixed speaker + channel + content pattern while other factors sit idle like dummy nodes. RBMs and DBMs approximate distributions well; they do not decompose factors into semantic units.

The same applies to DNNs: without supervision, they take convenient shortcuts

A DNN can learn $z = f_\theta(x)$, but even if the objective is speaker classification,

$$\min_\theta\ \mathcal{L}(\text{softmax}(W z), y),$$

it asks for features that distinguish speakers, not for nuisances to be discarded.

As soon as channel, language, or environment correlates with the label (even large datasets inevitably entangle a fixed set of people recorded in fixed environments),

$$I(u; y) > 0,$$

the model can actively use $u$ as a shortcut to predict $y$. The resulting representation naturally mixes

$$z \approx [\text{speaker info}] + [\text{nuisance cue}],$$

far from the desired $I(z; u) \approx 0$.

Separation rarely works without constraints or signals

This gives the following picture:

• likelihood-based training with RBM, DBM, or AE aims to reconstruct or imitate, so it has no reason to discard $u$;
• classification-based DNN training aims to predict correctly, so if $u$ helps, the model actively uses it.

Without explicit information such as conditions, constraints, augmentation design, adversarial signals, or regularization, the desired disentanglement does not arise as naturally as one might hope.

From this perspective, the GMM→$i-$vector→$x-$vector→ECAPA/SSL progression discussed next is also a history of how each era changed the constraint, structure, or learning signal used to make the model attend less to $u$.

Designing proper deep and deeper feature extraction

The meaning of “feature” in speaker recognition has changed over time. Early methods fed handcrafted speech features such as MFCC or PLP into a dedicated speaker-recognition model. Later approaches increasingly converged on extracting a fixed-length latent, or embedding. Deep learning nevertheless begins from the same place. The input may be a handcrafted feature such as MFCC/Fbank, or convolution may learn directly from the raw waveform. The important question is how to begin from these shallow features and extract the deep “speaker information” we truly want.

GMM-UBM: The era of probabilistic speaker models

The classic approach models a speaker’s utterance features with a GMM, a mixture of multivariate Gaussian distributions. GMM-UBM speaker verification is a representative family.

A GMM assumes that the sample distribution is a weighted sum of Gaussian distributions with several means and standard deviations.

(1) Input features

Frame-level features such as MFCC or PLP are generally used. Let each frame feature be $x_t \in \mathbb{R}^D$. The GMM is defined below, where $K$ is the number of Gaussian components. For speech features such as MFCC and PLP, a continuous speech signal is split along time into “frames” according to a window and stride. Frequency analysis within each sliding window describes the approximate frequency distribution at that location. In greater detail, the MFCC pipeline is STFT (Spectrogram) → Mel Filterbank (Mel-Spectrogram) → Cepstral Coefficients (MFCC). PLP differs somewhat, and explaining every step would take all day, so it is enough to think of them as extracting dominant frequency components. The figure below shows an MFCC example.

After extracting all features, we see a list of “indices” at each point in time. Each index represents a frequency band; the vertical list of values (drawn as tiles in the figure, though each is a number) can be treated as a feature embedding or vector. These are the multiple variables modeled by the GMM. GMM-UBM used 80 dimensions, so there are 80 indices.

$$p(x_t \mid \lambda)=\sum_{k=1}^{K} w_k \,\mathcal{N}(x_t;\mu_k,\Sigma_k)$$

Here $\lambda=\{w_k,\mu_k,\Sigma_k\}_{k=1}^{K}$. A GMM constructed this way models MFCC features from one speaker and therefore generalizes poorly across varied speakers. The UBM compensates for this weakness.

(2) UBM (Universal Background Model)

The UBM is a background distribution resembling the average over all speakers. First train UBM $\lambda_{\text{ubm}}$ on extensive utterance data from many speakers, then create each speaker model $\lambda_{\text{spk}}$ through MAP adaptation.

(3) Baum–Welch statistics and the $i-$vector

The UBM posterior produces a GMM for the original speaker information:

$$\begin{aligned} &\gamma_t(k) = p(k\mid x_t, \lambda_{\text{ubm}}) \\ &N_k=\sum_t \gamma_t(k), \quad F_k=\sum_t \gamma_t(k)\,x_t \end{aligned}$$

The simple principle is to remodel the speaker using a generalized speaker-distribution hypothesis $\lambda$.

These statistics feed directly into later $i-$vector and PLDA methods.

(4) Scoring for verification

The most basic method is a log-likelihood ratio:

$$\text{score}(X)=\log p(X\mid \lambda_{\text{spk}})-\log p(X\mid \lambda_{\text{ubm}})$$

Even a small change in channel, environment, or session destabilizes MAP adaptation of $\lambda_{\text{spk}}$. MFCC itself is also not particularly robust to noise.

$i-$vector: Compress everything into one fixed-length latent

If GMM-UBM creates a GMM distribution for every speaker, the improved $i-$vector approach

compresses one variable-length utterance into one fixed-length vector—the total-variability, or $i-$vector.

(1) Core idea: Reduce the supervector’s dimension

Let $M$ be the “supervector” formed by concatenating all GMM means:

$M = m + T\cdot w$

• $m:$ UBM mean supervector, or global mean
• $T:$ Total-variability matrix
• $w:$ The low-dimensional latent we want to extract—the $i-$vector

The prior is generally Gaussian: $w \sim \mathcal{N}(0,I)$. The key is that the structure models speaker and channel variation together in one space, hence “total variability.” It begins from the view that $w$ itself is likely to mix person and channel.

(2) Statistics-based estimation: UBM posterior → $w$ posterior

Statistics such as $N_k, F_k$ above give the posterior over $w$; its mean $\hat{w}$ is generally used as the $i-$vector.

One can think of it as finding one good prototype.

Back end: Cosine versus PLDA, and length normalization

After extracting $i-$vectors, we need to determine whether two belong to the same person.

The simplest method is cosine similarity:

$$\cos(\hat{w}_1,\hat{w}_2)=\frac{\hat{w}_1^\top \hat{w}_2}{\|\hat{w}_1\|\|\hat{w}_2\|}$$

A more sophisticated back end uses PLDA to separate speaker from session once more. PLDA is a probabilistic model assuming that embedding $z$ is generated as the sum of a speaker factor and a session factor, such as channel or environment. Its most common form is

$$z = \mu + F y + \epsilon,\quad y\sim\mathcal N(0,I),\ \epsilon\sim\mathcal N(0,\Sigma_w)$$

• $y:$ Speaker latent variable, shared by the same speaker
• $\epsilon:$ Within-speaker variation from session, channel, noise, and similar factors
• Training estimates $F,\Sigma_w$ from labeled embeddings $\{z_{s,i}\}$ and generally adds preprocessing so the data better fits the Gaussian assumption.
• Scoring compares two embeddings $z_1$, $z_2$ under $H1$: same speaker, sharing one $y$, versus $H0$: different speakers, with separate $y$ values, using a log-likelihood ratio. Despite the formal wording, it simply compares whether the two distributions fit better under a same-speaker or different-speaker assumption.

With DNNs, let the model extract features directly

The perspective now changes. Instead of indirect probabilistic compression, a DNN directly extracts the embedding. MFCC-dependent methods use shallow frequency features; a DNN asks a black-box neural network to extract and use more “abstract” features.

$d-$vector: Average the final hidden activations into an embedding

The $d-$vector is a typical early deep-learning embedding:

• process frames with a DNN,
• take the final hidden activations,
• and average them across the utterance to form an embedding.

$$h_t = f_\theta(x_t),\qquad z=\frac{1}{T}\sum_{t=1}^{T} h_t$$

Verification generally uses cosine similarity.

$j-$vector: Multitask learning over speaker and text content

Since attending only to the speaker is difficult in practice, the $j-$vector jointly learns speaker ID and text information as a multitask problem. Its perspective is candid: if content is difficult to ignore, place it inside the learning structure and control it explicitly. It does not seem like an especially fancy approach.

$x-$vector: TDNN + statistics pooling as the standard template

The $x-$vector remains a common baseline template. Its outline is simple:

• At frame level, a TDNN (Time-Delay Neural Network) observes a wide temporal context.
• Moving to utterance level, statistics pooling—mean, standard deviation, and so on—turns variable length into fixed length.
• The model trains on speaker classification and uses an intermediate-layer output as the embedding.

Statistics-pooling equation

For frame embeddings $h_t$,

$$\mu = \frac{1}{T}\sum_t h_t,\quad \sigma = \sqrt{\frac{1}{T}\sum_t (h_t-\mu)^2}, \quad s = [\mu;\sigma]$$

$s$ becomes the basis of the utterance-level representation. This is why descriptions of $x-$vector systems always emphasize the importance of pooling. After extracting the embedding, an $x-$vector system generally attaches cosine similarity or PLDA as a back end. More aggressively, it may use margin losses such as AM- or AAM-Softmax to make embeddings more separable.

DNN/$i-$vector hybrid: Replace only the alignments with a DNN

There was also an interesting intermediate stage: preserve the $i-$vector pipeline while replacing only the frame alignment—the posterior calculation—performed by the UBM-GMM with a DNN.

ECAPA-TDNN: A better $x-$vector

More recent architectures strengthen the $x-$vector family. The most famous branch is ECAPA-TDNN.

The original paper’s message is clear: retain the basic $x-$vector framework of TDNN + statistics pooling, but reinforce it through attention and aggregation.

• Retain the TDNN backbone.
• Improve attention over channels with SE-style components.
• Mix multi-layer features more effectively through aggregation.
• Make pooling more expressive with variants such as channel-dependent statistics pooling.

Raw waveform: Learn from the front end with models such as SincNet

Another line avoids handcrafted features such as MFCC and Fbank, entering directly from the waveform and learning the filter bank. SincNet is a representative example. It uses sinc-based convolution followed by a CNN pipeline for speaker classification and can extend to unsupervised embeddings from a mutual-information perspective. Its message is clean:

The narrowband features important for distinguishing people—pitch, formants, and so on—need not be extracted first by human design. Instead, impose interpretable constraints that let the network learn them directly. How to impose those constraints is the key.

SSL (wav2vec 2.0/HuBERT/WavLM): Grow the representation first

Finally comes the strongest recent trend across modalities: self-supervised learning. It is widely applied to vision, images, and video, although audio still seems not to have absorbed as much of the SSL wave.

• wav2vec 2.0: Latent-space masking + contrastive learning
• HuBERT: Masked prediction targeting clustering-based hidden units
• WavLM: Masked prediction combined with denoising, designed to work beyond ASR tasks

SSL is generally applied to speaker recognition in one of two ways.

Use it only as a feature extractor:

$$h_{1:T} = \text{SSL}(x),\quad z=\text{pool}(h_{1:T})$$

using mean or attentive-statistics pooling, for example.

Or fine-tune it with a speaker loss:

$$\min_\theta \ \mathcal{L}_{\text{spk}}(z, y)$$

Softmax is possible, but margin-based losses such as AAM- and AM-Softmax now seem more common. This is largely a difference in how the downstream task is performed. Speech still appears underexplored in SSL—why? Honestly, it feels as though we could build something better.

Conclusion

Speaker recognition is less a problem of making the model recognize a speaker than of making it choose what to discard and retain among everything it cannot avoid seeing. Observed speech $x$ never contains only $y$; it is a mixture including $u$—environment, channel, language, session, and condition. The central goal therefore remains identical from beginning to end: make embedding $z$ preserve as much person information as possible while containing as little nuisance information as possible.

From this perspective, the GMM→$i-$vector→$x-$vector→ECAPA/SSL progression records how each era designed structures, learning signals, and data to make the model attend less to $u$.

• GMM-UBM modeled distributions over shallow features such as MFCC and PLP to capture speaker differences, but was structurally vulnerable to channel and environment variation.
• The $i-$vector compressed an utterance into fixed-length $w$ to create a comparable latent. As the name total variability suggests, however, speaker and nuisance are easily mixed at the source within $w$. Back ends such as PLDA and length normalization therefore attempt a second separation.
• DNN embeddings—$d$-vector, $x$-vector, and ECAPA—let the network learn the features themselves, increasing expressiveness and pushing representations more strongly toward speaker identity through pooling, attention, and margin loss.
• SSL first builds a representation from large-scale data, then aligns it toward the speaker downstream. This too is a contest over which properties $z$ treats as invariant and which as discriminative.

The conclusion is simple:

A good speaker-recognition model is ultimately one that creates good embeddings, and good embeddings rarely emerge without good constraints—learning signals, structure, and data design.

Whether the architecture is a GMM, DNN, or SSL model, what matters more than parameter count is (1) which nuisances the design realistically assumes, (2) which constraints prevent the model from using those nuisances as shortcuts, and (3) how well those constraints hold in the real data distribution.

The next post will bring this progression one step closer to reality. Rather than list architectures that perform well in papers, it will examine the forms that $u$ takes in broadcast audio, and from a more practical perspective ask which training and inference designs actually increase $I(z;y)$ and reduce $I(z;u)$.