Test-time adaptation for speech enhancement with an autoregressive speech prior
Abstract
Test-time adaptation (TTA) offers a promising direction for improving speech enhancement models under mismatched acoustic conditions, without requiring access to labeled target data. In this work, we propose a single-utterance TTA method that regularizes a pretrained speech enhancement model using an autoregressive prior trained on clean speech latent representations extracted from a neural audio codec. Adaptation is performed by minimizing the Kullback-Leibler divergence between the enhanced speech distribution and the clean speech prior. Experiments across multiple noisy speech datasets show consistent improvements in speech quality, particularly under training-testing noise mismatch conditions. Code and audio examples are available online.
Keywords
Speech enhancement, test-time adaptation, neural audio codec, autoregressive model
Audio Examples
This example illustrates the behavior of the proposed method : TAAP (Test-time Adaptation based on an Autoregressive speech Prior for speech enhancement),
Progressive Test-Time Adaptation (30 steps)
Spectrogram evolution
DNS Challenge Audio Examples
The following examples illustrate the behavior of the proposed TAAP adaptation strategy on recordings from the DNS Challenge dataset.
For each example, we provide:
- Noisy input: original recording.
- Baseline enhancement (step 0): output of the pretrained enhancement model before adaptation.
- Oracle SIG step: adaptation step maximizing speech quality (SIG).
- Oracle BAK step: adaptation step maximizing background-noise quality (BAK).
- Predicted step: adaptation step estimated automatically using the logistic regression model described in the paper.
The examples are organized into three categories: 1. Cases where TAAP provides a clear improvement. 2. Cases where the baseline model is already strong and adaptation provides only marginal gains. 3. Failure cases where adaptation is not beneficial.