The relationship between classical and AI-based noise reduction

Introduction

In the 1980s, early digital broadband noise reduction systems used for music and speech employed dynamic filters that attempted to remove the continuous noise energy from the signal. The filters of choice at that time were developed from the Wiener Filter because these were mathematically elegant and, unlike simple low-pass, high-pass and notch filters, they were capable of removing significant amounts of noise with tolerably low degradation.

The frequency-domain Wiener filter adapts its response based on an estimate of the signal-to-noise ratio at each frequency. If the wanted signal is much louder than the estimated noise, almost all of the signal is retained. If the wanted signal and the noise are deemed to be equal, half the energy is retained. But if the energy is deemed to be mostly noise, most or all of it is removed. This provides a smooth response to the incoming signal.

For this technique to be useful, the audio had to be split into a high number of bands – typically 512 or 1,024. In early implementations, a spectral fingerprint of the noise was measured by the user during an otherwise silent passage to obtain the noise estimate in each band. If this measurement was accurate (if only at the instant that it was obtained) effective noise reduction could be undertaken. If it was flawed (perhaps by including some wanted signal) the performance of the system would suffer.

To overcome this limitation, the auto-dehisser was developed and introduced in 1994. This used statistical techniques to determine an estimate of the noise power spectrum at each moment without user intervention. This estimate was then provided to the filter to subtract the noise while minimising artefacts.

Denoisers based upon these techniques can be effective at removing continuous broadband noise, but are ineffective for anything else. Individual unwanted sounds such as a slammed door during a symphony are at best left untouched and at worst degraded into something that sounds ‘wrong’. In addition, rapidly changing broadband noise can outrun the system’s ability to update its noise estimate, generating twittering sounds caused by short bursts of noise being retained, or glugging underwater sounds caused by subtracting short durations of the wanted signal at given frequencies.

Structures that attempt to recognise wanted audio characteristics can be designed and coded into such systems. These are added by the system developer to improve the noise reduction and, until 2020 or thereabouts, most broadband noise filters for music and dialogue were based upon variations of this concept. But around this time, commercial noise reduction based upon neural networks began to appear. These were, in essence, pattern recognition systems that were trained to recognise sounds such as human speech, or 12-string guitars, or even things that you might assume are always noise but might be of interest in some contexts, such as helicopter rotor noises, or whatever else the developer wished the user to be able to retain from the input signal. Everything that was identified as fitting a pattern of the noise was discarded from the output.

To train a network, it’s typical to use thousands of clean examples of the target sounds, and a similar number of noise recordings. The developer mixes these randomly to create a huge library of noisy examples. The network then attempts to remove the noise, and compares its outcome with the original, clean recording. If the result is less than ideal, it adjusts its internal parameters and tries again. Over millions of attempts it learns to recognise the statistical patterns of the wanted signal and of the noise signals, and to answer the question, “what isn’t wanted signal?”. This means that, unlike a classical filter, it’s able to remove broadband noise and discrete sounds such as that slammed door.

Nonetheless, such systems are not foolproof. For example, a whisper or the chiff of a woodwind instrument might be misidentified as noise. In addition, the user may want to retain signal components – say, laughter or lip-smacks – in some contexts but treat them as noise in others. Errors can be addressed to a greater or lesser degree by increasing the complexity of the network and by additional training. As the library size increases, the performance of the noise reduction system (hopefully) does likewise, provided that the model has sufficient degrees of freedom to address the additional complexity.

Further AI models

Personalised voice models can be trained to enhance the noise reduction and speech enhancement for a particular speaker. The training can focus on attributes such as vocal tract characteristics, speaking rate and pitch range to help identify the speaker in noisy audio and when other people are speaking simultaneously.

Another set of processes are known as Generative models. When designed for cleaning speech, these can eliminate artefacts such as clipped syllables and replace lost words, and are therefore able to handle more severely degraded audio. However, the regeneration is based upon the system’s judgment of what speech is missing, which may not be accurate. This is of particular concern within the audio forensic and security sectors, in which the use of any form of generative AI is deprecated.

Transformers introduce the concept of attention mechanisms, allowing the model to determine which parts of a recording are most relevant, for example connecting disparate parts of a single sentence to help preserve consistency across long or interrupted utterances.

Elsewhere, AI-based source separation systems used for un-mixing audio can be trained to estimate the contribution of each source in a mixed signal. Such systems are computationally more demanding, but are of use in transcribing multi-speaker conversations as well as remixing, post production and hearing assistance. An improved approach uses multiple microphones to provide arrival time and intensity information to help determine where sounds originate.

Some AI-based systems are now being trained to perform multiple tasks simultaneously. These can include noise and reverberation reduction, noise classification, and voice activity detection. Such systems may perform their primary denoising tasks better because their models develop richer internal representations.

Latency

Off-line noise reduction systems can use a whole audio file (or even groups of files) to improve their noise estimates and improve performance, but many of the applications in the audio industry require real-time streaming solutions. The compromise then becomes one of latency versus performance.

Denoising systems based upon classical DSP can have latencies of less than one millisecond. This is important when the audio is being used in live situations – for example, when someone is being interviewed on air for TV and any significant latency will result in a loss of lip-sync. Similarly, if noise reduction is being used within the monitoring chains for singers and musicians, the latency must be near-zero or it may affect performance. This means that AI-based noise reduction systems with a latency of even a few tens of milliseconds are often unsuitable for these applications.

Where the field is heading

One target of AI-based audio processing systems is a comprehensive un-mixing model capable of separating sources, suppressing noise, echoes and reverberation, and producing natural-sounding audio recorded in or being streamed from challenging conditions. Where problems have well-defined mathematical solutions, using classical DSP can be more efficient and can sometimes offer much lower latency. In contrast, AI can tackle harder problems successfully. Much of the current research is therefore aimed at combining several ideas rather than relying on a single technique. A future system might combine:

• Training to learn general audio features
• Transformer architectures for extended temporal context
• Generative refinements for higher perceived quality
• Classical DSP to improve efficiency where appropriate and to apply equalisation and gain

Summary

Noise reduction systems using classical DSP ask the question, "Given the estimated signal and noise power at a given frequency, how much should I keep?". They can be fast and computationally inexpensive with low latencies, making them appropriate for use in situations such as live conversations, conferencing and streaming where any significant delay in the output would render the process unusable. It’s not unusual to find classical techniques being deprecated in current discussions, but modern implementations are non-trivial, and their benefits extend beyond low latency. In particular, their errors tend to be benign, residual noise can sound natural, and they are predictable in operation, so users can be confident that adjustments to parameters will result in expected changes in the output.

AI models act more like source-separation systems, learning what the desired signal sounds like and using context to recognise wanted sound structures. They ask, "Given everything I've learned about the signals on which I was trained, what is the most likely clean signal that produced this noisy recording?". This allows them to estimate the probability that each part of the signal belongs (or doesn’t belong) to a desired source, separating out unwanted sounds that occupy much of the same time/frequency space and reconstructing the audio accordingly. They can handle more complex problems, but at the expense of computational power and greater latency. Furthermore, if the wanted sound changes dramatically – for example, if an expert on Austrian culture demonstrates yodelling during a lecture – the yodelling may be removed, resulting in unexpected silence in the output.

Each approach has its place.