6. Digital Signal Processing

Generally speaking, we can use digital audio signals as input and/or as output in ML applications.

Possible applications:

• Classification (digital audio signal -> classifier)
  • Wake-up word detection
  • Voice activity detection
  • Speaker verification
  • Music Information Retrieval
    • Instrument classification
    • Mood
    • Genre
• Generation (digital audio signal -> digital audio signal)
  • Denoising
  • Echo Cancellation

Common Knowledge

Sound waveform represents sound pressure wave.

waveform == time domain signal

Waveforms can be:

• Periodic
  • simple (e.g. sine wave)
  • complex (multiple sine waves)
• Aperiodic
  • Continuous (static noise)
  • Transient (pulses, bursts)

Waveforms carry information about (physical vs. perceptual property):

• Frequency $\leftrightarrow$ Pitch
• Intensity $\leftrightarrow$ Loudness
• Time variations $\leftrightarrow$ Timbre

Frequency and Pitch

🔗 Fundamentals of Sound

• Perceptual attributes of acoustic waves: Pitch
• Pitch of Pure Tones: Pitch (perception of frequency) vs. Frequency

Power

• Leistung = Arbeit / Zeit
• energy per unit of time emitted by sound source in all directions
• Unit watt (W)

Intensity

• Sound power per unit area
• Unit W/m²

Threshold of Hearings

Threashold of Hearing and of Pain

• $TOH = 10^{-12}\ \mathrm{W/m^2}$
• $TOP = 10\ \mathrm{W/m^2}$

This is quite a large range ... not very handy.

Intensity Level

• Log Scale measured in decibels (dB)
• Ratio between two intensity levels
• Commonly the reference intensity level is at TOH

$$L_I = 10 \cdot log(\frac{I}{I_{TOH}})$$

• $I = I_{TOH}$ $\to$ $L_I = 0\ \mathrm{dB}$
• $10 \cdot log(2) \approx 3\ \mathrm{dB}$, Intensity doubles about every 3 dB

🔗 https://courses.lumenlearning.com/atd-austincc-physics1/chapter/17-3-sound-intensity-and-sound-level/

Sound Pressure Level, SPL

Intensity tells you how sound energy propagates through space, but the only way you can measure intensity is by measuring the pressure (rms). This is why people almost exclusively use expression sound pressure level, despite they are actually interested in sound intensity level.

$$I \propto p^2\ \text{and}\ p_0 = 20\ \mu \mathrm{Pa}$$

$$L_{p} = 10 \cdot log(\frac{p^2}{p^2_{0}}) = 20 \cdot log(\frac{p}{p_0}) = 10 \cdot log(\frac{I}{I_0}) = L_I$$

$$L_{I_1} - L_{I_2} = 10 \cdot log(\frac{I_1}{I_{TOH}}) - 10 \cdot log(\frac{I_2}{I_{TOH}}) = 10 \cdot log(\frac{I_1}{I_2}) = 20 \cdot log(\frac{p_1}{p_2})$$

• If the pressure is doubled the sound pressure level is increased with 6 dB (20 log (2)).
• If the intensity is doubled the sound pressure level is increased with 3 dB.

Loudness

🔗 https://en.wikipedia.org/wiki/Loudness 🔗 https://www.animations.physics.unsw.edu.au/waves-sound/quantifying/#6.4

• If the perceived loudness is doubled the sound pressure level is increased by 6 dB to 10 dB.
• The standard definition of a decibel indicates that it’s roughly the smallest amount of volume change that a person can subjectively perceive. That means variations of up to 1 dB ought to be pretty much imperceptible, while those at or beyond 1 dB are noticeable. (https://www.youtube.com/watch?v=uCnsw9oRDsM&ab_channel=CassAllen)

More about SPL and 3 vs 6 dB $\to$ www.sengpielaudio.com. Back to the 90s but content is OK 😉.

More

• timbre, harmonics, envelope (adsr) link from above
• vibrato, tremolo guitar vid

Sampling Rate

• ADC $\to$ Time series of amplitude values
• Sampling rate: how many amplitude values per second
• Resolution: Number of bits per sample
• Nyquist fequency $f_N = \mathrm{sampling rate} / 2$
  • Aliasing
• Signal to Noise ratio SNR
  • Definiton: Signal to noise ratio is the ratio of the signal power to the noise power
  • $SNR = S/N = 10 \cdot log(\frac{P_{Signal}}{P_{Noise}})$ dB
  • Good news: If you have your signal and noise already in dB then you can just sipmply substract both numbers to get the SNR.
  • And yeah, there is also Signal to Noise ratio given a certain bit resolution (quantization noise), check this [table](https://skillbank.co.uk/SignalConversion/snr.htm

CD example

• Sampling rate: 44100 Hz
  • 1 sample $\to$ 0.0227 ms = 22.7 µs
  • fun fact: sound travels in air about 30 cm per ms (ganz grob: 1 sample / cm)
• Resolution: 16 bits per channel
• Data rate (stereo): 1411.2 kilobits per second (Kbps) ($2 \cdot 16 bit \cdot 44.1 k samples / second $)

Audio Feature Extraction

What categorizes sound or what features capture different aspects of sound?

Low level features

• amplitude envelope, energy, zero crossing rate, spectral centroid, ...
• band energy ratio, spectral flux, ...

Mid and High level features

• melody, rhythm, genre, mood
• beat, pitch, harmony, timbre

Temporal scope

• Instantaneous
  • Temporal resolution of humans is around 10-20 ms (later important), i.e. 1 sample @44.1 kHz is way too short as perceivable audio chunk
• Segment level (seconds)
• Global

Signal domain

• Time domain
• Frequency domain
• Time-frequency domain $\to$ spectrograms (more on this later)

Learning Strategies

Traditional ML

Extract all the different low level features (hand picked) and feat into ML algorithm. The feature extraction happens per hand (feature engineering).

Deep Learning Approach

Typically the whole spectrum is fed into a NN. And the feature extraction happens in the hidden layers. In other word: The machine shall learn the features for the given task by itself (automatic feature extraction)!

Audio Feature Extraction

Time Domain

Framing of audio data into overlapping chunks (frames) of about 10-20 ms.

• 128 samples @44.1 kHz $\to$ 2.9 ms
• 512 samples @44.1 kHz $\to$ 11.6 ms
• ❗ Length/Duration depends on sampling rate. If you change the sample rate for any reason, you have to adjust the buffer size as well.

E.g. Framing with 50% overlap

• frame 1: sample 1...128
• frame 2: sample 64...192
• frame 3: sample 128...256

Notations:

• samplerate: typically not 44.1 or even 48 kHz, 8 kHz are acceptable
• frametime: time duration of one frame, typically 10-20 ms (or even slightly higher)
• framesize: number of samples in one frame, depends on samplerate and frametime
• framestride: number of sample before new frame starts, should be smaller than framesize and framesize/2 for 50% overlap
• n_segments: how many segments of overlapping frame you want to analyze

Quiz:

1. For 1 second of audio data samples with 8 kHz. How many frames with 128 sample each and 50% overlap do you have?
2. You have 124 frames with a framesize of 128 samples and 50% overlap. How many sample (non-overlapping) do you have?

Click to expand

1. For 1 second of audio data samples with 8 kHz. How many frames with 128 sample each and 50% overlap do you have?

• total_number_of_samples = 8000
• framesize=128
• framestride=64
• number_of_frames = 1+(total_number_of_samples - framesize)/framestride = 124

2. You have 124 frames with a framesize of 128 samples and 50% overlap. How many sample (non-overlapping) do you have?

• total_number_of_samples = framesize + (number_of_frames - 1) * framestride
• total_number_of_samples = 128 + (124 - 1) * 64 = 8000

Features are computed on those frames. Most commonly they are all based on frequency domain.

Frequency Domain

Fourier Transformation on short time frames $\to$ STFT (Short Time Fourier Transform). To avoid artefacts (keyword: spectral leakage) a window function has to be applied on the time domain audio frames (probably also possible after FFT ...?).

A window function provides a weighted selection of a portion of a time waveform for FFT analysis. It is generated by multiplying the original time waveform by a user-defined window function of some width. Note to myself: This is not a convolution, at least not in the time domain.

🔗 Window function

🔗 Hann Window example

I was always happy with the Hann window which has a nice property. If you ever want to re-transform the windowed frequency domain signal back to a waveform you need a apply an inverse Fourier Transformation. But then you need to do some kind of de-windowing as well, because you get chunks of overlapping waveforms that you need to combine to a continuous waveform ... Think about it!

inverse-fast-fourier-with-overlap: For a 50% overlap, there is a nice property of a VonHann window in that the appropriate weighting is implicitly correct in the transition region and you can simply sum the two modified signals over the overlap region.

)

Important properties so far:

• Frame length (typically around 10-20 ms)
• Hop length or size, defines the overlap (around 50%)
• Window function

🚀 Draw on boards (waveform, window fcn, overlap, pipeline etc.) or google 'audio analysis pipeline stft'

Fourier Transformation

Fourier analysis is a family of mathematical techniques, all based on decomposing signals into sinusoids.

There are different types of signals: continuous or discrete, periodic or aperiodic. Combinations of these features create different categories. For now, we concentrate on signals that are discrete and periodic. The corresponding Transform is the so called Discrete Fourier Transform - DFT.

Notes:

• Signals extend (at least theoretically) to positive and negative infinity
  • Sine and Cosine waves are defined as such
• There is no version of a Fourier Transform that only uses a finite length of the signal! Wait ... what?
  • You cannot synthesize something with finite length from a group of infinitely long signals
• The way around this dilemma is to make the finite data look like an infinite length signal. This is done by imagining that the signal has an infinite number of samples on the left and right of the actual points. The imagined samples are a duplication of the actual n points. In this case, the signal looks discrete and periodic, with a period of n samples. (motivation for windowing)
• Each of the Fourier Transformations can be subdivided into a real and a complex version.
• A transformation between the two domains does not add or remove information! It contains exactly the same amount, just in a different form! It's like changing the perspective or changing glasses to look onto the problem. Some feature may just become more clear or obvious.
  • One might argue (I do 😉) that the machine should in principle be able to perform on both domain signals.

Discrete Fourier Transform, DFT

DFT changes an N point input signal into two N/2+1 point output signals.

Image from here.

• Input signal in the time domain (given length by given sample rate)
  • Index 0...N-1
• Output signal in the frequency domain
  • Containing the amplitudes of the component cosine and sine waves (with some scaling)
  • Sometimes also called real and imaginary part, but in real DFT those are just real numbers (no complex numbers)
  • Index 0...N/2
• N typically power of two (128, 256, 512 etc.)
  • Important later for Fast Fourier Transform (FFT)
  • And N is even, or should be (don't know why ...)!!!

The sine and cosine waves used in the DFT are commonly called the DFT basis functions. In other words, the output of the DFT is a set of numbers that represent amplitudes. The basis functions are a set of sine and cosine waves with unity amplitude. If you assign each amplitude (the frequency domain) to the proper sine or cosine wave (the basis functions), the result is a set of scaled sine and cosine waves that can be added to form the time domain signal. This set is also called rectangular notation.

🔗 Show basis functions page 151

Example: Imagine a time domain signal with 128 samples. How do you label the axis in the frequency domain? ... OK, as written above, the two output signal have both N/2+1 samples, i.e. 65. There a two common options:

• Option 1: Label axis from 0 to 64
• Option 2: Label as a fraction of the sampling frequency running form 0 (DC) to 0.5 (one-half the sampling rate)

Quiz: If there a N samples entering the DFT, and N+2 (two times N/2+1) samples exiting, where did the extra information come from?

Answer page 153

Calculating the Inverse DFT

Synthesis equation:

$$ x[i] = \sum_{k=0}^{N/2} \mathrm{scaleFactor} \cdot ReX[k] cos(2 \pi ki/N) + \sum_{k=0}^{N/2} \mathrm{scaleFactor} \cdot ImX[k] sin(2 \pi ki/N)$$

Calculating the DFT

We will skip this. We will find out that there exists a way faster method $\to$ Fast Fourier Transform, FFT

Polar Notation

Alternatively to the rectangular notation, the frequency domain can be expressed in polar form. In this notation, the two amplitudes arrays (scaling factors for the sine and cosine basis functions) are replaced with two other arrays, called the Magnitude and the Phase. The magnitude and phase are a pair-for-pair replacement for the real and imaginary parts.

$$mag[k] = (ReX[k]^2 + ImX[k]^2)^{1/2}$$

$$phase[k] = arctan(\frac{ImX[k]}{ReX[k]})$$

🔗 Show page 161

With rectangular notation, the DFT decomposes an N point signal into N/2+1 cosine waves and sine waves, each with a specified amplitude. In N/2+1 polar notation, the DFT decomposes an N point signal into cosine N/2+1 waves, each with a specified amplitude (called the magnitude) and phase shift.

Question: Why does polar notation use cosine waves instead of sine waves? Sine waves cannot represent the DC component of a signal, since a sine wave of zero frequency is composed of all zero.

Fast Fourier Transform, FFT

Real DFT using complex DFT

🔗 Based on chapter 12 dsp-book

Complex DFT: A more sophisticated version of the real DFT discussed above. FFT is an algorithm for calculating the complex DFT. We won't deal with the algorithm itself here, for us it it important to understand how to transfer real DFT data into and out of the complex DFT format.

Basic picture:

Suppose you have an N point signal and you want to calculate the real DFT by means of an FFT:

1. Move the N point signal into the real part of the complex DFT's time domain, and then set all of the samples in the imaginary part to zero.
2. Calculation of the complex DFT results in a real and an imaginary signal in the frequency domain, each composed of N points. Samples 0 through N/2 of these signals correspond to the real DFT's spectrum. Done!

Calculating a real Inverse DFT using a complex Inverse DFT is slightly harder: Remember (see image above): Points 0 through N/2 in the complex DFT are the same as in the real DFT, for both the real and the imaginary parts.

1. For the real part, point N/2+1 is the same as point N/2-1, point N/2+2 is the same as point N/2-2, etc. This continues to point N-1 point being the same as point 1.
2. The same basic pattern is used for the imaginary part, except the sign is changed. That is, point N/2+1 is the negative of point N/2-1 and so on.

FFT algorithm

As for the DFT we leave it out ...

Speed and Precision Comparisons

🔗 Chapter 12 dsp-book

If interested: I have a self written real DFT using decomposition for comparison or benchmarks.

Questions

Why is Fourier Transform important for audio processing? Why was FFT such breakthrough?

Mel Spectrograms

Recap Frequency $\leftrightarrow$ Pitch from above: Humans perceive frequency logarithmically!

Example: Play with tone generator 60-260 Hz and 1600-1800 Hz. It is the same 'distancy' in frequency (200 Hz), but the perceived 'distance' (pitch) is totally different. Hope you agree 😁

In addition, if one wants to anaylse human voice all interesting information lies in a frequency range of about 100-8000 Hz. (check Voice frequency ranges)

So we want a:

• Time-frequency representation like discussed above ✔️
• Perceptually-relevant amplitude representation $\to$ sort of $10 \cdot \mathrm{log(amplitude^2)/ref}$ ✔️ (see e.g. librosa.power_to_db
• Perceptually-relevant frequency representation $\to$ Mel spectrogram

🔗 Mel scale

Convert frequencies to Mel scale

1. Choose number of mel bands
2. Optional: define upper and lower frequency
3. Apply mel filter banks to frequency spectrum

🔗 Example of mel filter banks including reference where i found some code, puh 😄.

🔗 Example of Spectrogram comparison from microphone signal.

❓ Can you convert a (modified) mel spectrogram back to time domain data?

MFCC

At least you should have heard about it.

🔗 yeah, a video