Below is our master schedule, with pointers to all reading
assignments, lecture overheads,^{1} and homework/lab
assignments for the course.

- Schedule Summary by Week:
- Intro/Overview, Research History
- Fourier Theorem Review
- Spectrum Analysis Windows: Optimality, Choice, and Design
- FIR Filter Design by the Window Method
- Tonal Spectrum Analysis
- Noise Spectrum Analysis
- Time-Frequency Display
- FFT Convolution via Overlap-Add (OLA) and Filter-Bank-Summation (FBS)
- Sinusoidal Modeling, Phase Vocoder, Applications
- Class Project Presentations, TA Research, Applications, Review, Worked Problems, Etc.

- Week 1 - Course Intro
**Reading**- Music 421 Overview (this document)
- Verify that you have had adequate prior exposure to elementary spectrum analysis, and digital filter analysis.
- First 25 pages of Chapter 5 of
**Spectral Audio Signal Processing (SASP)**entitled ``Spectrum Analysis of Sinusoids.'' (up to but not including ``Optimal Peak-Finding in the Spectrum,'' which we will return to later)

**Demos (Supplementary): Historical Overview of Audio Spectral Modeling****2016 Lecture Videos (Total Viewing Time 1.5 Hours)**- Audio Applications of the Fast Fourier Transform [10:56]
- Notation: Time and Frequency Signal Processing [2:10]
- Audio Spectrum Analysis Intro [3:12]
- Spectrum Analysis Windows, Zero-Centered and Causal, Real and Even Signals [2:31]
- Sinusoidal Spectrum Analysis Preview [33:37]
- Summary: Time Windowing = Spectral Smoothing [5:04]
- Rectangular Window and its Fourier Transform, Geometric-Series Closed-Form Sum, Sinc Function, Aliased Sinc Function, Decibel Scale, Convolution Theorem, Triangular Window, Main Lobe, Side Lobes, Side-Lobe Roll-Off Rate [24:12]
- Frequency Resolution Introduction and Examples [3:49]

- HW#1

- Week 2 - Fourier Review,
Spectrum Analysis Windows
**Reading**- (Supplementary) The acoustic origins of harmonic analysis by Olivier Darrigol
- (Supplementary) A History of Spectral Audio Signal Processing
- First three sections of Chapter 2 of
**SASP**entitled ``Fourier Transforms and Theorems''

**Demos (Supplementary)**- Historical Overview of Audio Spectral Modeling
- DFT Visualizations (be sure to check out the spectrum analyzer on Page 39 -- built up starting on page 28)

**2020 Lecture Videos (Total Viewing Time 2.5 Hours)****2016 Lecture Videos (Total Viewing Time 1.5 Hours)**- Frequency Resolution, Rectangular Window Resolving Power, Generalized Hamming Window Family, Mu-Law, Telephone Bandwidth, Audibility of Noise, How Many Digital Bits are Enough?, Hearing Threshold, Auditory Masking, Noise Shaping [18:44]
- Fourier Review: DFT, IDFT, DTFT, IDTFT, Continuous versus Sampled Frequency [8:57]
- Fourier Review, Continued, Spectrum of a Sinusoid, Spectrum of a Windowed Sinusoid, Main Lobe, Side Lobes, Rectangular Window and Transform Review, Roll-Off Review, Resolution Example Review, Four Cases of Continuous/Sampled Time/Frequency: Fourier Transform, Fourier Series, DTFT, DFT [19:43]
- Rectangular Window Transform Review [15:41]
- Fourier Review Continued: Linearity, Fourier Symmetries, Shift Theorem, Linear Phase Term, Convolution Theorem, Graphical Convolution, FFT Convolution, Cyclic and Acyclic Convolution, Convolutional Spreading, Time Aliasing, Matlab Example of FFT Convolution, Zero Padding to Next Power of 2, Correlation [19:48]
- (Supplementary) JC on Phoné: Early History of Computer Music [57:20]
- (Supplementary) Rectangular Window Transform Live Coded (Flipped Classroom Lecture 2016-04-13) [1:24:18]

- HW#2

- Week 3 -
Spectrum Analysis Windows,
Optimal Window Design by Linear Programming
**Reading**- Chapter 3 of
**SASP**entitled ``Spectrum Analysis Windows'' - First section of Appendix G of
**SASP**entitled ``Examples in Matlab and Octave''

- Chapter 3 of
**2020 Lecture Video (Total Viewing Time 1.4 Hours)****2016 Lecture Videos on FFT Windows (Total Viewing Time 3 Hours)**- Spectral Interpolation, Bandlimited Interpolation, Sinc Interpolation, Asinc Interpolation [21:00]
- Fourier Review Continued: Ideal Spectral Interpolation, Zero Padding, Hamming Window, Matlab Examples, Zero-Centered Zero Padding, Matlab's fftshift [20:58]
- Windows, Rectangular Window Properties Review, Asinc Review, Generalized Hamming Window Family, Hann Window, Hamming Window, Dolph Chebyshev Window and Transform, Chebyshev Polynomials, Optimal Windows [28:48]
- Windows Continued, Review of Previous Windows and Results [2:15]
- MLT Sine Window [5:25]
- Blackman-Harris Window Family, Frequency-Domain Convolution Implementation, Blackman Window, Three-Term Blackman-Harris Window [9:00]
- Power-Of-Cosine Windows [2:33]
- Bartlett Window [2:28]
- Poisson or Exponential Window, System Identification [5:46]
- Hann-Poisson Window, Eliminating Sidelobes, Convex Optimization, Gradient Ascent/Descent [9:49]
- Digital Prolate Spheroidal Sequence (DPSS) or Slepian Window, Kaiser Window, Matlab Implementation of DPSS [6:14]
- Kaiser and DPSS Windows Compared, Kaiser Window Transform, Time-Bandwidth Product, Beta and Alpha Parameters for Kaiser Window, Matlab Examples [9:40]
- Dolph-Chebyshev Window, Relation to Hamming, Impulsive Endpoints, Ripple [9:06]
- Optimal Window Design by Linear Programming, Chebyshev Window Revisited [2:31]
- Linear Programming Formulation, Matlab's linprog, Using cvx in place of linprog [5:26]
- Chebyshev Window Formulated as a Linear Programming Problem [15:06]
- Remez Multiple Exchange Algorithm [2:32]
- Remez Exchange Algorithm Continued, Matlab's firpm, Convergence Guarantee, FIR Filter Design [14:30]
- Constrained Chebyshev Windows Using Linear Programming
[13:30]

- HW#3

- Week 4 -
FIR Digital Filter Design
**Reading**- Chapter 4 of
**SASP**entitled ``FIR Digital Filter Design''

- Chapter 4 of
**Demo:**PaintFIR (iOS)**2020 Lecture Videos (Total Viewing Time 2.5 Hours)****2016 Lecture Videos on FIR Filter Design (Total Viewing Time 2 Hours):****Supplementary:**What We Are Covering and Why [44:05]- FIR Digital Filter Design, Lowpass Filter, Optimal Least Squares, Window Method, Hilbert Transform [16:33]
- Lowpass Filter Design Review, Frequency-Domain View of Window Method [5:38]
- Window Method Filter Design Continued, FIR Paint, Frequency-Sampling Method [4:02]
- Review of Theory for Window Method for FIR Filter Design [2:51]
- FIR Filter Design Examples Using Different Windows, Optimal Chebyshev FIR LPF Design, Highpass and Bandpass FIR Design from LPF, Matlab's fir1 and fir2, Window Method Summary [28:25]
- Hilbert Transform FIR Filter Design Example [6:03]
- Hilbert Transform Example Intro [0:28]
- Hilbert Transform Theory Review [4:43]
- Single-Sideband Filter from Hilbert Transform, Sampling the Hilbert Transform [10:26]
- Hilbert Transform Theory Summary, Kaiser Window, Oversimplified Window-Method Design in Matlab [9:16]
- Oversimplified Hilbert Design by Window-Method Continued, Transition Band Mistuning [7:56]
- Improving the Window-Method Hilbert Design by Reformulating the Desired Frequency Response [15:32]
- Comparison of Window Method to Remez Exchange for Hilbert Design [1:35]
- Trick for Converting a Real Lowpass Filter to a Complex Single-Sideband Filter [2:20]
- Remez Exchange Usage Continued: firpm Parameters for Single-Sideband Filter Design, Results [6:18]
- Summary of window method versus optimal methods for digital filter design [5:10]
- More about optimal filter-design methods beyond remez exchange and linprog | cvx, firpm, cfirpm, references, More explication of the manual hilbert-transform design (two questions from Bjoern) [5:53]

- HW#4

- Week 5 -
FIR Digital Filter Design,
Sinusoidal Spectrum Analysis,

Supplementary: Gaussian Windows, Transforms, and Chirplets**Reading**- Finish Chapter 5 of
**Spectral Audio Signal Processing (SASP)**entitled

``Spectrum Analysis of Sinusoids.''

- Finish Chapter 5 of
**2020 Lecture Video (Viewing Time 1.25 Hours)****2016 Lecture Videos:**- Lecture 4B, Audio Spectrum Analysis begins, Introductory remarks: Overview, Outline [1:00]
- Resolving spectral peaks, Blackman-Harris window family and frequency resolution [1:42]
- Blackman-Harris window family and frequency resolution, continued [2:20]
- Examples of Sufficient Resolution: Rectangular, Hamming, and Blackman windows, Effective length of Hamming and Blackman windows [1:40]
- Oboe spectrum analysis example under Rectangular window [2:20]
- Oboe spectrum analysis example under Hamming window [0:55]
- Oboe spectrum analysis example under Blackman window [2:14]
- Conclusions regarding Oboe spectrum analysis under various windows, Preemphasis typically used for voice (first-order) [2:36]
- Spectral Interpolation: Ideal and Quadratic [1:44]
- Quadratic Interpolation Formulation [1:50]
- Quadratic Interpolation Solution, Why dB magnitudes interpolate better [1:20]
- Matlab for Quadratic Interpolation [0:39]
- Spectral Phase Interpolation, Phase Unwrapping [2:24]
- Quadratically Interpolated FFT (QIFFT), Zero Padding, Quadratic Interpolation of DB Magnitude, QIFFT Bias [2:24]
- Parabolic Interpolation is a Second-Order Taylor Series Approximation which is Exact for Gaussian Windows [0:52]
- Performance Evaluation of QIFFT Spectral Interpolation, Bias versus SNR [3:19]
- Example Minimum Zero-Padding Factors for Perceptually Accurate Peak Frequencies Rectangular, Hamming, Blackman Rule of Thumb for Sufficient Zero-Padding [2:05]
- Optimal Estimation of Peak Frequencies, Intro [0:34]
- Least-Squares Sinusoidal Parameter Estimation, Problem Statement [1:57]
- Least-Squares Sinusoidal Parameter Estimation, Formulation [1:50]
- Error Linear in the Parameters [0:56]
- Sinusoidal Amplitude Estimation by Least Squares [1:05]
- Sinusoidal Amplitude Estimation by Least Squares: One Step of Newton's Method [1:36]
- Sinusoidal Amplitude Estimation by Least Squares: Differentiating and Equating to Zero [1:00]
- Sinusoidal Amplitude Estimation by Least Squares: Extension to Complex Amplitude [0:48]
- Sinusoidal Complex-Amplitude Estimation by Least Squares Using the Orthogonality Principle, Projection Error Orthogonal to the Model [6:10]
- Sinusoidal Frequency Estimation by Least Squares: Maximize DTFT Peak Magnitude, Maximum Likelihood Estimates [1:15]
- Sinusoidal Frequency Estimation by Least Squares: Summary [0:32]
- QIFFT Summary [1:25]

- HW#5
- Supplementary: Appendix C of
**SASP**entitled ``Statistical Signal Processing''

- Week 6 -
Noise Spectrum Analysis,
Pages 31-47 of Intro Demos
**Reading**- Chapter 6 of
**SASP**entitled ``Spectrum Analysis of Noise''

- Chapter 6 of
**2020 Lecture Videos (Total Viewing Time 2.5 Hours)**`https://ccrma.stanford.edu/~jos/Music421A-2020/Music421A-2020-05-05.mp4``https://ccrma.stanford.edu/~jos/Music421A-2020/Music421A-2020-05-07.mp4`

**2016 Lecture Videos:**- What is Noise? What is Randomness? White Noise, Linear Congruential Pseudo-Random Number Generation, Shift Register, Sequences, Dither [15:38]
- Power Spectral Density = DTFT of Autocorrelation [2:38]
- Filtered White Noise [2:46]
- Noise Synthesis in Matlab [0:32]
- Noise Synthesis in Matlab, continued; Synthesizing Gaussian Noise; Central Limit Theorem [2:01]
- Faust White Noise Implementation, Linear Congruential Pseudo-Random Number Generator [11:16]
- Stationary Noise as Filtered White Noise [1:17]
- Correlation, Autocorrelation, Cyclic/Acyclic Correlation, Sample Statistics, Expected Value [10:46]
- Autocorrelation, Sample Autocorrelation, Stationary, Stochastic, Ergodic [2:55]
- White Noise Definition [1:49]
- Sample Variance, Nonzero Mean is a Deterministic Component [3:17]
- Example: Sample Autocorrelations and and Power Spectral Densities for White Noise [3:15]
- Sample Power Spectral Density [2:01]
- Autocorrelation and PSD are Real and Even [0:32]
- PSD Integral is the Total Variance [1:04]
- Noise Review Noise drawn from a normal distribution, What makes white noise uncorrelated and stationary? [5:47]
- Definition of mean estimator of white noise, Expected value of stationary and ergotic functions, Definition of variance of mean estimators [9:09]
- Definition of variance of mean estimators continued [6:26]
- Variance of Filtered White Noise, Correlation and Convolution Theorems, Correlation of Filtered White Noise with White Noise, Granular Synthesis [8:55]
- Problem with Cyclic Correlation for Colored Noise [2:02]
- Zero Pad to Avoid Cyclic Correlation, Increasing Variance at High Lags [2:48]
- Fast FFT Cross-Correlation, Dividing Out Bias, Smoothing Power Spectral Density [6:01]
- Noise Spectrum Analyis Review [2:42]
- Sample Variance and Sample Mean Review [0:31]
- Sample Power Spectral Density Review [1:59]
- PSD Area equals Variance Review [1:25]
- Biased Sample Autocorrelation Review [1:50]
- Practical Autocorrelation Computation, Use in Linear Predicion [5:25]
- Summary Notes - Restricting Maximum Lag [1:43]
- Periodogram Introduction, Periodogram Expected Value and Variance, Welch's Method [5:20]
- Welch's Method in the Frequency Domain [0:55]
- Filtered White Noise, 1/F Noise, Examples in Nature, Matlab and Periodograms for 1/F Noise [7:14]
- Gaussian Noise, Uncorrelated Noises Sum on Power Basis, Variance of Sum of Two, Normally Distributed Random Variables [6:11]
- Processing Gain [9:22]
- Stereo Panning [4:25]
- Stereo Panning Wrap-Up [1:58]

- HW#6

- Week 7 -
Time-Frequency Displays,
Supplementary: Pages 48-56 of Intro Demos
**Demo:**Drawing ``minimum'' in a spectrogram using various vocalizations [5:21]**Reading**- Chapter 7 of
**SASP**entitled ``Time-Frequency Displays'' - Chapter 8 of
**SASP**entitled ``Overlap-Add (OLA) STFT Processing'' - Supplementary:
``A Task-Optimized Neural Network Replicates Human Auditory
Behavior, Predicts Brain Responses, and Reveals a Cortical
Processing Hierarchy (May 2018)''

- Chapter 7 of
**2020 Lecture Videos (Total Viewing Time 2.5 Hours)**`https://ccrma.stanford.edu/~jos/Music421A-2020/Music421A-2020-05-12.mp4``https://ccrma.stanford.edu/~jos/Music421A-2020/Music421A-2020-05-14.mp4`

**2016 Lecture Videos:**- STFT Introduction [1:27]
- STFT Continued, Inverse STFT, Constant Overlap Add [3:05]
- COLA Examples, Sliding FFTs [4:17]
- Sampled STFT [2:40]
- Matlab for STFT [3:28]
- Classic Speech Spectrogram, Formant Resonances [5:10]
- Spectrogram of Vowel Sequence "Beet, Bit, Bet, Bot, Boat, Boot" [2:58]
- Spectrogram Parameters [1:35]
- Hop Size R [2:09]
- Speech Spectrogram Matlab and Parameter Choice, Frequency/Formant Resolution Trade-Off, Preview of Time-requency Reassignment [6:36]
- Reassigned Time-Frequency Spectrogram [3:59]
- Examples of Reassigned Spectrograms, Matlab's crossspecgram [2:54]
- Reassigned Spectrogram Examples: Two Beating Chirps and White Noise [3:07]
- Perceptual Significance of Time-Frequency Reassignment, Similarity to Phase Vocoder [2:26]
- Reassigned Spectrogram References [0:58]
- Reassigned Spectrogram Examples Revised to Include Blanking in the Middle
[3:50]
- Matching Audio Spectrograms to Perception, Shortcoming of Classical Spectrograms, Uniform/Non-Uniform Fiter Banks [2:01]
- Auditory Filter Banks [3:32]
- Auditory Toolbox [1:16]
- Loudness Spectrogram, Loudness Curves, Multiresolution STFT, Bark/ERB [4:01]
- Specific Loudness and its Conversion to Instantaneous, Short-Term, and Long-Term Loudness [:43]
- Multiresolution STFT Example, Pitch Pulse [3:44]
- Excitation Pattern Spectrogram [1:07]
- Auditory Filter Shapes [2:49]
- Specific Loudness Spectrogram [2:03]
- Classic and Specific Loudness Spectrograms Compared [3:13]
- Instantaneous, Short-Term, and Long-Term Loudness Compared [1:46]
- Sones and Phons [1:52]
- Equal Loudness Curves [1:05]

- HW#7

- Week 8 -
Overlap-Add FFT Processors,
Cross-Synthesis and Spectral Envelopes,
Filter-Bank Summation (FBS) and Overlap-Add (OLA) Duality
**Reading**- First 4 sections of Chapter 10 (``Applications of the STFT'') on STFT spectral modeling
- Start Chapter 9 of
**SASP**entitled ``Filter Bank View of the STFT''

**2016 Lecture Videos:**- Music 421A Course Outline, First Half [2:47]
- Graphical Depiction of FFT Convolution Buffers, Overlap-Add of Filtered FFT Frames [3:44]
- FFT Convolution Example: FIR Lowpass Filtering of Sum of Sinusoids [6:54]
- Example 2: Time-Domain Aliasing, Insufficient Zero-Padding [1:58]
- Example of Overlap-Add Decomposition [1:11]
- COLA Constraint Review [0:27]
- Modified-Hamming Overlap-Add Example [1:49]
- Hamming Window Overlap-Add [2:40]
- Oversampled Perfect Reconstruction Filter Banks [2:59]
- Overlap-Add FFT Convolution Summary [2:26]
- Avoiding Time Aliasing in Overlap-Add FFT Convolution, COLA Condition [0:35]
- Zero-Padding Summary [2:51]
- Alternative Coverage of OverLap-Save (OLS) FFT Convolution (Different Notation but Nicer Drawings) [3:39]
- Overlap Save FFT Convolution [5:52]
- OLS Q&A and Discussion of First Frame and its Aliasing for Zero-Phase FIR Filtering [2:13]
- OLS, Second Frame [4:13]
- OLS, Continued [5:20]
- OLS, Wrap Up [2:14]
- FFT Convolution with Time-Varying Modifications (New FIR Filter every Frame) [5:56]
- Diagram of FFT Convolution with Time-Varying Modifications [3:10]
- Weighted Overlap-Add (WOLA) [1:11]
- When to Use FFT Synthesis Windows (aka Post-Windows) [1:25]
- WOLA Applications [3:07]
- WOLA Conclusion, Poisson Summation Formula (PSF) Intro [0:40]
- Poisson Summation Formula (PSF) Derivation [6:41]
- Poisson Summation Formula as the Sampling Theorem [3:22]
- PSF Q&A and Discussion to 2:19 (can skip), COLA Corresponds to Zeros at Harmonics of Frame Rate [3:36]
- Overlap-Add Value Equals Window DC Gain Over R [0:44]
- Strong and Weak COLA [2:30]
- PSF Dual and Graphic Equalizers [2:25]
- Typical Windows are Trivially COLA in the Frequency Domain [1:27]
- Short Time Fourier Transform (STFT), Overlap-Add (OLA) and Filter-Bank Summation (FBS) Interpretations Intro [1:05]
- Mathematical View of STFT OLA and FBS Interpretations [1:47]
- Filter Bank View of STFT, Continued [1:36]
- STFT Filter Bank for the Rectangular Window Case (Running Sum Lowpass) [1:36]
- Running Sum Filter Bank, Continued [1:37]

- HW#8

- Week 9 -
Filter-Bank Summation (FBS),
Perfect Reconstruction Filter Banks (pp. 39-52),
Vocoders and Sinusoidal Modeling,
Historical Review (from Intro)
**Reading**- Finish Chapter 9 of
**SASP**entitled ``Filter Bank View of the STFT'' - Chapter 10 of
**SASP**entitled ``Applications of the STFT''

- Finish Chapter 9 of
**2020 Lecture Videos (Total Viewing Time 2.5 Hours)**`https://ccrma.stanford.edu/~jos/mp4/GMT20200602-215940_Time-Frequ_2048x1152_TRIMMED.mp4``GMT20200604-215636_Time-Frequ_2048x1152_TRIMMED.mp4`

**2016 Lecture Videos:**- STFT Using Rectangular Window with No Overlap Relies on Aliasing Cancellation [6:05]
- Filter Bank Summation (FBS) View of the STFT [4:23]
- Running-Sum Lowpass Filter [1:52]
- Frequency Response of the Running-Sum Lowpass Filter, Making a Bandpass from a Lowpass by Heterodyning [4:05]
- Question: You Can Change the Frequency Every Sample [2:08]
- Running Sum Filter Bank [2:33]
- One Channel of the Running Sum Filter Bank [4:41]
- Running Sum Filter Bank Computes a DFT Every N Samples [0:33]
- Computational Example of the Running-Sum Filter Bank [3:40]
- Modulus of Running Sum Filter Bank Output [0:58]
- Beating Between Positive and Negative Frequencies, Complex Chirp Response [2:26]
- Running Sum Filter Bank Output Modulus for Complex Chirp Input [2:56]
- Generalizing Running Sum Filter Bank to Any Window and Downsampling Factor (STFT) [1:07]
- One Channel of STFT Filter Bank for R=1 [2:23]
- STFT Filter Bank with Downsampling by R [2:17]
- STFT Filter Bank Output Modulus for Hamming Window and Real Chirp Input [1:49]
- STFT Filter Bank Output Modulus for Hamming Window and Complex Chirp Input [1:26]
- Overlap-Add Constraints in the Frequency Domain [6:09]
- Nyq(N) Windows Give Perfect Frequency-Domain OLA, Portnoff Windows, Comparison of Hamming and Portnoff [1:37]
- Portnoff Window Examples [2:53]
- Review of COLA/Nyquist OLA/FBS Duality [2:01]
- STFT Filter Bank Maximum Hop Size for Different Windows [5:16]
- STFT Filter Bank Interpretation Review [3:24]
- STFT Filter Bank, Hamming Window Case Reviewed [1:33]
- FBS Window Constraints for R=1 [3:45]
- Portnoff Windows [0:45]
- COLA/Nyquist OLA/FBS Duality [3:04]
- Hamming Window Transform and Frame Rate, FBS View, Filter Bank Reconstruction [6:50]
- Downsampled Filter-Bank Channel in STFT Filter Bank [2:38]
- Modified-Hamming Overlap-Add Example, Kaiser Window Overlap-Add [3:06]
- Q&A Regarding Kaiser Window OLA [1:30]
- FBS Fixed Modifications [6:38]
- FBS Time-Varying Modifications [1:28]
- STFT Summary and Conclusions [2:42]

- Week 10 - Orchi's research update,
Gaussian Chirplets,
Nonuniform FFT Filter Banks,
History of Spectral Modeling since the
Phase Vocoder (from p. 7)
Spectral Envelopes, STFT Spectral Modeling Time-Scale Modification (TSM) and Frequency Scaling

- Supplementary:
AI Signal Processing Primer

Download intro421.pdf

[Comment on this page via email]

Center for Computer Research in Music and Acoustics (CCRMA), Stanford University