Pitch tracking and Harmonization

	Mus 220C Project - Pitch Tracking and Harmonization By Kapil Krishnamurthy Project Goal - The main goal of this project is to basically extract pitch information from an incoming signal - guitar or vocals. Using this information, I want to create interesting textures and sounds that accompany the input signal. One of the main applications of this project would be to synthesize midi information that could be used to trigger off other midi based instruments. Milestones - Week 1 - Proposed project topic - Research in parametric coding techniques, more specifically Spatial Audio Object Coding (SAOC). Idea rejected after reading lots of papers and still not really having a clue where to begin implementing the different modules. This was partly because a lot of the research involved resynthesizing an audio image from two channels of compessed audio that the brain perceives as being similar to the combination of 5 raw channels of audio. These are complicated topics that have been researched for years and are still under study. Week 2 - Proposed project topic - Designing an iPhone equalizer. Idea rejected due to several issues. The first thing was that it takes a while to actually get acquainted with the audio engine on the iPhone and I didn't think it was worth the time. I'd much rather get started on working on a platform that I'm already familiar with. The second issue was that even the new iPhone SDK (version 3.0) still does not permit access to the music library on the iPhone. So there was no real future for this app anyway. Week 3 - Do a brief literature survey of some onset detection and pitch tracking methods and do some analysis in Matlab to estimate the these characteristics with a guitar input signal. I got started with working in Matlab and wrote some code for onset detection of the notes played on an acoustic guitar. This was done by looking at the flux of the spectrum and then basically setting a threshold for the flux value which gave the onset peaks. I wrote some code to do pitch detection by counting the zero crossings but these seemed to be a little more complicated than I thought it would be. First of all the shape of the time domain waveform is really characterized by the type of input signal and the filters being applied to it. In my code I initially filtered off some low and high components to the guitar input signal and this has some effect on the signal shape. I've decided to try other methods of pitch detection. One of them will be taking the auto-correlation of the signal and looking at the frequency of the peaks. I plan to have this done by next week. Week 4 - Have some form of pitch detection working. Also apply the same algorithms for onset and pitch detection to different types of input signals and look at what the results are and how these algorithms may need to be individually tweaked based on the type of input. For this week, I've got the pitch detection working with auto-correlation. I'm also planning try another method of pitch detection by looking at the difference of several periods of the chunk of a signal with a shifted version of the signal. The frequency of occurrance of nulls in the output will give me the frequency of the signal. I just got this working so I haven't yet had the time to try it on different instruments. I will do that by next week. Week 5 - Get a VST project up and running in Xcode. Also be able to compile some fft library which is need for the spectral domain analysis. Getting a project set up in Xcode is not as trivial as it sounds. XCode loves to spit out errors at you and often refuses to compile. Never swear at XCode cause it can give out 846 errors like how it did with me (I eventually got around to its good side :) ). Anyway, for this week I browsed through some of the XCode documentation and understood some of how to substitute for a make file in terms of the project settings. View step by step instructions on getting it setup here. This is only for Mac users though. Sorry Windows. Week 6 - Get started on the basic pitch tracking VST plugin and see how it comes along. Once that is up and running, start prototyping other ideas. Onset detection method is partially working but is not really accurate. The pitch tracking is also partially working but gives a lot of junk values. The Average Magnitude Difference function method for pitch tracking has been rejected as it does not work too well for complex signals. I'm going to stick to the Autocorrelation method and get that working as the main frequency detection method. Week 7 - I'm mainly going to be working this week on getting the mid term paper in. I did the paper in Latex which is pretty awesome : ) . No other real updates for this week Week 8 - Work on getting midi messages back to the DAW after detecting onsets and doing the pitch detection. This week is where I have really got stuck. I've spent more than 10 - 12 hours trying to getting the midi messages to talk back to the DAW. For some reason the DAW absolutely refuses to acknowledge the messages even though I get a status message through the VST plugin that the DAW has received it. I'm going to be working on this for a bit longer until I feel that it is more important to move on. Week 9 - I'm working on developing a more robust onset detection algorithm. I'm making some progress and plan to focus my presentation around that instead of the VST plugin. I have the new onset detection algorithm working pretty well. I'm calling this the Peak^2 detection algorithm and will be presenting this at my final presentation. Week 10 - Final Presentation - A copy of the final project report can be downloaded from here.

Project Goal -

The main goal of this project is to basically extract pitch information from an incoming signal - guitar or vocals. Using this information, I want to create interesting textures and sounds that accompany the input signal. One of the main applications of this project would be to synthesize midi information that could be used to trigger off other midi based instruments.

Milestones -

Week 1 - Proposed project topic - Research in parametric coding techniques, more specifically Spatial Audio Object Coding (SAOC).

Idea rejected after reading lots of papers and still not really having a clue where to begin implementing the different modules. This was partly because a lot of the research involved resynthesizing an audio image from two channels of compessed audio that the brain perceives as being similar to the combination of 5 raw channels of audio. These are complicated topics that have been researched for years and are still under study.

Week 2 - Proposed project topic - Designing an iPhone equalizer.

Idea rejected due to several issues. The first thing was that it takes a while to actually get acquainted with the audio engine on the iPhone and I didn't think it was worth the time. I'd much rather get started on working on a platform that I'm already familiar with. The second issue was that even the new iPhone SDK (version 3.0) still does not permit access to the music library on the iPhone. So there was no real future for this app anyway.

Week 3 - Do a brief literature survey of some onset detection and pitch tracking methods and do some analysis in Matlab to estimate the these characteristics with a guitar input signal.

I got started with working in Matlab and wrote some code for onset detection of the notes played on an acoustic guitar. This was done by looking at the flux of the spectrum and then basically setting a threshold for the flux value which gave the onset peaks. I wrote some code to do pitch detection by counting the zero crossings but these seemed to be a little more complicated than I thought it would be. First of all the shape of the time domain waveform is really characterized by the type of input signal and the filters being applied to it. In my code I initially filtered off some low and high components to the guitar input signal and this has some effect on the signal shape. I've decided to try other methods of pitch detection. One of them will be taking the auto-correlation of the signal and looking at the frequency of the peaks. I plan to have this done by next week.

Week 4 - Have some form of pitch detection working. Also apply the same algorithms for onset and pitch detection to different types of input signals and look at what the results are and how these algorithms may need to be individually tweaked based on the type of input.

For this week, I've got the pitch detection working with auto-correlation. I'm also planning try another method of pitch detection by looking at the difference of several periods of the chunk of a signal with a shifted version of the signal. The frequency of occurrance of nulls in the output will give me the frequency of the signal. I just got this working so I haven't yet had the time to try it on different instruments. I will do that by next week.

Week 5 - Get a VST project up and running in Xcode. Also be able to compile some fft library which is need for the spectral domain analysis.

Getting a project set up in Xcode is not as trivial as it sounds. XCode loves to spit out errors at you and often refuses to compile. Never swear at XCode cause it can give out 846 errors like how it did with me (I eventually got around to its good side :) ). Anyway, for this week I browsed through some of the XCode documentation and understood some of how to substitute for a make file in terms of the project settings. View step by step instructions on getting it setup here. This is only for Mac users though. Sorry Windows.

Week 6 - Get started on the basic pitch tracking VST plugin and see how it comes along. Once that is up and running, start prototyping other ideas.

Onset detection method is partially working but is not really accurate. The pitch tracking is also partially working but gives a lot of junk values. The Average Magnitude Difference function method for pitch tracking has been rejected as it does not work too well for complex signals. I'm going to stick to the Autocorrelation method and get that working as the main frequency detection method.

Week 7 - I'm mainly going to be working this week on getting the mid term paper in.

I did the paper in Latex which is pretty awesome : ) . No other real updates for this week

Week 8 - Work on getting midi messages back to the DAW after detecting onsets and doing the pitch detection.

This week is where I have really got stuck. I've spent more than 10 - 12 hours trying to getting the midi messages to talk back to the DAW. For some reason the DAW absolutely refuses to acknowledge the messages even though I get a status message through the VST plugin that the DAW has received it. I'm going to be working on this for a bit longer until I feel that it is more important to move on.

Week 9 - I'm working on developing a more robust onset detection algorithm. I'm making some progress and plan to focus my presentation around that instead of the VST plugin.

I have the new onset detection algorithm working pretty well. I'm calling this the Peak^2 detection algorithm and will be presenting this at my final presentation.

Week 10 - Final Presentation - A copy of the final project report can be downloaded from here.