Lab 3

Matlab/Python: Spectrum Analysis

In the Matlab/Python portion of this lab, we’re going to perform some basic spectral analysis using the Discrete Fourier Transform. You may recall from the last lab that we said we could create any sound by adding together sinusoids, and sinusoids are the most basic building blocks of sound. Here we are going to create a function that takes any audio signal and tells us the sinusoids that create that sound.

The Fast Fourier Transform (FFT)

The Fast Fourier Transform (FFT) is an algorithm for computing the frequency content of a time series. For real signals (like audio signals), the magnitude of the first and second halves of the values returned by the FFT will be symmetrical, or even – you can think of the second half corresponding to negative frequencies. In other words, the absolute value of the output of an FFT is a list of the amplitudes of sinusoidal components evenly spaced from 0 Hz to half the sampling rate (fs/2 Hz), and then from -fs/2 to 0 Hz, respectively. The negative frequency component amplitudes are always going to be the same as the positive frequency components, as long as we are not feeding complex numbers into the FFT. So we really only care about the first half of the values returned by the FFT, since the second half is redundant for real signals.

You do not need to be familiar with the specifics of signal processing to do this assignment, but it is important to know that the FFT returns both positive and negative frequencies to accurately plot your results. The function you write in either Matlab or Python should only return the positive components and their corresponding frequencies.

Matlab

Create the following function:
[Y, F] = getSpectrum(input, fs)
Input variables:
input = any input signal
fs = sampling rate
Outputs:
Y = amplitudes of sinusoidal components
F = corresponding frequencies for amplitudes in Y

Notice this function has two outputs: Y and F. Matlab functions can output an array of variables, which is why the return values must be listed in square brackets. Y is a list of amplitudes of sinusoids spaced evenly from 0 Hz to fs/2 Hz non-inclusive, i.e. [0:fs/2). F contains the frequencies that correspond to each amplitude in Y.

Take a look at the documentation for the Matlab function fft(y,N). This is the fast Fourier transform, which takes a signal from the time domain into the frequency domain, outputting complex values for individual frequency components (sinusoids) of a given input signal. We are interested in the magnitude of these complex numbers, corresponding to the amplitudes of the sinusoids, which you can get with abs(). N is the number of sinusoids that fft() will return, spaced from (0:fs] - this corresponds to the size of the FFT window. The higher the value of N, the higher the frequency resolution of the spectrum. If N is larger than the number of samples in the input signal, the signal will be zero-padded, meaning zeros will be added to the end of the signal. A good number to use for N is the next power of two larger than the length of the input signal (check out nextpow2()).

Python

Create the following function:
def getSpectrum(input, fs)
Input variables:
input = any input signal
fs = sampling rate
Outputs:
Y = amplitudes of sinusoidal components
F = corresponding frequencies for amplitudes in Y

In Python, you can return multiple values from a function simply by writing return Y, F. In this function, Y and F will be the same as in the Matlab version.

Unlike Matlab, raw Python does not have a built-in function to compute the FFT. There are also multiple libraries that compute the FFT. For this lab, we will be using NumPy's np.fft.fft() function. The LibROSA library includes a re-implementation of this algorithm, along with a variety of additional utilities for the analysis of music and audio, and is extremely useful for more in-depth applications. However, for the purposes of this lab, NumPy's functions are sufficient.

As with the Matlab portion, the np.fft.fft() function will return a list of complex numbers, and we are interested in the magnitudes of those complex numbers. As in Matlab, the abs() function will return the magnitude. N is the number of sinusoids that fft() will return, spaced from (0:fs] - this corresponds to the size of the FFT window. The higher the value of N, the higher the frequency resolution of the spectrum. If N is larger than the number of samples in the input signal, the signal will be zero-padded, meaning zeros will be added to the end of the signal. A good number to use for N is the next power of two larger than the length of the input signal. There is no root nextpow2 function in Python, so computing this value may involve some logarithms.

Matlab/Python Deliverables:

Submit your getSpectrum() function along with a short (3-5 seconds) .wav sample of your choice (don’t use anything you’ve generated, it should be a music or audio sample), and a plot of the output of your getSpectrum() function with the .wav sample as the input signal. Make sure to label your plot! Here’s an example of how to do this:

plot(F,Y);
title(‘Spectrum of example.wav’);
xlabel(‘Frequency (Hz)’);
ylabel(‘Amplitude’);

Game Development: Mixed Reality

Develop a game in mixed reality. You are free to use any virtual reality (VR) equipment you have, but the main assignment is designed for augmented reality (AR) games built for smartphones. We’re intentionally leaving the rest of the assignment up to you. Now that you’ve had some experience with Unity, it’s time to explore what you can do when freed from the confines of a 2-D window into your game!

Some things to consider:

The player now has the freedom to look around. This adds all kinds of possibilities for gameplay elements that happen ‘offscreen,’ requiring the player to look around to discover things, interact with the game, etc. A mixed reality game that only happens in front of the player is missing the point of the medium.
Sight happens in front of us, but sound happens around us. Sound is a great way to tell the player they have to look somewhere. Use sound sources with spatialization to direct the players attention. Note: on sound sources there is a ‘Spatial Blend’ attribute that determines how much the sound source is 3D spatialized, and the default is 2D - you’ll want to move this setting all the way to 3D on each of your sound sources to take full advantage of Unity’s built-in sound spatialization. You’ll want to use headphones for the best experience.
If you are building for your own VR gear, keep in mind that continuous movement in VR can be uncomfortable and potentially nauseating. Many early VR games suffered from causing discomfort in the player by moving them around too fast. Many more recent VR titles solve this problem by employing a ‘teleport’ locomotion mechanic - the player selects where they want to move to and they ‘blink’ to that location. This may be difficult to implement on the Oculus since you won’t have a tracked controller, but it’s something to consider. If you choose to implement more traditional movement, make sure it’s at a speed that doesn’t make the player feel sick (playing seated often helps). It’s also possible to create a game that doesn’t rely on player locomotion (i.e. the player stays in one location and only looks around).
If you are building an AR game for smartphone, keep in mind that scaling in augmented reality can be tricky to manage. Even though Unity, ARKit, and ARCore all think in meters, motion forces won't scale with the size of your game, making it hard to generalize games to multiple spaces. Luckily, Unity has some facilities to help navigate this problem, which are detailed in this blog post.
Remember that directions need to be considered relative to where the player is looking! If you map forward movement to the global z axis, for example, it will be unnatural for a player looking to their left to move to the right when they try to move forward. You want to base player movement / actions off the direction the camera is facing.
Take advantage of your device's inputs. If you're using a VR system, make good use of the variety of inputs you have from your headset and remotes. If you're building an AR smartphone game, remember that your phone has a variety of data sources including the touchscreen, microphone, cameras, accelerometers, and more. Think creatively about game mechanics!
This lab is ripe for using trainable controls with Wekinator (and it is pretty easy to use if you haven’t before). You can train it with any of the inputs you have available, enabling you to code much more complex behaviors than would be possible in vanilla Unity. Wekinator’s a great way to use machine learning to get usable control from imprecise input. If you go this route, you’ll want to use OSC messages to send values to and from Wekinator. You can find out more about Wekinator on the resources page.

Getting Started with Mixed Reality (AR/VR) in Unity

For people building on VR gear: if you’re working with the Acer headset, this is Microsoft’s official page on setting up Unity to work with VR. This is another page that talks about working with MR on windows. If you are working with the Vive, I highly recommend this tutorial for getting started.

For people building AR smartphone games: iOS and Android each have their own framework for AR development — ARKit for iOS and ARCore for Android. Luckily, Unity has developed its own cross-platform asset set, called AR Foundation, to enable development in both environments. Unity has an overview of AR Foundation 2.1 (the version for Unity 2019.3 and higher) and a GitHub repo containing samples for demonstrating the functionality of AR Foundation. If you are using Unity 2020.1, you'll need to use AR Foundation 3.0, and if you're using Unity 2020.2 you'll need AR Foundation 4.0.

AR Development Quirks

Deploying AR games to smartphones is a little complicated when working within Apple's ecosystem. Although Unity's interface implies it can build for iOS devices, it actually can't. Instead, it builds the Unity project as an Xcode project, which must then be built from Xcode to work on an iPhone; Xcode is specific to Apple computers, and does not exist for other OSes. Even then, the game will only work on a trusted device physically connected to the Mac used to build the game from Xcode.

Android devices are somewhat easier to build for. Unity will build directly for Android phones, and those games can be installed directly from a download.

Although AR Foundation is designed to be cross-platform, there are some functions in both ARKit and ARCore that are not yet included in AR Foundation; you can see a list of features here. Perhaps the most significant of these for our purposes are ARKit's 3D object tracking and body tracking functions, which are not supported by ARCore and will therefore only work properly in iOS devices. If you use this functionality, it will be more complicated to build your game for multiple platforms. However, for this assignment, you should focus on building a game for your own device, and not worry about deploying it more broadly.

Game Design Deliverables:

Submit links to your pitch and playtest videos (details on the Lab Overview page if you’ve forgotten). You do not need to submit the actual Unity project. Be aware that screen capture may not be the best option for this Lab.

Both individual Matlab and Group Game Development portions of Lab 5 are due Friday, May 1st by 10:30AM. We will play your mixed reality games in class that day.