Hannah Choi's Final Project: Interactive Audiovisual System

Video Demo

Hear the Shape is an interactive audiovisual game that challenges the user to determine the shape of their environment by utilizing audiovisual cues available in the game.

Hear the Shape bloomed from my interest in auditory and visual neuroplasticity, specifically compensatory plasticity, in which it is known that individuals who are blind often have enhanced hearing and equip their occipital (visual) cortex in auditory tasks.
In addition, from a relatable anecdotal stance, I think we have all experienced the phenomenon of closing our eyes and being able to pinpoint & visualize exactly what we are hearing. Truly, we can see a great amount with our ears.

Hear the Shape puts the user into a visually limited environment. The screen starts off pitch black excluding a neon green globe. This neon green globe represents you, the user.



In addition to the green globe, you may hear a specific pitch being repeated - Eb. This starting pitch represents you and is played throughout the entire level.

Attached to the neon green globe is a trail. This trail will follow you around. As the user, you can use your arrow keys to navigate the environment.



As you navigate the environment, with your arrow keys, your green globe may come to a halt, in which it is met with bright, colored fireworks and an added pitch. This is an indication that you have hit a side of your enclosed environment.



It is important to know that each side of your enclosed environment corresponds with a unique color and added pitch. The number of colors and added pitches should equal each other. This number should also equal the number of sides the enclosed environment possesses.

Additionally, one may also decode the environment shape by firing out globes. Press the 'f' key to fire out a globe. This globe is initally the same color as you - neon green, but once the fired globe collides with a side, it will change color. The changed color will again correspond with the side. Essentially, the color to which the fired globe changes is the same color of the fireworks emitted when you collide with the side.



You have 15 chances to collide with the sides of your environment. Once 15 collisions are used up, you will be asked to choose which shape you think your enclosed environment was.



Then, you will receive feedback congratulating you or letting you know that your choice was incorrect. You will then be able to go onto the next level, or shape.




Once the final level is over (there are 4 levels), you will see an end screen in which you have the option to return to the start screen or quit.



Instructions are displayed in the game as follows:



Just to clarify game play, I will provide an example. Let's say you have a square. A square has 4 sides. This means that the starting pitch of Eb will be accompanied by 4 other pitches. Let's say the added pitches could be an Eb an octave above, a G, a Bb, and a Db. This would make the sound environment an Eb7 chord. These pitches will be continuously played, so you can determine the number of sides of the environment by carefully listening to the pitches and the quality of the chord. Also, you can collide with the sides of the square and count the number of different colored fireworks you encounter; you can also make note of the locations of the collisions. You can fire globes and count the number of color changes that appear; you can also make note of the locations of color change. Lastly, you can use your trail attached to your globe to trace the shape.

Maybe, I shouldn't have provided you with so thorough of an example, but these would be the ideal strategies to equip to perform well in this game.

Again, Hear the Shape is an audiovisual challenge which ultimately shows you that you can see a lot by hearing, even with sparse visual feedback.

Download Hear the Shape here:

Download, unzip file, click on the app, and play!

In response to Chapter 7 of Artful Design:

I quit social media and it has made me more social. This choice has made me happier and has allowed for me to focus on the present and on the individuals in my presence.

Don’t get me wrong. I know social media can be great if you harness it in the right way. It can allow for communication across the globe. It can allow for broadcasts of positive, enriching stories. It can be a simple way to stay in touch with your friends and those you meet throughout your life. Social media is all so instant, simple, and usable, but its pros could also be its cons as these same features could lead to a buildup of artificiality and desensitization in one’s social life.

On the other end, without social media, without Instagram, without Facebook, without Twitter, and without Snapchat, it could feel like being in a void. It could feel like you’re missing out on so much, but what is often unconsidered is that these social platforms are often a means to an end, meaning that there are other ways to stay interconnected with pals or get the information you might possibly need from these platforms.

Perhaps, I am interested in how my friend is doing on a completely different continent as she pursues her studies at Oxford. One route could be to take the social media route, receiving frequent updates on her outings and the events in her life and liking those posts, or it could be the long-awaited face-to-face route, where her arrival home for summer break results in a meet where we share stories of our years and we really cherish the time we spend together - with our phones off. If these meets aren’t feasible, perhaps even a video chat or phone call every once in a while could be pleasant.

I think social media traps us into believing that we must partake in it, that it is a required form of socializing, and if we don’t partake in it, that we are missing out, we are unsociable, and we are odd, but the truth is social media often prevents us from partaking in face-to-face human interconnection, arguably the most human and social form of interaction. As Principle 7.3 states, “Technology should strive to get out of the way of human interaction.” However, social media is doing the complete opposite of that – it’s getting in the way of human interaction, keeping us facing a screen instead of facing a human within an arm’s reach.

What if social media users engaged in communal activities such as picking up litter at a park, volunteering at a homeless shelter, communicating with elders at senior centers, fundraising for disease-related research, etc.? Or, simply, what if social media users spent more time engaging with their families instead of scrolling through their feeds? The list is endless of all the enrichening social activities one could partake in to make the world a better place, but the problem is that many of us don’t do them because we’re addicted to an online mode of interaction.

Obviously now is not the time to encourage in-person social interaction while we are in a pandemic, but post-pandemic, these social activities of societal benefit are possible to partake in and must be considered. What is unfortunate is that most will stay glued to the glitzy online world and addictive social media platforms as these platforms convince us that documenting our lives and receiving high approval ratings are of the true essence of life and that humility is of no essence. Instagram enticingly convinces girls, at younger ages every year, that focus on self-appearance is what matters in life and makes a woman powerful, and the wealthy endlessly flex as they convince us that money is the meaning of life – that it is success and happiness.

As Cornel West states, “Clever gimmicks of mass distraction yield a cheap soulcraft of addicted and self-mediated narcissists.” This is a telling quote from Dr. West’s speech on self-examination which discusses how there is a lack of societal self-examination due to “gimmicks of mass distraction” such as reputation and image. These gimmicks prevent many from focusing on introspection, who we are, our relationships with others, and what we want to do with our lives. These gimmicks take away from our identity and life. These “gimmicks of mass distraction,” such as a focus on one’s reputation and image, are deeply rooted in social media. Altogether, they are pulling us in the wrong direction toward consuming in place of giving and artificiality in place of authenticity.
https://scalar.usc.edu/works/index-2/a-voice-for-the-humanities-in-a-divided-america

Social media is pulling our society in the wrong direction. Especially vulnerable are the youth. If social media is increasing addiction and depression and is taking away from human interaction, it is essential that we attempt to turn the tide to pull society in the right direction. This is not to delegitimize all the good social media is possible of doing. This is to say that we need to make sure that users of social media use it the right way and that they are in control of the tool instead of the other way around where the tool is in control of them. This is also in reflection of those I have seen around me and of my experience. What if we used social media in balance with face-to-face human interaction? What if we engaged in as much active interaction as passive interaction?

Here’s what we can do:
1) Reduce social media usage to a certain time frame or take a social media cleanse – document how you feel throughout/about the experience and see what works for you. This might even involve shutting off your devices at times.
2) Set a weekly or monthly goal to do good for others in a human interactive social context whether it is as small as speaking with your next-door neighbor or traveling to a foreign place to teach children music.
3) Think of creative ways to communicate with your family, friends, and others that do not involve social media platforms, such as sending postcards.
4) When engaging with the social media world, ask yourself, “Is what I’m doing/about to do necessary/beneficial/right, or could/should I be doing something else?”

In this busy world, I feel that it is essential to be able to have self-control, to be able to know who we are, what matters to us, and what we aim to do with our time here. It is important for us to be able to formulate our own ideas and principles, devoid of pressure and harmful influence. On top of that, it is crucial for us to cultivate an unwavering moral compass. I also believe that is important for us to grasp the concept of human understanding and of human oneness – that we are all the same and we are all in this journey of life together – and also in the same vein, that just because we cannot see the other doesn’t mean that the person behind the screen isn’t human, doesn’t possess human feelings. Lastly, it is important to realize that we are not the center of the universe. The world does not revolve around us. Me, me, me, should be us, us, us. Human existence has always been a joint effort and the stronger that joint effort is to keep everyone healthy, happy, and peaceful, the better the outcome.

In response to Chapter 8 and the Coda of Artful Design:

The Realization
“Life isn’t about waiting for the storm to pass. It’s about learning how to dance in the rain.” – Vivian Greene

We design as a means to dance in the rain during the storm of life.
We design to live.

Our arrival in this world is never our choice, but because we exist, we are forced to deal with life – what it is and what to make of it.

It begins with childhood. In one’s childhood, one might be fooled. One might be fooled that all stories end happily, that your family and friends will always be there for you, that life is simple.

Then, time passes.

You realize that life is not simple. You realize that your family and friends can come into and out of existence. You realize that stories might not always end happily, and if they do, the path to reaching a happy ending might be crippling.

You might even see that the designer and the design of human beings & all sentient beings might be faulted. You might notice that one’s genetics, one’s biological makeup makes it difficult and nearly impossible to not be selfish and to not resist the natural survival mechanisms encoded within us.

So, what do you do?

As my uncle says, there's always the option of pulling the ejection seat. You could pull the ejection seat to make a grand exit, realizing that the world and your existence are immutably doomed, but many do not have the courage to do so.

So, we instead, turn to the sublime, in hopes that the sublime will make life worth living, and that in the aims of reaching for the sublime, we might chance upon pieces of it, even if they are miniscule.

If we didn’t absorb ourselves in the sublime aspects of life, we would not be here right now. If we did not learn how to dance in the rain during the storm of life, we would not be here right now.

We are all hopeless romantics – we are all attempting to soothe ourselves – we are all our own clinical psychologists in one sense, in search of humanity, morality, justice, and the sublime to distract ourselves and find purpose in this life & in this world since there is nowhere else to go. We design to calm ourselves. We design to convince ourselves that life is worth living and that it can be sublime.

Altogether, we are pathetically hopeful, attempting to undo the selfish survival instincts hardcoded within our genes and trying to convince ourselves that we can coexist in harmony, but at least in our journey towards good, we can be losers that are able to face ourselves in the mirror without feeling shame, losers that can actually sleep at night. We can be victorious in our own way. We can be victorious losers.