This project is part of a class assignment on interactive AI tools.
This etude allowed for explorations into multimodal interactions using Wekinator.
For the first exploration I wanted to see if I could include playfulness into medical guidance systems for the sake of improved surgeon user experience.
I used an electromagnetic tracking system intended for tracking of medical instruments to sonify the distance between two electromagnetic sensors.
Information about the sensor location is sent to Unity. Using UnityOSC the distance between the two sensors is sent
to Wekinator where it is mapped to differnet sounds depending on distance between sensors during the training step.
Wekinator sends an output OSC message to Chuck where the sound is generated based on the Wekinator output parameters. The ChucK file used for this exploration was the provided example file without any modifications.
The second exploration turns a smartphone into an instrument by sonifying different rotational values. This exploration uses the app Syntien OSC
on the smartphone to read accelerometer data and sends that information to Max/MSP to join multiple UDP messages to one and then sends this bundled OSC message to Wekinator.
In Wekinator the interactive AI gets trained on different sounds for extreme rotational positions. The mapped parameters are send to ChucK via OSC where the final sound synthesis happens. (ChucK file)
This small and very rudimentary application turns the smartphone into a playful instrument.
This last and by far most useless and fun experimentation using Wekinator was the imaginative tea bag soundscape. I love drinking chamomile tea, especially late in the evening.
I found myself wondering what it must be like to be tea bag and what it must feel like to take a dip into the hot water. This wekination uses webcam input to track a colorful object.
This color tracking by Rebecca Fiebrink provides x,y coordinates of an object as input into Wekinator.
I attached a pink post-it to the label of the tea bag and used that to track how far the bag has entered the water. Wekinator outputs 5 parameters to a ChucK file which uses sound samples to create the water soundscape.
All three wekinations are summarized in this video:
This experimentation and the use of Wekinator was extremely fun. At first I had to get used to the idea of making something useless for a day-to-day activity. At first I looked around my lab where I found a neat application for Wekinator that might actually enhance usability of medical guidance systems, but the result was not very playful, but rather functional. The second experiment of making a smartphone into an instrument was already more playful, but the outcome still seemed a little clunky. It was not until the last of the three wekinations that I finally found playfulness in the application of Wekinator. I think as with any new tool I first wanted to understand what it can do and more importantly, what it is good at doing. I very much appreciated how quickly one can build a plethora of multimodal interactions with just a few clicks in Wekinator. The creation of AI interactions using Wekinator was low effort, yet the outcomes were sophisticated and worked astonishingly well! The usage of Wekinator was my fisrt time exploring an interactiev AI system. If all interactive AIs are as easy to use as Wekinator, then I am all for a small-data mindset! I hope that I get to further explore the useage of different input and output modalities in Wekinator to prototype new and unconventianal interactions in a fast manner.