Designing an Emotionally-Aware IDE

I am working on adding logging of mundane actions and emotional states to miniAudicle.

In its most basic form, this is done via the addition of the "WTF" button.

(This project was a side-project that I worked on in the beginning of Music 220c.)

(Week 3) Proof-of-Concept Logging and IDE Buttons

I am able to create logs from within ChucK, and I am able to modify the GUI of miniAudicle and execute code when buttons I added are pressed!

The next steps will be to make sure that old logs don't get overwritten when new programs are run; to systematically add logging calls everywhere I'd like to; and to make the emotion reporting buttons more prominent and more satisfying to press. Especially during the second of these next steps, this blog will be a little bare as this is sort of a mundane activity, and I've already done the brainstorming for where logs should go. (See entry below.)

(Week 2) Brainstorm of What Data I'd like to Collect

• AST (abstract syntax tree) nodes: to tell what program features people are using and (perhaps) what emotions are linked with those features. Will help guide the development of learning materials and the (re-)design of language features.
• Exceptions: these are indicative of common user errors (e.g. not understanding how a particular syntax works) which can lead to improvements to the language. Plus, getting an exception is a highly emotional event.
• Emotion button presses: so that users can self-report emotions. (Future versions may use computer vision or skin conductance to measure emotion more empirically.)
• Other IDE button presses: for example, switching off the VM to stop all sound may indicate frustration as well (especially if this button is pushed five times in a row).
• Print statements: these represent program state that the programmer wanted to know. Maybe it will be useful to me as well.
• Program comments / other snapshots of the code itself: perhaps useful for sentiment analysis (e.g. "// stupid oscillator sounds glitchy!" or "// glitchy oscillator, so cool!". Code can also be analyzed manually to determine common mistakes that lead to frustration. (Automatic analysis of programs for similarity is a very difficult problem that is unlikely to be solved by me working alone.)
• "Live" snapshot of instructions running / audio being generated: this might be useful for determining why a user pushed a particular button. However, this surfaces a lot of problems:
  • How far back should we look for the particular event that caused a person to be frustrated (or some other emotion)?
  • How do we store the massive amount of data created by logging at the level of individual machine instructions?
  • How do we log this efficiently enough so that even simple programs don't become glitchy and unresponsive? (For example, logging just one time every time the users waits for time to pass still needs to run at much faster than 20 microseconds because users can wait for time to pass at the granularity of every sample.)
  Because of these drawbacks, I'm tempted to leave this aspect of logging off the table for now and come back to it.

(Week 2) Selecting and Building a Logging Library

I need to log user actions to capture how they interact with the IDE. If possible, I'm also considering logging "state of the world" like the audio that was produced recently or the instructions that were run recently, which might help give some context as to why the user reported certain emotions. However, such logs would need to be especially performant, so as not to interrupt the main audio thread and cause glitches in the sound.

Therefore, I decided to select a logging library, hoping to find one that is tuned to be much faster than a naive implementation I write myself would be. I looked at a few options:

• log4cpp: this library is popular, but I didn't find any performance data about it.
• g3log: this library was the clear frontrunner since it had a low latency (a few microseconds) and very good worst case latency. It can also log information when programs crash, which would be great because ChucK (and therefore miniAudicle) crash a lot! Unfortunately, it featured a complicated build process that didn't interact well with miniAudicle's build process.
• spdlog: this library has the fastest average latency of any logger that I found. Unfortunately, it uses features that are too new to be compiled on my operating system (which I'm currently stuck using).
• plog: the main feature of this library is that it has a user-friendly interface and that it's implemented in headers only, meaning it's very easy to add to existing project ecosystems. Typical log calls using this library take about 25 microseconds on average, which is the length of an entire audio sample at the common sampling rate of 44.1kHz. I'll be using this library going forward, but this means that I will have to reduce the rate at which I can log environment context information; for now, I might limit myself to only logging user actions and information about the code only when it is first run and when it errors/crashes.