SmartKeyboard Mobile App Generator Documentation

SmartKeyboard Mobile App Generator Documentation

faust2smartkeyb is a tool to generate ready-to-use musical Android and iOS applications using the Faust programming language. Unlike faust2android and faust2ios, faust2smartkeyb ignores the standard user interface (UI) declared in the Faust code (e.g., hslider, button, etc.), and replaces it by a SmartKeyboard UI.

The SmartKeyboard UI allows to implement a wide range of controllers (basic keyboards, isomorphic keyboards, pads, X/Y controllers, etc.) on a touch-screen and can be configured directly in the Faust code using the SmartKeyboard metadata.

This documentation demonstrates how to use faust2smartkeyb and provides a series of links to tutorials on how to turn mobile devices into musical instruments.

faust2smartkeyb is part of the Faust distribution and will be automatically installed on your system with the latest version of Faust.

WARNING: this tool is still being beta tested and there are probably a few bugs. If you find a bug, please report it to rmichon_at_ccrma_dot_stanford_dot_edu.

Compatibility

Apps generated with faust2smartkeyb should work on any iOS device running a "reasonably recent" version of iOS (less than 4 years old). Things are a bit more restrictive on Android and target devices must run at least on Jelly Bean (version 4.1). However, for optimal performances (especially regarding latency), we recommend you to be at least on KitKat (5.0). Interested readers might want to read this page on Android audio latency to learn more about this topic.

Setting Up Your System

This section shows how to configure your system to use the various features of faust2smartkeyb.

iOS

Aaaaah iOS... This OS has been (and still is, even though Android is catching up fast) by far the best one to make musical apps involving real-time audio DSP. However, Apple likes to make the life of developers hard, and in order to develop apps for this platform, you used to need to have an Apple developer account ($100/year, yes). Things changed since then, and it is now possible to develop iOS apps and upload them on your device for free! However, the number of apps you can install using this technique is very limited (5, if I remember correctly). Oh, and needless to say that YOU WILL NEED A MAC to generate iOS apps with faust2smartkeyb.

In any case, to use faust2smartkeyb (as well as faust2ios) or to do any kind of iOS development in general, you will either need a "pro" Apple developer account or a free one. This documentation doesn't show how to take care of that. For more information about this, visit the Apple website. Good luck!

Once you took care of this, make sure that Xcode is properly installed on your system (using the Apple Store, not some weird technique) and that it is UP-TO-DATE. A little bit more Apple c[...]p: if the iOS device you're using is running the latest version of iOS, then you will need to have the latest version of the iOS SDK installed on your system. Unfortunately, the only way to do that is to also have the latest version Xcode which will require the latest version of OSX! Once again, good luck!

After all these steps, you should be ready to go.

Android

Android apps can be developed on all major platforms (Windows, OSX and Linux). Unlike Apple (see previous section), Google provides a lot more flexibility to app developers and it is not required to sign up for any developer account in order to install apps on your device. On the other hand, the installation of the Android development tool chain is a little bit more complex than on iOS. It should be quite similar on Linux and OSX and we describe it below. We don't provide information for Windows since faust2smartkeyb will probably not work at all on this platform (sorry).

First, install Android studio. When you run it for the first time, we advise you to choose the option where you don't import previous settings (unless you know what you're doing, of course). Then, during the configuration process, choose a "Standard" setup. The sdk installation path should be prompted to you after that. Take note of it as we'll need it later (should be ~/Library/Android/sdk on OSX). Once Android studio finished setting up, click on "Configure" and then "SDK Manager" (you might have to go in "Tools/Android/SDK Manager" on Linux). In the SDK Tools tab, install the NDK (Native Development Kit) which is necessary to compile apps generated by faust2smartkeyb (faust2smartkeyb uses faust2api internally, if you want to get more details about why you need to do all that, have a look at the faust2api documentation).

In order to be able to compile Android apps in your terminal, you must configure the ANDROID_HOME and ANDROID_NDK_HOME environment variables so that they point to where the sdk and the ndk are installed. To do this, add the following line to ~/.bashrc on Linux or ~/.bash_profile on OSX (if this file doesn't exist, create it):

export ANDROID_HOME=/home/r/Android/Sdk
export ANDROID_NDK_HOME=$ANDROID_HOME/ndk-bundle

where paths should be replaced with the ones on your system, of course.

Finally, if you want to be able to install the generated Android apps directly from faust2smartkeyb, you will have to install adb on your system using your package manager.

That's it! You're now ready to forge Android apps with faust2smartkeyb!

Quick Tour: From a Simple Faust Code to a Working App

faust2smartkeyb uses faust2api internally. Thus, Faust codes provided to faust2smartkeyb must respect the same standards as for faust2api (check the faust2api documentation for more information about this).

The following code is similar to the one presented in the faust2api documentation in the MIDI Enabled Polyphonic Object:

import("stdfaust.lib");
freq = nentry("freq",200,40,2000,0.01);
gain = nentry("gain",1,0,1,0.01);
gate = button("gate");
cutoff = 1000;
envelope = gate*gate : si.smoo;
process = os.sawtooth(freq)*envelope : fi.lowpass(3,cutoff) : *(0.25) <: _,_;

It implements a simple synthesizer based on a filtered sawtooth wave and it is polyphony-compatible thanks to its freq, gain, and gate parameters. Note, that the gain of the synth is scaled. This is done to prevent clicking if several polyphony voices are played at the same time. When using an effect with a synth, it's better practice to carry out this type of gain scaling in the effect section to save up computation (see effect).

If this code is provided as such to faust2smartkeyb, its UI declaration will be ignored and replaced by the default SmartKeyboard interface. This interface can be configured at the beginning of the Faust code using the SmartKeyboard metadata:

declare interface "SmartKeyboard{
    // configuration keys
}";

and by placing a set of key/value pairs between the two curly brackets. For example,

declare interface "SmartKeyboard{
    'Number of Keyboards':'2',
    'Keyboard 0 - Number of Keys':'13',
    'Keyboard 1 - Number of Keys':'13',
    'Keyboard 0 - Lowest Key':'72',
    'Keyboard 1 - Lowest Key':'60'
}";

will create an interface with 2 keyboards of 13 keys each. The lowest note of the top keyboard will be MIDI note 72 (C4), and the lowest note of the other keyboard will be 60 (C3).

A SmartKeyboard interface can stream a set of standard Faust parameters to control the given Faust DSP. For example, the cutoff frequency of the lowpass filter of the previous Faust code can be controlled with the Y position of the finger on the key simply by adding 'Keyboard 0 - Send Y':'1','Keyboard 1 - Send Y':'1' to the interface declaration and by using the associated standard y parameter:

fingerY = nentry("y",0.5,0,1,0.01) : si.smoo; // y is always normalized between 0 and 1
cutoff = fingerY*1960+40; // mapping

The two following sections give an overview of the different configuration keys and standard parameters that can be used with SmartKeyboard interfaces. Also, the Additional Resources section provides links to tutorials on how to design various kinds of instruments using this system. We recommend you to check these resources since complex mappings (that are not presented here) can be created by combining different interface configurations with specific uses of standard parameters. Example codes can be found in the /examples/smartKeyboard folder of the Faust distribution. Finally, the Compilation section demonstrates how to compile Faust codes such as the one presented above using faust2smartkeyb.

SmartKeyboard Configuration Keys

This section presents the different configurations keys of SmartKeyboard and their function. For practical use cases, check the Additional Resources section.

`Inter-Keyboard Slide`

When 1, fingers can slide between keyboards.

Default value: 1

`Keyboard M - Key N - Label`

Allows to set the text inside a key on a specific keyboard. The corresponding value should be a string.

Default value: null

This is a keyboard and key specific parameter where M is the keyboard number and N the key number.

`Keyboard N - Lowest Key`

Defines the MIDI note number of the lowest key on a specific keyboard.

Default value: 48

This is a keyboard-specific parameter. For example, if Number of Keyboards = 2, then there are 2 keyboards in the interface that can be configured independently with the Keyboard 0 - Lowest Key and the Keyboard 1 - Lowest Key keys.

`Keyboard N - Number of Keys`

Defines the number of keys of a specific keyboard in the interface.

Default value: 13

This is a keyboard-specific parameter. For example, if Number of Keyboards = 2, then there are 2 keyboards in the interface that can be configured independently with the Keyboard 0 - Number of Keys and the Keyboard 1 - Number of Keys keys.

`Keyboard N - Orientation`

Defines the orientation of a specific keyboard: left to right when 0 and right to left when 1.

Default value: 0

This is a keyboard-specific parameter. For example, if Number of Keyboards = 2, then there are 2 keyboards in the interface that can be configured independently with the Keyboard 0 - Orientation and the Keyboard 1 - Orientation keys.

`Keyboard N - Piano Keyboard`

When 1, note names are displayed and black and white keys are differentiated (if 0, all keys are white).

Default value: 1

This is a keyboard-specific parameter. For example, if Number of Keyboards = 2, then there are 2 keyboards in the interface that can be configured independently with the Keyboard 0 - Piano Keyboard and the Keyboard 1 - Piano Keyboard keys.

`Keyboard N - Root Position`

Position of the root (as a key number starting from 0) on a specific keyboard. This parameter is very useful when dealing with specific scales.

Default value: 0

This is a keyboard-specific parameter. For example, if Number of Keyboards = 2, then there are 2 keyboards in the interface that can be configured independently with the Keyboard 0 - Root Position and the Keyboard 1 - Root Position keys.

`Keyboard N - Scale`

Defines the scale of a specific keyboard.

Default value: 0

With:

Keyboard N - Scale = 0: Chromatic scale, note names are displayed
Keyboard N - Scale = 1: Chromatic scale, note names are not displayed
Keyboard N - Scale = 2: Major scale, note names are not displayed
Keyboard N - Scale = 3: Harmonic minor scale, note names are not displayed

Custom scales can be created by specifying an array of intervals. For example, the chromatic scale can be defined as Keyboard N - Scale = {1,1,1,1,1,1,1,1,1,1,1,1} where the 1s are the intervals in semitones between each note. The only rule is that the sum of the intervals specified in the array must always be equal to 12. For example, the major scale can be declared as Keyboard N - Scale = {2,2,1,2,2,2,1}.

`Keyboard N - Send Freq`

When 1, the freq and bend parameters are computed and sent to the Faust DSP object.

Default value: 1

`Keyboard N - Send Key X`

When 1, send the normalized X position of the finger in the current key (associated with the kbMkNx standard parameter where M is the current keyboard and N is the current key).

Default value: 0

This is a keyboard-specific parameter. For example, if Number of Keyboards = 2, then there are 2 keyboards in the interface that can be configured independently with the Keyboard 0 - Send Key X and the Keyboard 1 - Send Key X keys.

`Keyboard N - Send Key Y`

When 1, send the normalized Y position of the finger in the current key (associated with the kbMkNy standard parameter where M is the current keyboard and N is the current key).

Default value: 0

This is a keyboard-specific parameter. For example, if Number of Keyboards = 2, then there are 2 keyboards in the interface that can be configured independently with the Keyboard 0 - Send Key Y and the Keyboard 1 - Send Key Y keys.

`Keyboard N - Send Key Status`

When 1, send the status of the current key (associated with the kbMkNstatus).

Default value: 0

This is a keyboard-specific parameter. For example, if Number of Keyboards = 2, then there are 2 keyboards in the interface that can be configured independently with the Keyboard 0 - Send Key Status and the Keyboard 1 - Send Key Status keys.

`Keyboard N - Send Numbered X`

When 1, send the normalized X position of the finger in the current key for a specific finger (associated with the xN standard parameter).

Default value: 0

This is a keyboard-specific parameter. For example, if Number of Keyboards = 2, then there are 2 keyboards in the interface that can be configured independently with the Keyboard 0 - Send Numbered X and the Keyboard 1 - Send Numbered X keys.

`Keyboard N - Send Numbered Y`

When 1, send the normalized Y position of the finger in the current key for a specific finger (associated with the yN standard parameter).

Default value: 0

This is a keyboard-specific parameter. For example, if Number of Keyboards = 2, then there are 2 keyboards in the interface that can be configured independently with the Keyboard 0 - Send Numbered Y and the Keyboard 1 - Send Numbered Y keys.

`Keyboard N - Send X`

When 1, send the normalized x position of the finger in the current key (associated with the x standard parameter).

Default value: 0

`Keyboard N - Send Y`

When 1, send the normalized y position of the finger in the current key (associated with the y standard parameter).

Default value: 0

`Keyboard N - Show Labels`

When 1, shows note names on a specific keyboard. This parameter is overridden if Keyboard N - Piano Keyboard is defined.

Default value: 1

This is a keyboard-specific parameter. For example, if Number of Keyboards = 2, then there are 2 keyboards in the interface that can be configured independently with the Keyboard 0 - Show Labels and the Keyboard 1 - Show Labels keys.

`Keyboard N - Static Mode`

When 1, keys don't change color when touched.

Default value: 0

This is a keyboard-specific parameter. For example, if Number of Keyboards = 2, then there are 2 keyboards in the interface that can be configured independently with the Keyboard 0 - Static Mode and the Keyboard 1 - Static Mode keys.

`Number of Keyboards`

Defines the number of keyboards in the interface.

Default value: 1

`Max Fingers`

Defines the maximum number of fingers allowed on the touch-screen. This parameter is independent from the number of polyphony voices of the DSP object.

Default value: 10.

`Max Keyboard Polyphony`

Defines the number of polyphony voices of each keyboard (and more, see below).

Default value: 16

This parameter has some special cases:

Max Keyboard Polyphony = 1: in that case, keyboards are monophonic. Monophonic mode can be configured using the Mono Mode key.
Max Keyboard Polyphony = 0: the first voice of the DSP object is activated. This is very convenient to create an app generating a continuous sound or based on a physical model of a musical instrument. The voice allocation system is deactivated in that case (although the freq and bend parameters can still be sent using the Keyboard N - Send Freq option).

`Mono Mode`

Configures the way monophonic keyboards work. Keyboards are made monophonic by setting Max Keyboard Polyphony to 1.

Default value: 1

With:

Mono Mode = 0: the finger currently on the keyboard keeps priority: all other fingers are ignored.
Mono Mode = 1: priority goes to any new finger on the keyboard. When the finger holding the note leaves the keyboard, the note is transferred to the closest finger on the same keyboard. This mode is probably the most natural one to implement a guitar interface, for example.
Mono Mode = 2: priority goes to any new finger on the keyboard. When the finger holding the note leaves the keyboard, the note is terminated.
Mono Mode = 3: same as Mono Mode = 1, but priority is given to new fingers only if they are at higher pitch than the current note.
Mono Mode = 4: same as Mono Mode = 1, but priority is given to new fingers only if they are at lower pitch than the current note.

`Rounding Cycles`

The number of cycles before rounding is activated.

Default value: 5

`Rounding Mode`

Configures the way the bend parameter associated with the current finger is quantized.

Default value: 0

With:

Rounding Mode = 0: keys are rounded to the nearest integer: bend is always equal to zero.
Rounding Mode = 1: bend is continuous (new notes might sound out of tune).
Rounding Mode = 2: keys are rounded to the nearest integer when the finger is not moving or when a new note is started but become continuous when fast movement are happening. The behavior of this system can be fine-tuned using the Rounding Update Speed, Rounding Smooth, Rounding Threshold and Rounding Cycles parameters.

`Rounding Smooth`

The pole of the integrators used for smoothing movements during rounding.

Default value: 0.9

`Rounding Threshold`

Rounding is deactivated when the output of the smoothers (see Rounding Threshold) goes above this value.

Default value: 3

`Rounding Update Speed`

Speed in seconds at which the rounding loop is updated.

Default value: 0.06

`Send Current Key`

When 1, send the number of the current key in the current keyboard (associated with the key standard parameter).

Default value: 1

`Send Current Keyboard`

When 1, send the number of the current keyboard (associated with the keyboard standard parameter).

Default value: 1

`Send Fingers Count`

Send the number of fingers present on each keyboard using the kbNfingers standard parameter where N is the keyboard number.

Default value: 0

`Send Sensors`

When 1, sends the raw sensor (accelerometer and gyroscope) values to the DSP object. If the acc metedata is declared with one of the parameters, sensors data will be used to control this parameter through a specified mapping.

Default value: 1

SmartKeyboard Standard Parameters

This section presents the different standard Faust parameters that can be used with SmartKeyboard interfaces. For practical use cases, check the Additional Resources section. Also, example codes can be found in the /examples/smartKeyboard folder of the Faust distribution.

`freq`

The reference frequency in Hz of the current event. This value is provided at the same time than gate and only once. To implement continuous pitch control such as slides, vibrato, etc. use the bend parameter.

`bend`

A coefficient to multiply to freq to bend it (bend = 1 corresponds to no bend so continuousFreq = freq*bend). This parameter can be used to implement vibrato, slides, etc. It heavily relies on Rounding Mode.

`gate`

The trigger signal sent when a finger touches the screen (1) or when it stops touching it (0). If Rounding Mode = 0, then "0" and "1" are sent everytime a finger slides to a new key (a new voice is allocated everytime).

`key`

The key ID in the current keyboard.

`keyboard`

The ID of the current keyboard.

`kbNfingers`

The number of fingers present on a specific keyboard N.

`kbMkNstatus`

Status of the current key N in keyboard M with:

status==0: finger up on the key
status==1: finger down on the key
status==2: finger is moving inside the key
status==3: finger moved to another key
status==4: finger moved from one key to the current key

`kbMkNx`

The normalized (0-1) X position of the finger in a specific key where M is the current keyboard and N is the current key.

`kbMkNy`

The normalized (0-1) Y position of the finger in a specific key where M is the current keyboard and N is the current key.

`x`

The normalized (0-1) X position of the finger in the current key.

`y`

The normalized (0-1) Y position of the finger in the current key.

`xN`

The normalized (0-1) X position of the finger in the current key for a specific finger where N is the finger ID in order of appearance on the screen starting from 0 (e.g., x0 for the X position of the first active finger to touch the screen, x1 for the X position of the second active finger to touch the screen, etc.).

`yN`

The normalized (0-1) Y position of the finger in the current key for a specific finger where N is the finger ID in order of appearance on the screen starting from 0 (e.g., y0 for the Y position of the first active finger to touch the screen, y1 for the Y position of the second active finger to touch the screen, etc.).

Compilation

Overview

We assume that you followed the steps described in the Setting-Up Your System section before reading this. Only 2 arguments are required to use faust2smartkeyb: the target platform (iOS or Android) and a Faust code:

faust2smartkeyb -ios mySynth.dsp

will compile mySynth.dsp into an iOS app (mySynth.app) with a SmartKeyboard interface and

faust2smartkeyb -android mySynth.dsp

will do the same for Android (mySynth.apk).

While this should be very smooth on Android if your followed the steps in Setting-Up Your System, this will probably not work on iOS. Why? Because the bundle identifier associated with the template app used to compile a SmartKeyboard app for iOS is not yours. So unless you change it directly in the source code of the Faust distribution (/architecture/smartKeyboard/iOS/Faust.xcodeproj) and re-install Faust, it will not work (yes, we know, we know... :( ).

However, in practice, you'll rarely want to compile directly your Faust code into an app package. Instead, you might prefer to create an Android Studio on an Xcode project associated to your Faust code and update it when necessary. This is a much better solution because compiling an app from scratch every time (especially on Android) takes a lot of time (>1 minute in most cases). To do this, you can use the -source option in combination with -reuse:

faust2smartkeyb -android -source -reuse mySynth.dsp

In that case, faust2smartkeyb will not compile mySynth.dsp to an app but will create a folder called faustsmartkeyb.mySynth in the current folder containing an Android Studio project. Every time the previous command will be run, the portion of the app source code corresponding to mySynth.dsp will be updated (if we only used -source without -reuse then faustsmartkeyb.mySynth would be erased and re-created). The same steps can be followed when using -ios.

The app can be easily installed on your device from Xcode or Android Studio. On iOS, bundle identifier issues can be fixed directly in the Xcode project contained in faustsmartkeyb.mySynth which is more convenient than doing it in the Faust source code.

The end of this section gives an overview of the different options that can be used with faust2smartkeyb.

`-android`

Asks faust2smartkeyb to generate an Android application.

`-debug`

Debug mode: prints all the details.

`-effect`

Allows to specify an effect Faust file to be connect to the provided synth file. For example:

faust2smartkeyb -android -effect effect.dsp synth.dsp

Will compile an app where synth.dsp is the Faust code containing the source for the polyphonic synthesizer and effect.dsp the effect chain that will be connected to the output of the polyphonic synth. Indeed, in most cases, it is not necessary to have a different effect chain for each voice.

`-help`

Prints the documentation of faust2smartkeyb.

`-install`

Installs the generated app on any device connected to the computer. This option is only available on Android and requires adb to be installed on your system.

`-ios`

Asks faust2smartkeyb to generate an iOS application.

`-nvoices`

Allows to specify the maximum number of voices of polyphony of the generated synth. This option is only used as a safeguard since only active voices are allocated and computed.

`-reuse`

Asks faust2smartkeyb to reuse the same project and to only update its portion corresponding to the provided Faust code. This option will not prevent compilation from happening. Instead, use -source for that.

`-source`

Asks faust2smartkeyb to only generate the source of the app and to not compile it. The source of the app will be placed in a folder called faustsmartkeyb.faustFileName.

Additional Resources

Version/Licence

Version 0.0.

See LICENCE.md in the SmartKeyboard source code (faust/architecture/smartKeyboard).