OpenMixer vs. Systemd
Introduction
OpenMixer is a multichannel audio routing and mixing system designed for our Listening Room. It turns one of our fanless Linux workstations into an appliance that runs OpenMixer on boot. This was happily working for a long time in Fedora 14 but a hardware upgrade of the workstation prompted me to also upgrade the operating system to Fedora 17 and bite the bullet of making everything work under systemd... thus this short Ping Note...
Before systemd
In the BSD times (Before SystemD) I started OpenMixer from a simple (and ugly) SysV init script that was configured to start at the end of the multiuser graphical boot process:
#!/bin/bash
#
# openmixer Startup script for the OpenMixer project
#
# chkconfig: - 99 5
# description: The OpenMixer project is an open source diffussion mixer
# processname: openmixer
# pidfile: /var/run/openmixer.pid
# Source function library.
. /etc/rc.d/init.d/functions
# Path to the openmixer script
openmixer='/home/openmixer/openmixer'
prog=openmixer
pidfile=/var/run/openmixer.pid
lockfile=/var/lock/subsys/openmixer
RETVAL=0
start() {
echo -n $"Starting $prog: "
# hack: make sure the openmixer user owns the alsa devices
chmod 666 /dev/snd/*
# end hack
daemon --user=openmixer --pidfile $pidfile $openmixer $OPTIONS
RETVAL=$?
PID=`pidofproc $prog`
echo ${PID} >$pidfile
echo
[ $RETVAL = 0 ] && touch ${lockfile}
return $RETVAL
}
stop() {
echo -n $"Stopping $prog: "
# hack: make sure sclang stops by itself
touch /home/openmixer/control/stop
# end hack
killproc -p ${pidfile} $openmixer
RETVAL=$?
echo
[ $RETVAL = 0 ] && rm -f ${lockfile} ${pidfile}
}
# See how we were called.
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status $openmixer
RETVAL=$?
;;
restart)
stop
start
;;
*)
echo $"Usage: $prog {start|stop}"
RETVAL=3
esac
exit $RETVAL
There is nothing particularly nice about this rc file, but it had worked well for quite a long time. There is more complexity hidden, of course, the "/home/openmixer/openmixer" program is a short perl script that daemonizes itself (ie: detaches from the terminal that launches it), and in turn starts and monitors the status of the real openmixer program which is written in SuperCollider. The SuperCollider code is responsible to starting Jack and all the ancillary programs that support OpenMixer (jconvolver, ambdec, etc).
The SysV init script also takes care of running all the openmixer code as the "openmixer" user and group as there is no need or desire for anything to run as root.
New requirements
I was also running a custom built Jack package instead of the default available for Fedora 14. The stock Jack needed to have dbus running to work, otherwise it refuses to start (I think that is a bug). Dbus is used to talk to Pulseaudio and request ownership of the soundcard. With this Jack can run without problems even when Pulseaudio is present and active, Pulseaudio just gives the soundcard to Jack and all is well. That is not really necessary for OpenMixer but it would be nice to be able to use the stock Jack package.
In this new system I also wanted to have an X server associated with OpenMixer so it could have a GUI interface as well as the existing USB MIDI control surfaces. But again, I did not want a solution that depended on an "autologin" account but rather a second, dedicated X server that would only interact with the SuperCollider program.
After Systemd
Converting an rc script to systemd is (supposed to be) easy. A good starting article on managing the transition is this "systemd for Administrators, Part III" article by Lennart Poettering. Based on it and my previous ugly script I created an initial systemd service (just drop it in /etc/systemd/system/openmixer.service):
[Unit] Description=OpenMixer multichannel audio mixer and router After=multiuser.target [Service] User=openmixer Group=openmixer ExecStart=/home/openmixer/bin/start_openmixer ExecStop=/home/openmixer/bin/stop_openmixer Type=forking [Install] WantedBy=multiuser.target
Nice and small. "User=" and "Group=" were perfect for starting the service using the openmixer user and group. The "start_openmixer" and "stop_openmixer" scripts took care of starting and stopping the system. In particular I could get "start_openmixer" to first start an instance of the X server on a vt and then start the main "openmixer" script going (the same perl program as before) which would have X available for use within the SuperCollider program using its QT GUI.
But of course this did not quite work. While the whole system started up fine, X server and all, the Jack processes ended up with no access to SCHED_FIFO or SCHED_RR scheduling or to locking memory. The whole configuration was correctly setup for normal programs but it just would not work within the context of systemd. I added the "jackuser" and "audio" groups to the "openmixer" user, created a no restrictions /etc/securtiy/limits.d/ configuration file and so on and so forth.
My service can't get realtime!
I finally figured out that for some reason cgroups were not allowing my service to get realtime scheduling. I found at least three workarounds in this page.
But none of them would work at all. For example:
[Unit] Description=OpenMixer multichannel audio mixer and router After=multiuser.target [Service] User=openmixer Group=openmixer ExecStart=/home/openmixer/bin/start_openmixer ExecStop=/home/openmixer/bin/stop_openmixer Type=forking ControlGroup=cpu:/ [Install] WantedBy=multiuser.target
This should relocate the systemd unit to the / control group and that would give complete access to running processes in the SCHED_FIFO or RR scheduling rings without any limits. Regtretfully it did not work.
I got sidetracked for a while when I realized that some of the functionaliry of cgroups can't be made available in an rt patched kernel (like I was using in the audio workstations), RT_GROUP_SCHED and PREEMPT_RT_FULL are mutually incompatible. But booting into the latest and greatest Fedora kernel did not make a difference. No rt privileges. So that was not the problem.
User= and Group=
It took me a long time to find out why, but essentially I had to get rid of the "User=" and "Group=" options and start my service as root. When I did that suddenly the service was able to use realtime privileges:
[Unit] Description=OpenMixer multichannel audio mixer and router After=multiuser.target [Service] ExecStart=/home/openmixer/bin/start_openmixer ExecStop=/home/openmixer/bin/stop_openmixer Type=forking ControlGroup=cpu:/ [Install] WantedBy=multiuser.target
So I would have to manage the ownership of the process myself, but I was doing that before anyway. My startup script now needed a second level script which would be started from "start_openmixer" as the user and group "openmixer". With that in place everything worked, except that the service would shut itself off right after starting. We needed one more incantation to fix that ("RemainAfterExist=true") and this is the complete service description:
[Unit] Description=OpenMixer multichannel audio mixer and router After=graphical.target [Service] ExecStart=/home/openmixer/bin/start_openmixer ExecStop=/home/openmixer/bin/stop_openmixer Type=forking RemainAfterExit=true ControlGroup=cpu:/ [Install] WantedBy=graphical.target
The first level script ("start_openmixer") is:
#!/bin/bash # # Start the openmixer script # (C) 2013 Fernando Lopez-Lezcano # # this script runs as root and is started as a systemd # service (see /etc/systemd/system/openmixer.service) # make sure cpu frequency switching is off /usr/bin/cpupower frequency-set -g performance # launch openmixer su - openmixer -c /home/openmixer/bin/launch_openmixer
And finally the script that really starts OpenMixer ("launch_openmixer") is:
#!/bin/bash
#
# Start X server and openmixer
#
# (C) 2013 Fernando Lopez-Lezcano
#
# this script runs as the "openmixer" user
# where to run it
VT="vt5"
DISP="2"
PIDFILE="/home/openmixer/run/x.pid"
# start the second X server as a background process
nohup /usr/bin/X :${DISP} ${VT} -background none -nolisten tcp /dev/null &
echo $! > ${PIDFILE}
# and set the display environment variable
export DISPLAY=:${DISP}
# wait until the X server accepts connections
XH=`/usr/bin/xhost &>/dev/null`
while [ "$?" == "1" ] ; do
# echo "waiting for X server to start..."
sleep 1
XH=`/usr/bin/xhost &>/dev/null`
done
# allow openmixer to use the display
/usr/bin/xhost +openmixer@localhost &>/dev/null
# launch openmixer
exec /home/openmixer/bin/openmixer
I'm sure this could all be made much cleaner but this works for now. Suggestions welcome!.
X vs. Dbus
As a side effect of running X I found that now dbus gets autostarted by Jack, the reservation logic implemented by Jack and Pulseaudio works as intended, and I can use the stock Jack package in Fedora 17. Why dbus needs X I don't know and don't have the time to find out. I'm sure it is all very logical...
Conclusion
While it was possible to switch to systemd it was by no means easy. The difficult part in this case was giving my systemd service access to realtime scheduling and memory locking while running as a non-root user and group (in short: it does not work).
The workarounds would have worked out of the box except for the fact that I was using User= and Group= to define which user and group the service runs as. Maybe that is documented somewhere, but I had to find out about that interaction and what to do about it the hard way.