CCRMA Documentation links:   index   contents   overview   rooms   account   staff   about


(contents of this file: links to each section)

The Command Line Is Your Friend

The “command line” (aka shell, terminal…) is an ancient way of interacting with computers by typing commands and seeing what they print. All the components of this programming style have been mature and widely available for Unix-like operating systems (including the Mac Terminal) for decades and are likely to persist long into the future of computing.

Piping” these commands together connects them in potentially powerful ways; this creates an “ecosystem” of simple programs like echo, wc, tr, and grep that excel at one task and make sense in context. The programs that process the text are themselves text and can be the output of other programs. Writing raw html is painful and everbody should be able to use markdown instead.

This is great info: CS 107 Unix Guide

What is the Shell?

A command-line shell (aka “Terminal”) is the part of the operating system that allows user interaction via typing commands and seeing what they print. It’s a form of Command-line interface; one way to “talk” to “a computer.” There are many Unix shells, often with names ending with “sh” (with bash one of the most common); they have important differences but are fundamentally similar. A shell is an interpreter for a programming language that excels at many kinds of tasks.

Everything occurs as a series of text within a specific window or teleprinter. First the shell prints a prompt, often just a single character such as $, telling you it’s ready for input. Then you type your command (followed by the return key), then the command runs, possibly printing output text (to the same window, screen, or teleprinter , writing to files, making sound, etc. When the command finishes the shell prints the next prompt and it starts all over; in normal usage you go back and forth dozens or hundreds of times, issuing commands and seeing what they print.

Each commmand begins with the name of a program, which is a series of letters and numbers. This might be one of the many standard programs found on most Unix-like operating systems or the name of an executable file in your search path, perhaps one of your own shell scripts.

For example, the program named date will print the current time:

$ date
Tue Apr  6 15:49:09 PDT 2021

After the program’s name there could be space followed by arguments to the program. For example, the program echo simply prints out the arguments it receives:

$ echo pleased to meet you
pleased to meet you

An option (aka flag or switch) is a special kind of argument that controls the behavior of the program in a case-by-case way. (Each program has its own options.) For example, the program du (“disk usage”) can take the option -s to print only a summary of the disk usage:

$ du -s
126016  .

It can also take the option -h to print the result in “human-readable” form:

$ du -s -h
 62M    .

Equivalently, these two options can be written together:

$ du -sh
 62M    .

More simple commands include:

Getting a Terminal

OSX
It’s a built in “Utility” named Terminal, aka /Applications/Utilities/Terminal.app
windows
10 Ways to Open the Command Prompt in Windows 10
Linux
It’s the most basic thing built into the desktop environment. Worst case there’s xterm.

Installing Shell Commands

Many shell commands come pre-installed with your operating system. To add more, you probably want to use a package manager.

Shell Features

The shell itself has many nice and powerful features keeping track of what you already typed or might type so you don’t have to.

Tab Completion

The best thing ever. You don’t have to type full names (of commands, pfiles…); just the prefix then the tab key and it will auto-complete.

See Autocomplete: CS107 Unix Guide

Shell History

reference past shell events

Scripts

You put any commands you want into a text file and then you can call them just by typing the name of the file!

shell script

The script can optionally start with a line saying which shell or programming language is supposed to interpret the script, a “shebang”, like so:

 #!/bin/bash

You probably have to make your script executable to be able to run it, like with chmod

Filesystem Structure

All the files on the computer have a directory structure that you can access from the command line.

Directory structure absolute / relative paths Being “in” a directory

See The Filesystem: CS107 Unix Guide and Commands To Navigate The Filesystem

The tree command (works at CCRMA, not Macintosh) is great for seeing this.

Pipe

One of the most powerful parts of Unix-like operating systems is the ability to redirect programs (text) input and/or output to and from files and other programs.

There are three special characters:

<
Read input from a file
>
Write output to a file
|
Make one program’s output be the input to the next program (Like => in chuck)

Unix Pipeline

Processing Text Files Line by Line

Extract the fourth field from a TSV tab-separated values file:

cut -f4 myTSVfile.tsv

Extract the fourth field from a CSV comma-separated values file:

cut -d, -f4 myCSVfile.csv

Extract the fourth field from a CSV file, keeping only those that “look like an email address”, i.e., letters and numbers followed by @ followed by letters and numbers containing at least one .:

cut -d, -f4 myCSVfile.csv | grep -E "[[:alnum:]]+@[[:alnum:]]+\.[[:alnum:]]+"

Same thing, but extracting only the portion(s) of that field that look(s) like an email address:

cut -d, -f4 myCSVfile.csv | \
    sed 's|.*\([:alnum:]\+@    XXX unfinished
    

Suppose you have a folder full of files that should each contain a certain string. You want to extract the text after that string and until another string, then make a single file containing just that one extracted bit from all the files, sorted.

s="url=https://ccrma.stanford.edu/~"
fgrep "$s" *.html | awk -F "$s" '{print $2}' | awk -F / '{print $1} | sort

Here s is a bash variable that holds the certain string, and the second string is just a single forward-slash character /.

grep

Print the lines of a file that contain a substring or match a pattern: grep

awk

Double-space a file:

awk '{ printf("%s\n\n", $0); } ' singlespaced.txt > doublespaced.txt

Quoting

Sorry, this is the part that makes bash scripting really ugly.

Some characters have special meanings, for example * has to do with sets of filenames. Normally the shell treats * specially (replacing the argument containing * with all the filenames that match the pattern). So if you really mean the character * you have to quote it somehow, e.g., with a backslash or 'single quotes'.

$ echo *
[prints the name of every file in the current directory]
$ echo \*
*
$ echo '*'
*

To have a command continue onto the next line you can put a backslash at the very end (thereby quoting the newline character ending the line, so that it doesn’t have its normal meaning of “this is the end of the command; please run it now” but rather it’s just another space character dividing the “words” of the command):

$ echo "hello" | \
> tr 0 a
hello

(Note that the shell prints not just the usual ready-for-the-next-command prompt $ but also the ready-for-the-next-line-of-a-multiple-line-command prompt >.)

Iteration

Bash loops

for / while / continue / break

You can write the loop “interactively” from the shell prompt; with each subsequent > prompt wanting more of the for loop until it’s done:

$ for s in *
> do
> echo == $s
> fgrep "url=https://ccrma.stanford.edu" $s
> done

Then in terms of the shell’s history functions, it’s as if you’d typed it all on the same line; for example, the up-arrow button would give this:

$ for s in *; do echo == $s; fgrep "url=https://ccrma.stanford.edu" $s; done

The point of this particular example is to show which files in the directory do not contain a particular string (url=https://ccrma.stanford.edu), by printing the name of each (starting with something easy to parse visually, here ==) and then grepping for the string. If two consecutive output lines both begin with ==, then the first gives the name of a file that doesn’t contain the string.

find

The find command

Remove all .DS_Store files within the current directory:

find . -name .DS_Store -delete

Use Cases

Processes

Unix-like operating systems simultaneously run multiple processes.

You can list them with ps.

You can stop a naughty process with kill or sometimes you need kill -9

Media Use Cases

sox

e.g., to convert a 10-channel sound file to 9 channels:

$ sox input.wav output.wav remix 1 2 3 4 5 6 7 8 9

ffmpeg

Do anything with video files, e.g., stitch together 1000 still images into a movie file while adding a soundtrack from another file.

ecasound

ecasound (https://ecasound.seul.org/ecasound) is an open-source command-line DAW; it can record multichannel etc

Also MrsWatson a command-line audio plugin host.

Macintosh-only

open

The command-line way to emulate double-clicking on any file. For directories it will open a Finder window. For html files it will open the page in your default browser. Etc.

macintosh$ open .
macintosh$ open foo.html
macintosh$ open foo.maxpat

The inverse of this is that if you drag a folder or file from the Finder onto a Mac Terminal, the full path to that file will appear as if you’d just pasted it. So you might type cd plus a space, then drag a folder onto the Terminal window.

pbcopy / pbpaste

Interaction with the macOS copy/paste clipboard (aka “paste” board, hence pb).

pbcopy
Read from stdin (e.g., a pipe) into the clipboard (e.g., so you can go to another application and type command-V to “paste”). Copy from the commandline to the clipboard.
pbpaste
Dump out the current contents of the clipboard to stdout (e.g., so you can pipe it to something). Paste from the clipboard to the commandline.

Example: How many words are in the copied text?

pbpaste | wc -w

Example: suppose you have selected and copied a range of cells in a spreadsheet in a web browser, and you want to know how many unique values it contains. (Converts tabs to newlines with tr '\t' '\n' so each cell becomes its own line.)

pbpaste | tr '\t' '\n' | sort | uniq

Example: use the command-line (sed) to search+replace within the currently-copied text:

pbpaste | sed -e ‘s/foo/bar/g’ | pbcopy

Example: use octave with no GUI (the flag --no-gui-libs makes it instead a good old fashioned command-line program) to make up a string of 20 random characters (i.e., to be a password), save them in the copy buffer, and also print them:

#!/bin/bash

echo 'length=20; disp(char(int8(48+rand(1,length)*(122-48))))' \
     | octave --no-gui-libs | pbcopy
pbpaste

say

Text-to-speech.

macintosh$ say help I am trapped inside this computer
macintosh$ say -v Victoria hello my name is victoria
macintosh$ say -v \?

Matt’s saynames.sh script:

voices=`say -v \? | awk '{print $1}'`;

for v in $voices; do
    echo "My name is $v"
    say -v $v "My name is $v"
done

osascript

You can invoke AppleScript commands with osascript.

Open Sound Preferences and go to the “input” tab:

osascript -e 'tell application "System Preferences" to activate'
osascript -e 'tell application "System Preferences" to reveal anchor "input" of pane id "com.apple.preference.sound"'

Computer Music Utilities

jacktrip

You can launch jackd and jacktrip then make and break JACK connections, all from command-line.

Max

Convert a text file into one that the Max/MSP coll object can read (line number, then a comma, then the data, then a semicolon):

awk '{printf("%d, %s;\n", NR, $0)}' data.txt > data.coll.txt

SLOrk scripts

Every SLOrk piece has one or more corresponding bash scripts that launch and initiate any software that needs to run on a particular laptop in order to play the piece.

The most commonly run “piece” is hemi-test, just a series of clicks out the six channels of the hemispherical loudspeaker attached to this machine:

system-sound-to-motu
pushd ~/slork/users/ge/hemi-test/
chuck --bufsize1024 -c8 impulse.ck
popd

This first sets the system sound to use a particular audio interface, by calling another script system-sound-to-motu (which uses another program audiodevice that is not built in). Then it uses pushd to go to the directory containing the files for the piece, invokes chuck from the command line to run a certain chuck program with certain settings, then uses popd to go back to the original directory.

At least one script uses pipe, the one for Perry Cook’s piece “Take it for Granite” (YouTube). A user interface written in tcl/tk outputs text, which the Chuck program parses in realtime to control the sound generation. The script granite starts the software (using the same pushd/popd technique as hemi-test):

pushd ~/slork/users/prc/granite/
wish granite.tcl | chuck -c8 granite.ck
popd

The script could take an argument such as a number from 1 to 12 for “which player am I?”, for example allegory (which in turn passes the client number to an even more complicated script feedjack-client):

 #!/bin/bash

if test $# -ne 1; then
    echo usage: $0 [client-number] 
    exit -1
fi


pushd ~/slork/users/matt/allegory-2018/
open Allegory-program-notes.rtf
open Allegory-detailed-performer-instructions.rtf 
open ~/slork/users/matt/max-choose-JackRouter.maxpat
open allegory.maxpat
popd

feedjack-client $1 ~/slork/users/matt/allegory-2018/allegory.maxpat

And in case you can’t remember the names of the scripts needed for this concert’s pieces, there’s a script that prints (a text file containing) the names of the other scripts, e.g., print2019scripts:

FILE=/Users/slork/slork/docs/perform/2019.06.08/bing2019.txt

cat $FILE
echo $FILE

And in case you can’t remember what year it is or type more than one letter, there’s another script named simply p, updated annually, that calls the print script for this year. The file p literally contains nothing more than the name of the other script:

print2019scripts

Last but not least, the most popular script die, which forcibly kills every program that might be lingering after running a SLOrk piece:

killall chuck; killall miniAudicle; killall audicle
kill -9 `ps -ef | grep /Applications/Max.app | awk '{print $2}'`
kill -9 `ps -ef | grep /Applications/FaceTime.app | awk '{print $2}'`
kill -9 `ps -ef | grep /Applications/Wekinator.app | awk '{print $2}'`
kill -9 `ps -ef | grep FaceOSC_Wekinator_14Inputs.app | awk '{print $2}'`
kill -9 `ps -ef | grep /users/lja/vr2019 | awk '{print $2}'`

killall jacktrip; killall jackd

killall FaceOSC; killall python
killall Preview
SLOrk feedjack scripts

You don’t want to know.

Websites

curl

The command-line way to download one file from a web server. For example, you could download and print the source for this very web page:

curl https://ccrma.stanford.edu/docs/common/command-line.md

wget

The command-line way to download a web page, entire website, etc. The -r flag makes it recursive (grabbing not just the page but also the pages that the page links to, and the pages those link to, etc., up to the “level” of recursion set with - l).

Example: download a copy of Graham Coleman’s Chuck tutorial from the Internet Archive WayBack machine, specifically, from this 2015 snapshot, including the files that it links to, but only the ones from his website, not the ones that are links to chuck documentation, etc.

wget -r -l 3 --accept-regex '.*/web.archive.org/.*/~gcoleman/.*' \
  http://web.archive.org/web/20150320041518/http://www.dtic.upf.edu/~gcoleman/chuck/tutorial/tutorial.html

Scripting pandoc

pandoc, “a universal document converter” (pandoc.org) is one of the best things ever.

Among many many other formats, it can convert from [markdown format](https://en.wikipedia.org/wiki/Markdown] to HTML, so you never have to code raw HTML again:

pandoc foo.md -o foo.html

Convert every .md file in this directory to the corresponding .html file:

for m in *.md; 
    do 
    echo converting $m; 
    pandoc $m -o `basename $m md`html; 
done

One-line version:

for m in *.md; do pandoc $m -o `basename $m md`html; done

Go through every Markdown file (whose name ends with .md) within the current directory (recursively looking in sub-sub-directories etc.) looking for a certain URL (https://en.wikipedia.org):

$ grep -nri https://en.wikipedia.org . --include \*.md 
[...prints many lines of many files...]

Whoa that’s a lot! Same thing pipe wc:

$ grep -nri https://en.wikipedia.org . --include \*.md | wc
     182     528   17428

OK, but how many unique links are there? XXX

Parsing html with pup

Pup “is a command line tool for processing HTML.”

Extract all the links from an html file:

pup 'a attr{href}' < foo.html 

Extract all the headings from an html file:

pup 'h1,h2,h3,h4,h5,h6' < foo.html

CCRMA Docs website

This page is part of an overall “CCRMA Docs” documentation effort with a somewhat detailed “About” / README page including links to the scripts that build the site including a simple navigation system and an automatically-generated list of pages and overall table of contents. Each page links to its own section of the overall table of contents.

See Also

https://en.wikipedia.org/wiki/In_the_Beginning..._Was_the_Command_Line


This page of CCRMA documentation last committed on Wed May 19 18:43:12 2021 -0700 by Matthew James Wright.