Disk space and usage at CCRMA
- Where can you store files here at CCRMA
- How much space you are using?
- How can you optimize your usage?
- How do backups work at CCRMA?
Home directory usage and the file server quotas
The files you create in your home directory reside on one of the disks of our main file server. The amount of disk space you can use is subjected to some restrictions. Every user has a maximum disk quota. You can find out what your disk quota is by running the program "/LocalApps/Hog". Hog will tell you your quota, your current disk usage, whether you are over or under quota, and many other wonderful things. If you are logged in remotely, you can run the program "/Local/Unix/bin/hog" instead... it will provide the same information. There are other programs that can be used for calculating your disk usage, just keep in mind that whatever is the method you use, you have to include the usage of both your home directory and your personal mail spool file (which lives in "/usr/spool/mail<login name>").
Your disk usage is checked periodically by an automatic process which can send you results via email. If you are using 90% or more of your disk quota, you will receive a letter warning you that you are approaching your quota. If you exceed the quota, you will receive a very nasty letter. After you've received 2-3 of these letters, your risk losing your account at any time, so don't ignore these warnings... your account can and will be deleted or disabled without warning.
There are several options for removable storage of files and of course there are programs which you may use to further optimize your usage.
Removable storage
There are several options available at CCRMA to store and retrieve large amounts of information. As always there are tradeoffs between speed, capacity and reliability.
Floppies (1.4Mbytes / 2.8MBytes)
This one is the trivial solution. All slabs (the NeXTs that look like a pizza box), PC's and Mac's have floppy disk drives. Machines running NextStep can read and write Mac, Dos and Unix formatted disks so a floppy is a good data exchange medium. The capacity, however, is just 1.4Mbytes, not much (you can also use 2.88Mbyte floppies in the NeXT drives).
Optical Cartdriges (256MBytes)
This one is for people that like to live dangerously. The optical drives on the NeXT cubes are slowly dying so this is probably not a good long term choice. In fact, if you have important information that is currently living only in opticals I would suggest moving to more reliable alternatives before it is too late.
IMPORTANT TIP: If you are still using opticals write protect them all the time (except, of course, when you actually want to write a file to them). Writing is always the first thing to go when a drive is dying and just browsing through the filesystem causes the operating system to actually write to the drive (because it is updating the inode access time for each file and directory being browsed). If you write protect the optical there's one less possible cause for filesystem corruption.
This is a list of currently known good drives and the machines in which they live:
- cmn6 (Trailer left)
- cmn9 (Nando's office)
- cmn10 (Classroom Annex) [kind of flakey lately]
- cmn16 (Perry's office)
- ccrma (Machine room, not for public use)
Syquest Cartdriges (270MBytes)
Syquests are hard drives with removable cartdriges. They store 270MBytes in each cartdrige (about 220MBytes after formatting and unix filesystem overhead). They are fast so that they can be easily used for storing and playing soundfiles (for example I have found no problems in playing 4 channel soundfiles from them, something which the opticals could never do).
These are the machines that currently have Syquest drives installed (more to come!):
- cmn9 (Nando's office)
- cmn7 (Ballroom front)
- cmn3 (Studio D)
You can buy cartdriges locally from NCA, Fry's or through mail order. They cost between 63 and 72 bucks each, depending on who you buy from and quantity.
DAT backup tapes (1.3 to 2Gbytes)
This is the best option for long term and cheap storage of really big files. There is a publicly available DAT drive connected to cmn10 (Classroom Annex). DDS (Digital Data Storage) DAT tapes for this drive come in two flavors, 60 meters (which can store up to 1.3Gbytes uncompressed) and 90 meters (up to 2Gbytes uncompressed). The drive itself does file compression so that you'll be able to store more than the uncompressed capacity (how much more depends on what kind of files you are storing... soundfiles regretfully don't compress well). Again, you can buy them locally from NCA, Fry's or through mail order. Cost is on the order of 10 - 15 dollars for a 90 meter DDS1 tape.
There is a NextStep specific program called SafetyNet.app that can be used to backup and restore files from tape without having to deal with arcane unix commands like tar or dump. It mantains an online catalog in your home directory so that you have immediate access to the table of contents of all your tapes. When restoring files it fast forwards to the block on the tape where the file starts so it is fairly fast (for a tape drive, of course).
To use SafetyNet.app just double click on "/LocalApps/SafetyNet.app". The online help is clear enough... I recommend creating a couple of directories beforehand to store the Archives and Profiles that you will be generating. They could be "~/Library/SafetyNet/Profiles" and "~/Library/SafetyNet/Archives".
Soundfile storage
We have several disks dedicated exclusively to soundfile storage. Space is allocated in first come first served basis. To get a personal directory in one of the disks you have to contact the system administrator. The disk space is only temporary and can only be obtained if you are working in a project that involves use of big soundfiles. The space may ONLY be used to store soundfiles. It is not a general purpose storage space. You are required to empty the space when your project ends.
What are the /pub and /zap directories for?
There is much more disk space available to you than just what is on the file server. Almost every machine has local directories called /pub and /zap which you can write to. /pub is generally for files which you will be using over a period of time. /zap is only for files which you need immediately as the directory is cleared of all files automatically when you log out.
To put files in one of the /pub subdirectories, create a subdirectory using your own login name, and go for it. If you need to access /pub on a machine which you are not logged into, you can do this via the unix commands "rcp" or "ftp". There are unix man pages on both these programs.
Since there are no quotas of any sort on the /pub disk space, we need to try to cooperate on its usage. Generally, I tell anyone who wishes to write anything there to be sure to back it up before they leave, as anyone and everyone has the "right" to erase it at any time. I also tell people who say they want to erase other people's stuff to be sure they check with the owners first. In other words, be respectful of other people's files, but don't trust them to be respectful of yours. Please do not leave stuff in /pub and forget you put it there.
More sophisticated users are urged to tar and compress the files they leave on these local areas. See the section optimizing space usage for details.
How do backups work at CCRMA?
Files which you leave on the server are backed up to tape 2 or 3 times a week using an incremental backup scheme. If you delete a file accidently, in most cases, it can be recovered quite easily. Send email to the sysadmin with the name of the file and the approximate date of the file you want restored. This date is important: we have backup tapes going back now for 4 years, so any file you have kept around for a long time will likely live on many tapes. Thus you need to specify that you want file x as it was 3 weeks ago, or last Oct., or whatever.
Files which you leave on the local disk areas ("/pub") and on the soundfile server are not backed up at all... it is your responsibility to back them up yourself. Usually people don't back up their files until a disk crash happens and they loose months of work. Surely you don't want to go through that learning experience! See the removable storage section for more details on different options you have. Another possibility is to tar and compress them, then copy them into your account.
Files which you leave on the local disk area "/zap" are deleted as soon as you log out.
Programs you can use to calculate disk usage
You can find out what your disk quota is by running the program "/LocalApps/Hog". Hog will tell you your quota, your current disk usage, whether you are over or under quota, and many other wonderful things. If you are logged in remotely, you can run the program "/Local/Unix/bin/hog" instead... it will provide the same information.
DarkForest is a very nice program to manage your disk usage. It lives in "/LocalApps/DarkForest.app" but is also available as a NextStep Service, so you can, for example, select a directory in the File Manager and go to the "Services" "Size in DarkForest" menu to activate the program. It will show you a browser with the files in it ordered by total size. You can browse through the directory you selected and erased old or unnecessary files.
The Workspace Manager can also give you information about total size of a directory. Just open the inspector window with the directory selected and press on the "Calculate size" button.
du enables you to calculate directory sizes from the unix command line. For more information type "man du" in a terminal window.
Programs you can use to optimize your usage
Tar, compress and gzip are three standard unix utilities with many uses. They are often used together to help you squeeze more information into your alloted disk quota. Tar, which stands for "tape archiver" (the "tape" part in the name is largely an anachronism), takes a group of files and/or directories and combines them into a single file, called a a tar archive. It leaves the original files unchanged. Alternately, tar takes a tar archive (which is a single file) and extracts the file or files you specify from it. Both compress and gzip takes a file as its argument and compresses it into a much smaller file.
Tar has more bells and whistles than most mortals ever master, but the important ones are pretty simple. If you wish to create a tar archive, the proper incantation is
where "archiveName" is the name of the tar archive you wish to create, and "filelist" is a list of the files and/or directories you want to include in the archive. The "cf" tells the tar program that you wish to create a file (i.e. tar archive). For example, suppose I have a directory called "work1". To turn it into a tar archive, I would execute the following:
The name of the archive file is "work1.tar". The name is arbitrary, though virtually everyone uses the ".tar" extension to help them recognize tar files. Some programs even depend on this.
To "undo" this tar file, retreiving the orignal files...
Here, the "xf" tells the tar program you wish to "eXtract file(s)". Tar can selectively extract any file and/or directory in a tar archive (tar xf work1.tar foo), or, if you omit the name of the files you want extracted, it will simply extract everything.
Compress is a program which "squeezes" a file into a much smaller space. A compressed file will often be less than %25 the size of the original. You cannot read or edit the compressed file... it must be "uncompressed" before you can do anything useful with it. To use compress, simply type "compress filename". Compress creates a new file, called "filename.Z", and deletes the original file. Later you can "uncompress" the file via "uncompress filename.Z". Uncompress is the opposite of compress...it takes a compressed file and turns it back into an ordinary file.
Not every file can be compressed efficiently. In particular, most sound files cannot be compressed. Power users may also wish to read "man zcat" for another method of uncompressing files.
Tar and compress are often used together to back up directories. Suppose I have a directory called /user/grd/mydir, and mydir itself contains other directories and files. To back it up, I first create a tar archive...
Now I will "compress" this file.
This creates a file called mydir.tar.Z, in other words, a compressed tar archive. I might then copy mydir.tar.Z onto a floppy (or optical) and delete the mydir directory.
Later, when I wish to restore mydir, I'll copy mydir.tar.Z from the floppy, I'll first copy it into /user/grd. Then I'll "uncompress" it...
and finally I will "untar" it...
Now I will have the mydir directory again, just as it was before I deleted it. As a final step, I'll delete the file mydir.tar, since I don't need it any more..
©1996-2000 CCRMA, Stanford University. All rights reserved.