40 Watt Productions [Adat HD24 Utilities]

Adat HD24 Utilities

Overview

Command-line utility for reading projects, songs and tracks from an IDE drive formatted using the Alesis ADAT/FST Version 1.10.

Features

Open-source so we can all play with, examine, and change it.
Cross-platform.
About 50x faster than FTP transfers (24MB/sec extracting 24-track songs).
On Linux (and OS X?), only requires a standard Python 2.3 or above installation.
On Windows requires a Python 2.3 or 2.4 installation, plus the 'ctypes' library. Python 2.5 and above include 'ctypes' as standard.

Mis-Features

Only extracts WAV files (not AIFF).
Doesn't handle WAV files greater than 2GB properly.
No audio preview facility (unless you use FUSE).
Command-line version only, no GUI.

Tested on Linux and Windows, but should work on Mac OS X.

Download

hd24-0.0.1.zip (Windows CR/LF, 31K)
hd24-0.0.1.tar.gz (Linux LF, 23K)

Scripts

The main script is hd24.py which contains the command-line utility code. The script adatfst.py contains the code that parses the file-system structures, and extracts the recorded data into WAV files. The script win32disk.py is Windows-only and requires the 'ctypes' module to be installed in addition to a standard Python 2.4 (will work with 2.3) installation.

The script hd24fuse.py is Linux-only and allows an HD24 drive to be mounted (read-only) like any other drive, using the FUSE (File-system in User-space) virtual file-system. See later for the disadvantages to this.

The top-level scripts 'hd24.py' and 'hd24fuse.py' depend on 'adatfst.py', which conditionally depends on 'win32disk.py'.

Hardware

We used a compatible drive-bay installed in a PC and arranged as the Secondary-Master (Alesis drives are set to Master), which involved moving a CD-ROM drive to Secondary-Slave. It was also tested (on Linux) with an external USB2 drive-enclosure and worked fine, but required the IDE drive to be moved from one disk caddie to another.

Installation

Linux
Make sure you have at least Python2.3 installed. As root or using sudo, copy the 'hd24.py' and 'hd24' files into '/usr/bin/'. If you are going to use FUSE (you probably wont, see later) copy 'hd24fuse.py' in there as well.

Note: I added my normal user to the 'disk' group, so that I could access the raw block devices. Don't forget to log out and back in again

$ ls -la /dev/hdX
brw-rw---- 1 root disk 3, 64 2006-08-27 02:17 /dev/hdX
$                 ^^^^

Note: The 'hd24' shell-script start with '#!/bin/env sh' which may not work on your system. On Kubuntu I changed it to '#!/bin/sh'. Likewise, the 'hd24.py' script starts with '#!/bin/env python'. On Kubuntu I changed it to '#!/usr/bin/python'.

Windows
Extract the scripts into a directory like 'C:\hd24\' and add an entry to the path using 'Start|Settings...|Control Panel', double-click on 'System' and then click on 'Environment Variables' under the 'Advanced' tab. (slightly different on XP). Remember that ';' is the path separator.

Note: The user needs to be an administrator or power-user to access the PHYSICALDRIVE.

Test your installation

Linux
By running the following command in a shell prompt:

$ hd24.py /dev/hdX

where X is a->d depending on which IDE cable your caddie and HD24 IDE drive are connected to.

Windows
By running the following in a command prompt:

C:\hd24>c:\python23\python.exe c:\hd24\hd24.py \\.\PHYSICALDRIVEX

where X is 0->3 depending on which IDE cable your caddie and HD24 IDE drive are connected to.

If you get the drive wrong, you will get a message like this:

"Error on Device '/dev/hda': Not an 'ADAT FST' file-system." or
"Error on Device '\\.\PHYSICALDRIVE0': Not an 'ADAT FST' file-system."

If it works, you'll get something like this:

DRIVE: drive-name='Drive Name' num-projects=2 total-songs=7
PROJECTS:
  name='Project01 - Reels'  6 songs:
    name=Song01 - Reel 1                    24trk, 48000Hz file-size=250.4MB song-size=  5.9GB
    name=Song02 - Reel 2                    24trk, 48000Hz file-size=110.1MB song-size=  2.6GB
    name=Song03 - Reel 3                    24trk, 48000Hz file-size=258.2MB song-size=  6.1GB
    name=Song04 - Reel 4                    24trk, 48000Hz file-size=225.8MB song-size=  5.3GB
    name=Song05 - Reel 5                    24trk, 48000Hz file-size= 51.1MB song-size=  1.2GB
    name=Song06 - Reel 6                    24trk, 48000Hz file-size=237.6MB song-size=  5.6GB
    Project size = 26.6GB
  name='Project02 - Test Name'  1 songs:
    name=Song01 - Test                      2trk, 44100Hz file-size=227.1MB song-size=454.2MB
    Project size = 454.2MB
Total disk space required = 27.0GB

To make the 'hd24.py' script easier to use, I have supplied a Windows batch-file hd24.bat, and a bash shell-script hd24, that, once customised with the correct path and drive information, will reduce typing. Once you get the explicit versions above working, edit the 'hd24' (Linux) or 'hd24.bat' (Windows) file appropriately.

Extracting Projects, Songs and Tracks

Using the example drive listing above, we can see that Project01 contains 6 songs, all 24-track and recorded at 48Khz. The 'file-size' is the size of each track when extracted as a WAV file. These files are identical to the ones extracted using FTP so you can check that the script works by comparing files extracted by the script and by FTP using the 'diff' command:

$ diff --binary -r -s mnt/Project01\ -\ Reels /mnt/sda1/Project01\ -\ Reels

The 'song-size' is 'file-size'*number-of-tracks. The 'Project Size' is the sum of the 'song-size's in the project, and the 'Total disk space required' is the sum of the 'Project Size's.

There is a verbose mode (use -v) which displays more information and the exact sizes in bytes.

Change to a directory where you want the files to go, and type:

$ hd24 -x

This will extract all projects from the drive. If you just want one of the projects use this:

$ hd24 -pPP -x

where PP is the number of the project (1-99). If you just want one of the songs, use this:

$ hd24 -pPP -sSS -x

where SS is the number of the song (1-99). It is possible to extract individual tracks, even more than one track, using this:

$ hd24 -pPP -sSS -tTT

where TT is the number of the track (1-24). Multiple -tTT switches can be used to extract multiple tracks at once, but it isn't as efficient as extracting all tracks at once.

Currently it isn't possible to give a single command to extract multiple named projects or multiple named songs or 'all the track01s from each song'. For our purposes (and using smallish drives), extracting projects individually is an unnecessary luxury :-)

Examples

These examples assume you've customised the 'hd24.bat'/'hd24' scripts and that they are on the path.

Built-in help is available by typing:

$ hd24 --help

The program version is displayed by typing:

$ hd24 --version

Extracting all projects to the current directory:

$ hd24 -x

Extracting all projects to a named directory:

$ hd24 -x -o /mnt/sda1/

Extracting Project02 only:

$ hd24 -x -p2

Extracting Project02, Song04 only:

$ hd24 -x -p2 -s4

Extracting Project02, Song04, Track10 only:

$ hd24 -x -p2 -s4 -t10

Extracting Project02, Song04, Track01, Track02 and Track08 only:

$ hd24 -x -p2 -s4 -t1 -t2 -t8

Viewing Project01, Song01 in detail:

$ hd24 -p1 -s1 -v

FUSE

You need the 'fuse.ko' kernel module loaded before trying any of this, and the fuse utilities will be useful as well. FUSE is included in recent kernels, and will probably be available as a package for your Linux distribution.

Create a mount point:

$ mkdir mnt/

Mount the file system:

$ ./hd24fuse.py -s -o drive=/dev/hdX mnt/

You MUST use the -s switch or it will appear to work, but actually produce corrupted files. You have been warned!

When finished with it, unmount it:

$ fusermount -u mnt/

The same directory structure used by the FTP server (except for the WAV and AIFF directories) should be visible at the mount point.

This is useful for previewing the audio, but too slow for extracting (even though it's 5x faster than FTP).

For comparison, when using the 'hd24.py' stand-alone script, copying a complete 24-track project results in a data-transfer rate of about 24MB/sec which is very respectable. But when copying a single track from the same song, the data-transfer rate is a measly 2.4MB/sec (using 'hd24.py' or the FUSE script).

This turns out to be unrelated to Python, FUSE, or fusepython though. It is the disk format itself:

A recording is a series of 0x90000 (576K) bytes blocks, which I call clusters. They are often contiguous, but for each song, there is a list of offsets and counts for each of these contiguous blocks.

Each recording has a fixed number of tracks (the HD24 does 2, 6, 8, 16 and 24 track recordings) and each cluster contains a 'chunk' for each track:

2-track recording:
 <-------------------------- a 576K long 'cluster' ------------------>
 +-------------------------------------------------------------------+
 |[         this is track 01       ][        this is track 02       ]|
 +-------------------------------------------------------------------+

24-track recording:
 +-------------------------------------------------------------------+
 |[01][02][03][04][05][06][07][08][09][10]...[19][20][21][22][23][24]|
 +-------------------------------------------------------------------+

For normal use (recording and playback), this is fine and efficient enough to work with standard drives. But if you want to extract a single track, say track04, from a 24-track recording (the way a GUI file explorer would do when copying files) it becomes quite inefficient, and the transfer rate is reduced to the chunk size (24KB in this case) divided by the access-time of the hard-drive (10ms or 0.01sec) resulting in a 2.4MB/sec top speed. This is quite annoying, but goes some way to explaining why the HD24 only has a slow 10MB/sec Ethernet interface, with a one-file-at-a-time FTP server.

The drive in question is 40GB, and a modern 320GB drive might not need to perform a seek for every chunk.

There might be a Konqueror-specific way of notifying a helper KIO driver that a multi-file transfer is taking place, I dunno.

Notes

The script extracts the song locations but doesn't show or use them for anything. The song locations are (name, location) pairs with the location in samples.

A nice GUI front-end could be written for this using e.g. wxPython. A nice installer for Windows could be written to go with it. I started getting bored after the approx. 3-days it took to figure out the file format ...

The FUSE script was fun to write and would be perfect on Linux if it wasn't for the poor performance (not Python or FUSE related) :-(

Licence

I've decided to licence all of this using the LGPL which makes it easy to reuse in another project.

Resources

Python website: http://www.python.org/
FUSE:           http://fuse.sourceforge.net/
ctypes:         http://starship.python.net/crew/theller/ctypes/
Alesis HD24:    http://www.alesis.com/products/hd24/
HD24tools:      HD24tools Project Page (Found after mostly completing this!)

Contact

Author: John Popplewell
Email : john@johnnypops.demon.co.uk
Web   : http://www.johnnypops.demon.co.uk/