Writing udev rules

by Daniel Drake (dsd)
Version 0.55

The most recent version of this document can always be found at:
http://www.reactivated.net/udevrules.php

Contents

  1. About this document
  2. History
  3. Software versions used at time of writing
  4. Terminology: devfs, sysfs, nodes, etc.
  5. Why? (The purpose of this document)
  6. The basics of writing rules
  7. Additional automated customisation for NAME and SYMLINK parameters
  8. Using shell-style pattern matching in keys
  9. Key-writing basics
  10. Identifying devices through basic keys
  11. Identifying devices through SYSFS files
  12. Example: Writing a rule for my USB printer
  13. Example: Writing a rule for my USB-Storage digital camera
  14. Additional notes on writing rules for USB storage
  15. Example: Writing convenience rules for my CD drives
  16. Example: Writing a rule to name my network interface
  17. Tips for finding the appropriate places in SYSFS
  18. Using multiple SYMLINK style rules
  19. udev vs Nvidia's graphics drivers
  20. Author and credits

About this document

udev is targetted at Linux kernels 2.6 and beyond to provide a userspace solution for a dynamic /dev directory, with persistent device naming. The previous /dev implementation, devfs, is now deprecated, and udev is seen as the successor. udev vs devfs is a sensitive area of conversation - you should read this document before making comparisons.

udev is a well thought out solution, but I was initially very confused how I might customise it for my system. This document attempts to make the process of rule writing a little bit clearer.

I'm all open to feedback - please contact me with any comments, problems, and suggested improvements.

This document assumes that you have udev/hotplug installed and running OK with default configurations. If you do not yet have udev configured and running, I would suggest that you follow Decibels udev Primer to get to this stage (contains some Gentoo Linux specifics, but should be useful for other distro's too).

History

June 20th 2004 v0.55: Added info on multiple symlinks, and some minor changes/updates.

April 26th 2004 v0.54: Added some Debian info. Minor corrections. Re-reverted information about what to call your rule file. Added info about naming network interfaces.

April 15th 2004 v0.53: Minor corrections. Added info about NAME{all_partitions}. Added info about other udevinfo tricks.

April 14th 2004 v0.52: Reverted to suggesting using "udev.rules" until the udev defaults allow for other files. Minor work.

April 6th 2004 v0.51: I now write suggest users to use their own "local.rules" file rather than prepending "udev.rules".

April 3rd 2004 v0.5: Minor cleanups and preparations for possible inclusion in the udev distribution.

March 20th 2004 v0.4: General improvements, clarifications, and cleanups. Added more information about writing rules for usb-storage.

February 23rd 2004 v0.3: Rewrote some parts to emphasise how sysfs naming works, and how it can be matched. Updated rule-writing parts to represent udev 018s new SYSFS{filename} naming scheme. Improved sectioning, and clarified many points. Added info about KDE.

February 18th 2004 v0.2: Fixed a small omission in an example. Updated section on identifying mass-storage devices. Updated section on nvidia.

February 15th 2004 v0.1: Initial publication.

Software versions used at time of writing

Linux Kernel 2.6.7
udev 027
hotplug 20040401

Terminology: devfs, sysfs, nodes, etc.

A basic introduction only, might not be totally accurate.

On typical linux-based systems, the /dev directory is used to store file-like device nodes which refer to certain devices in the system. Each node points to a part of the system (a device), which might or might not exist. Userspace applications can use these device nodes to interface with the systems hardware, for example, XFree86 will "listen to" /dev/input/mice so that it can relate the users mouse movements to moving the visual mouse pointer.

The original /dev directories were just populated with every device that might possibly appear in the system. /dev directories were typically very large because of this. devfs came along to provide a more managable approach (noticably, it only populated /dev with hardware that is plugged into the system), as well as some other functionality, but the system proved to have problems which could not be easily fixed.

udev is the "new" way of managing /dev directories, designed to clear up some issues with previous /dev implementations, and provide a robust path forward. In order to create and name /dev device nodes corresponding to devices that are present in the system, udev relies on matching information provided by sysfs with rules provided by the user. This documentation aims to detail the process of rule-writing, one of the only udev-related tasks that must (optionally) be performed by the user.

sysfs is a new filesystem to the 2.6 kernels. It is managed by the kernel, and exports basic information about the devices currently plugged into your system. udev can use this information to create device nodes corresponding to your hardware. sysfs is mounted at /sys and is browsable. You may wish to investigate some of the files stored there before getting to grips with udev. Throughout this document, I will use the terms /sys and SYSFS interchangeably.


Why?

As stated above, writing rules for udev is an optional process. By default, you can plug a device in, and the a relevant node (e.g. /dev/sda for a mass-storage device) will be there, just like in previous /dev implementations.

However, udev allows you to customise the naming of device nodes. There are two reasons why you might want to do this: convenience, and persistent naming.

Take the example of using udev, so that when your printer is plugged in, it gets named as /dev/printer and also as the usual /dev/lp0. It's not only convenience (e.g. reading and interpreting "printer" as opposed to "lp0"), its a solution for non-persistent naming. Say that I have two printers - a HP laser printer and an Epson inkjet. When they are both plugged in and on, I have /dev/lp0 and /dev/lp1.
How do I know which node refers to which printer? There is no easy way. The first printer that got connected was assigned name "lp0", and the second "lp1". Plugging in my printers in a different order would swap the names here, and that would mess up my scripts that always expect my HP laser printer to be lp1.

However, if my HP laser printer got named lp_hp (as well as lpX) and my other printer got named lp_epson (as well as lpY), then my scripts could just refer to those names. udev magic can control this and ensure that these persistent names always point to the device that I intended.

For external mass-storage devices (e.g. usb hard disks), persistent naming is very helpful in that it allows you to hardcode accurate device paths into your /etc/fstab.

The basics of writing rules

When populating /dev, udev decides which nodes to include, and how to name them, by reading a series of rules files.

Default udev rules are stored in /etc/udev/rules.d/50-udev.rules. You may find it interesting to look over this file - it includes a few examples, and then some default rules proving a devfs-style /dev layout. However, you should not write rules into this file directly, to reduce hassle while updating your udev installation in the future.

Files in /etc/udev/rules.d/ are parsed in lexical order. udev will stop processing rules as soon as it finds a matching rule in a file for the new item of hardware that has been detected. It is important that your own rules get processed before the udev defaults, otherwise your own naming schemes will not take effect! I suggest that you keep your own rules in a file at /etc/udev/rules.d/10-local.rules (this doesn't exist by default - create it). As 10 comes before 50, you know that your rules will be looked at first.

As your own rules will effectively mask out the udev defaults which create the base /dev layout, it is recommended that you also specify devfs-style names/symlinks for the rules you write, so that you get the sensible defaults plus your own names.

In rule files, lines starting with a "#" are treated as comments. Every uncommented line in the file corresponds to a rule.

The basic form for a rule is: 
key,[key,...] name [, symlink]
  1. At least one key must be specified. Keys are used to identify which devices the rule matches.
  2. The name parameter is required. It tells udev what that device should be named as in the /dev tree. It is written in the format NAME="X", where X is what the node will be named. You can specify multiple symlinks here, seperate them with a space.
  3. The symlink parameter (optional) allows for you to specify additional places where this node will be linked.
Remember that udev will only create one node for one device. If you want it to be accessible through multiple nodes, then you have to specify the other nodes in the SYMLINK parameter.

I'll take a slightly modified udev example rule to illustrate this: 
BUS="usb", SYSFS{serial}="HXOLL0012202323480", NAME="lp_epson", SYMLINK="printers/epson_stylus"
The keys here are the BUS and SYSFS{serial} parameters. udev will match this rule against a device that is connected through the USB bus and with a serial number of HXOLL0012202323480. Note that all (as opposed to any) specified keys must be matched for udev to use the rule to name a device.
udev will name this node lp_epson, and it will be located at /dev/lp_epson.
udev will also create a symlink to /dev/lp_epson, located at /dev/printers/epson_stylus (the printers directory will be automatically created). You can now print to your Epson printer by sending data to /dev/printers/epson_stylus or /dev/lp_epson.

Additional automated customisation for NAME and SYMLINK parameters

In the NAME and SYMLINK parameters of your rules, you are able to use basic operators to assist the naming of devices. Hackers will know this sort of thing as printf-like string substitution. There are a number of operators which can compose some or all of your NAME/SYMLINK parameters. These operators refer to kernel-data relating to the device. Take this example: 
BUS="usb", SYSFS{vendor}="FUJIFILM", SYSFS{model}="M100", NAME="camera%n"
The %n operator will be replaced with the "kernel number" for the camera device, to produce a NAME such as camera0, camera1, etc.

Another common operator is %k. This represents what the kernel would name the device, e.g. "hda1". You may often see rules which have NAME="%k" to produce the default names for the hardware. In these rules, customisation is usually done through the SYMLINK parameter.

A full list of operators, with explanations, can be found in the udev man page.

Using shell-style pattern matching in keys

You can use shell style pattern matching to provide even more flexibility when writing keys. Taking a default udev rule: 
KERNEL="ts*", NAME="input/%k"
The * operator is used here, which matches literally anything - zero, one, or more characters of any kind. The rule literally says:
Match a device identified by a KERNEL name starting with the letters "ts" optionally followed by anything at all, and name it with the KERNEL name (%k) under the input directory.
The ? operator is similar, and matches any single character (but not zero characters).

You can also use square brackets [ ] to match any single character. Direct quote from udev man page:
For example, the pattern string "tty[SR]" would match either "ttyS" or "ttyR".
You can also specify ranges that can be matched, e.g. [0-9] would match any single digit. Using an example rule from a default udev installation: 
KERNEL="fd[0-9]*", NAME="floppy/%n"
This rule says:
Match a device identified by a KERNEL name starting with the letters "fd", followed by any single digit, optionally followed by anything at all. Name the device with the kernel number of the device (%n) under the floppy directory.
You can use these wildcards/pattern matches in any type of key, including both basic keys and sysfs-based identification (see below for explanations of these key types).

I have purposely left out some information on this topic (particularly the flexibility of using [ ] operators) that is out of the scope of basic rule-writing documentation. More information on this topic can be found in the udev man page.

Key-writing basics

udev provides a few basic key matching methods, and also provides flexible ways of matching information in SYSFS. A typical rule will match both normal keys (e.g. BUS and KERNEL), as well as SYSFS keys to differentiate between different hardware plugged in throught the same port.

You may be wondering, "How do I find the serial number of my printer? What is the model of my camera?". Rule writing isn't as hard as it sounds. The trickiest bit is finding your device in /sys, and deciding which info to use.

Identifying devices through basic keys

See the udev man page for more info on these keys.

The valid keys are: The ID and PLACE keys do have their uses, but they are not commonly used in rules. This document focuses on using BUS and KERNEL keys, as well as SYSFS{...} keys (detailed in the next section). I will show how to use these keys by example.

For extra flexibility, udev also provides keys to call external scripts and examine their result. This is out of scope of this document. Look at the udev man page for more details.

Identifying devices through SYSFS files

Background information: SYSFS stores many small files under a tree of directories which provide information about your hardware. One file typically contains just one "data item" - e.g. device name, manufacturer, or product ID.

Note that SYSFS{...} keys can be combined with the basic keys described in the previous section.

You can use keys in the format SYSFS{filename} to match specific info from SYSFS, where filename corresponds to a file in your SYSFS tree. For example, when my camera is connected, there is a file located at /sys/block/sda/device/model which contains "USB 2.0M DSC". To match this, I could use the following key: SYSFS{model} = "USB 2.0M DSC"

Note that any file in sysfs can be matched in this manner, but if you match more than one file (through multiple keys), then you must only match files that exist in the same directory. Typically, there will be several directories giving information about one device. You cannot mix and match (as shown by example below).

Luckily, the process of rule writing does not entail hunting through millions of files in SYSFS, the udevinfo utility does the hard work. This program is included in the udev distribution.

The first thing you need to do is find a directory somewhere in /sys that corresponds to your hardware, and includes a file named "dev", as udevinfo can only work on directories of this type. These directories are all found under either /sys/block or /sys/class - there is no point looking anywhere else! However, udevinfo will follow links through this directory and read info found from other sections of sysfs.

Once you have found a directory of this type, you can use the following command to assist you in the creation of writing keys for udev rules: 
# udevinfo -a -p /sys/path/to/hardware/info
You may find that finding the correct place in /sys to run udevinfo on is not obvious. Chances are the device you just plugged in has already careted a device node (e.g. /dev/sda), in which case, udevinfo can be helpful! Taking the example of my /dev/sda node, running the following command will point you to the appropriate area of sysfs: 
# udevinfo -q path -n /dev/sda

/block/sda
The output of the command (shown above) is telling me that the sysfs path to start at is /sys/block/sda. I would now run "udevinfo -a -p /sys/block/sda". These two commands can be stringed together, like so: 
# udevinfo -a -p `udevinfo -q path -n /dev/sda`
Moving on to rule-writing, some snipped output of the results of my "udevinfo -a -p /sys/block/sda" command is shown below, with colour added.

follow the class device's "device"
  looking at the device chain at '/sys/devices/pci0000:00/0000:00:02.1/usb3/3-3/3-3:1.0/host0/0:0:0:0':
    BUS="scsi"
    ID="0:0:0:0"
    SYSFS{detach_state}="0"
    SYSFS{type}="0"
    SYSFS{max_sectors}="240"
    SYSFS{device_blocked}="0"
    SYSFS{queue_depth}="1"
    SYSFS{scsi_level}="3"
    SYSFS{vendor}="        "
    SYSFS{model}="USB 2.0M DSC    "
    SYSFS{rev}="1.00"
    SYSFS{online}="1"

  looking at the device chain at '/sys/devices/pci0000:00/0000:00:02.1/usb3/3-3':
    BUS="usb"
    ID="3-3"
    SYSFS{detach_state}="0"
    SYSFS{bNumInterfaces}=" 1"
    SYSFS{bConfigurationValue}="1"
    SYSFS{bmAttributes}="c0"
    SYSFS{bMaxPower}="  0mA"
    SYSFS{idVendor}="052b"
    SYSFS{idProduct}="1514"
    SYSFS{bcdDevice}="0100"
    SYSFS{bDeviceClass}="00"
    SYSFS{bDeviceSubClass}="00"
    SYSFS{bDeviceProtocol}="00"
    SYSFS{bNumConfigurations}="1"
    SYSFS{speed}="12"
    SYSFS{manufacturer}="Tekom Technologies, Inc"
    SYSFS{product}="USB 2.0M DSC"
The udevinfo tool provides a lot of information which you can simply copy-paste as udev rules. The reason that I have colour coded the above output is to point out that you generally cannot mix and match information from different parts of the udevinfo output. In the above output, I could not combine information from the different coloured sections - this is because each section of output refers to a different directory in SYSFS. For example, the following rule would not work: 
BUS="scsi", SYSFS{manufacturer}="Tekom Technologies, Inc", NAME="%k"
This rule would not work because I am combining information found in the section beginning with BUS="scsi" (green) with information only found in the blue section. The rule would work if I used BUS="usb", sticking only to information found in the blue section above.

You will notice that a lot of information is not relevant for writing basic rules (there is so much of it!), you should generally be looking for information that you recognise and know will not change (e.g. model name).

Note that if you write your own rule to identify a device, the default devfs-style rules will not take effect! It is usually sensible to use NAME="%k" and specify your own extra names in the SYMLINK parameter so that you do not lose the default sensible names.

I will show three examples of this rule writing based on udevinfo output process below. I will then attempt to list some device-dependant tips and tricks for locating the correct info.

A reader wrote to me and informed me that he found KDE's control centre useful for writing rules. Apparently, information about USB devices (and others) can be found in the "Info Centre" section of the KDE Control Centre. This interface shows information such as serial number, vendor ID, etc. If you prefer a GUI-like approach, you might want to investigate this.

The current releases of gnome-volume-manager are unable to treat symlink-nodes as real devices. Conversely as described above, you may wish to specify your own naming in the NAME parameter and specify %k in the SYMLINK parameter.

The behaviour of your own rules masking the defaults can be overcome if you write multiple-SYMLINK style rules.

Example: Writing a rule for my USB printer

After plugging in my printer, I started looking around some /sys directories for a relevant place to start. I didn't get anywhere, but I noticed that my printer had been given device node /dev/lp0. udevinfo was able to provide me with a useful path: 
# udevinfo -q path -n /dev/lp0
/class/usb/lp0
Running "udevinfo -a -p /sys/class/usb/lp0" provided me with a heap of info, as usual. I picked out the relevant bits for unique device identification: 
looking at the device chain at '/sys/devices/pci0000:00/0000:00:02.1/usb3/3-3':
BUS="usb"
SYSFS{manufacturer}="EPSON"
SYSFS{product}="USB Printer"
SYSFS{serial}="L72010011070626380"
My udev rule becomes: 
BUS="usb", SYSFS{serial}="L72010011070626380", NAME="%k", SYMLINK="epson_680"
And my printer nodes exist at /dev/lp0 (or /dev/lp1 if another printer was plugged in beforehand) and /dev/epson_680 always points at the device node for that particular printer.

Example: Writing a rule for my USB-Storage digital camera

Quick Intro: My camera identifies itself as an external SCSI hard disk (it uses the usb-storage driver which is also used by devices such as USB hard disks and flash-card readers). I can then mount the partition on that disk and copy images over. Not all cameras work like this - many require external software (e.g. gphoto2) to be able to access photos.

This one is a bit tricky. Several nodes are created by default when my camera is connected : /dev/sda and /dev/sda1, and possibly even /dev/sg1. This is an example where specifity is important - if your rule is not specific enough, it could match any of the above 3 nodes.

sda1 is the node that I would like as my /dev/camera, as that is what gets mounted. udevinfo did not point out any useful differences between sda, sda1, and sg1. I decided that a reliable way to differentiate between these 3 nodes would be to look at the KERNEL name.

A key such as KERNEL="sd?1" would match KERNEL names such as "sda1", "sdb1", "sdc1", and equally importantly, it will not match KERNEL names such as sda, sdb, or sg1. The purpose of this key is to ignore the /dev/sda and /dev/sg1 nodes. The device is a digital camera - I would not dream of fdisking it or anything like that, so these 2 nodes are pretty useless to me. The key attempts to capture the /dev/sda1 node, which is mountable and therefore useful!

As this node (sda1) is treated as a block device, looking in /sys/block would be a good place to start.

In my /sys/block, I have a directory named sda. In my /sys/block/sda, I have a directory named sda1. Both of these directories have dev files in, so they are OK to run udevinfo on. Running the following dumps a lot of information about my camera and the USB port it is connected through. 
# udevinfo -a -p /sys/block/sda/sda1
In the udevinfo output, I also noticed this bit of useful and understandable information: 
SYSFS{product}="USB 2.0M DSC"
So that gives me my rule. For completeness, I also include a BUS key (this was also found in the udevinfo output). 
BUS="usb", SYSFS{product}="USB 2.0M DSC", KERNEL="sd?1", NAME="%k", SYMLINK="camera"
Now, when my camera is plugged in, it will be named /dev/sda1 (or, if sda1 isnt available, it might be called /dev/sdb1) and will always be correctly linked to from /dev/camera. The /dev/sda (or sdb) node still appears as normal, but the important thing is that my custom persistent "camera" symlink points to the mountable partition.

Additional notes on writing rules for USB storage

Carl Streeter, the owner of a large USB hard disk, wrote to me and explained that unlike in my digital camera example, the /dev/sda node is useful to him. He pointed out that he does occasionally need to use tools such as fdisk and hdparm on that node.

Carl's rule is: 
BUS="usb", KERNEL="sd*", SYSFS{product}="USB 2.0 Storage Device", NAME="%k", SYMLINK="usbhd%n"
This rule creates symlinks such as: We agreed that depending on the situation and device in question, there are reasons for both wanting and not wanting the non-mountable /dev/sda node. Use whichever setup suits you best.

Another difficult situation is having a multiple-slot USB-storage card reader. These types of device generally do not inform the host when new cards are plugged in or out, so plugging a card into an unused slot while the reader is plugged in will not create the extra device node needed for mounting!
This problem also applies to other USB disks - e.g. if you create a new partition, the new partition node will not appear until you re-plug the device.

udev provides a solution here - it is able to create nodes for all partitions of a block device. For every rule that you specify, the block device will have all 16 partition nodes created. To achieve this, you can simply modify the NAME key, as shown below:
BUS="usb", SYSFS{product}="USB 2.0 Storage Device", NAME{all_partitions}="usbhd"
You will now have nodes named: usbhd, usbhd1, usbhd2, usbhd3, ..., usbhd15.

Example: Writing convenience rules for my CD drives

I have two CD drives in my PC - a DVD reader, and a CD rewriter. My DVD is hdc and my CDRW is hdd. I would not expect this to change, unless I manually changed the cabling of my system.

Still, some people (myself included) like to have nodes such as /dev/dvd and /dev/cdrw for convenience. Since we know the "hdX" values for these drives, writing rules is simple. The examples below should be self explanatory. 
BUS="ide", KERNEL="hdc", NAME="%k", SYMLINK="dvd cdroms/cdrom%n"
BUS="ide", KERNEL="hdd", NAME="%k", SYMLINK="cdrw cdroms/cdrom%n"
You may have noticed that the default 50-udev.rules file contains a rule which runs a script to produces names for block devices. Do not be confused by this - as usual, because your own rules are located in a file which is processed before the default rules, the defaults will not be used when naming the hardware you have written rules for.

Example: Writing a rule to name my network interface

An interesting new feature in recent udev versions is the ability to write rules to name your network interfaces. Network interfaces do not show up in /dev, but they are generally referenced by names (e.g. with ifconfig). Despite the differences, the rule writing process is almost identical.

As usual, udevinfo comes to our aid in rule-writing. In my example, I wish to rename my "eth0" network device (the following output is snipped): 
# udevinfo -a -p /sys/class/net/eth0/
  looking at class device '/sys/class/net/eth0':
    SYSFS{address}="00:52:8b:d5:04:48"
Every network adapter has its own unique MAC-address, so I chose to use this when writing my rule. This will not change, unless you change your network card. An example rule is shown below: 
KERNEL="eth*", SYSFS{address}="00:52:8b:d5:04:48", NAME="lan"
You will need to reload the net driver for this rule to take effect. You can either unload and reload the module, or simply reboot the system. You will also need to reconfigure your system to use "lan" rather than "eth0". I had some troubles getting this going (the interface wasn't being renamed) until I had completely dropped all references to eth0.
After that, you should be able to use "lan" instead of "eth0" in any calls to ifconfig or similar utilities.

Tips for finding the appropriate places in SYSFS

I'm looking for some more device specific tips here. Please contact me with any you can provide.

Using multiple SYMLINK style rules

Another recent feature is the ability to write rules that do not specify a NAME, but instead they simply specify SYMLINK keys. This allows you to avoid the issue where your own rules effectively mask the udev defaults.

Take the rule:
KERNEL="hdc", SYMLINK="dvd"
When udev finds this rule, it will take a mental note of it. Upon finding another rule matching the same device which also includes a NAME parameter, udev will create the node as specified by the NAME parameter, plus symbolic links as specified by the SYMLINK parameters of both rules.
To put it into practical terms, when udev is naming nodes for my hdc device, it will use the default rules for block devices as usual, with the addition of my personal symlink "dvd".

Similarly to normal rules, rules of this type will only take effect if udev is able to find them before it finds a rule specifying a NAME parameter.

udev vs Nvidia's graphics drivers

This section isn't really relevant to the purpose of this document, but judging from the hits I get from google, this is a hot topic. I will leave it here for now.

Nvidia's graphics drivers (the closed-source ones, not the ones that come with XFree) do not work with a default installation of udev - you are unable to start X. This is because the nvidia module is loaded by X, but the /dev/nvidia* nodes are not created quick enough, so X bails out.

The solution to this problem is to autoload the nvidia module on bootup. Yes - you are *supposed* to do this - the NVidia FAQ confirms this! On devfs-based systems, devfs did this automatically at bootup anyway. Your linux distribution will have created a file which you can list modules to be loaded on bootup (e.g. /etc/modules.autoload.d/kernel-2.6 for Gentoo, /etc/modules for Debian).

This isn't all - you will also need to patch the nvidia kernel interface to export some basic info to SYSFS so that udev will create the devices. Martin Schlemmer has written a patch against the 1.0.5336 version of the nvidia drivers, which can be found here. The Gentoo package nvidia-kernel-1.0.5336-r4 contains this patch.

Another solution is to simply create the nvidia specific nodes on bootup. X will then load the module when required, and as the nodes are already in existance, you will not run into the problem described above. Place these commands in a file that is automatically executed on bootup (e.g. /etc/conf.d/local.start for Gentoo): 
mknod /dev/nvidia0 c 195 0
mknod /dev/nvidiactl c 195 255
You should now be able to get into X with no problems.

Author and Credits

This document is written by Daniel Drake <dan@reactivated.net>
Please do not hesitate to send feedback!

Additional thanks to:

Copyright (C) 2003-2004 Daniel Drake
This document is licensed under the GNU General Public License, Version 2.