This page describes a collection of Unix tools for filtering the input and output of a shell session. Written by Simon Tatham.
This is one of my unfinished, and not very polished, pieces
of software. Documentation is sparse; the collection of tools is quite
eclectic and random and not all of them will be useful to anyone else;
too many of the tools have too-generic names which wouldn't go well in
the /usr/bin of a serious distro. At the moment I'm
mostly considering it to
be symbiosisware. But I put
it up on the web anyway in case it's useful to anyone: either the
specific filtering tools I've already written, or the framework that
makes it easy (ish) to write extra ones.
There already exist Unix tools that run within a terminal session, and
wrap a program in a second pseudo-terminal. Some examples written by
other people include the standard script command (records
your session output to a log file); ttyrec (the same, but
with timestamps, so you can replay it in real time using a tool such
as IPBT); and luit (translates
character sets, so that you can run a program expecting one character
set in a terminal that speaks another).
This is my own personal collection of pty filtering tools of this kind. I wrote the general framework around 2002, making it easy to replace just the part that processes the input and/or output. I've been adding tools to the collection ever since, whenever I find a thing that would be useful.
Some of them replicate the functionality of other people's tools. All the tools I mention above have equivalents in this collection, because I very slightly prefer my versions for my own use (and because once I had the framework it doesn't need a lot of effort to write another tool). That's one of the ways in which this collection is symbiosisware. But others are more unusual.
All these tools have a set of common features. By default each one
makes a pseudo-terminal and runs a shell in it. You can use it to wrap
a command other than a shell by writing -- on the command
line followed by the command you want to run; there are common options
to swap out the pty for a pair of pipes, or even
an AF_UNIX socketpair.
Some tools have additional options of their own to control their filtering behaviour, but all of them support these standard ones.
The very simplest tool of all: don't do anything to the input or the output, just pass it all through unmodified.
This sounds pointless, but I find it useful surprisingly often! Some example uses:
nullfilter's internal pty which is going to be
thrown away anyway, instead of the main one in your session which
you wanted to keep.less loses all
the colour, even if you used less -R which would have
been able to cope. But running it inside nullfilter
and then piping to less -R gets the colour back.nullfilter gives the shell a more normal terminal.)
Almost as simple: throw away all terminal input. I've used this when giving demos of a terminal-based application to an in-person audience, to avoid the audience accidentally (or on purpose) interrupting the demo by prodding the keyboard.
(Of course, that meant I couldn't interact with the demo myself either. But that's OK, because it was a ttyrec recorded ahead of time, and played on loop by IPBT.)
Deletes any terminal output that ends in a carriage-return character
without a following line feed. The idea is that some tools (the build
tool ninja, in particular) will print interim progress
reports by sending some text and then a CR, to move the cursor back to
the same line, so that each progress report overwrites the previous
one. Those progress reports are removed by this filter, so that you
only get ninja's "permanent" output – the stuff that persists on the
screen after it finishes running.
I mentioned above that one of my uses for nullfilter is
to run something colourful inside it and then pipe to less
-R. One common example of "something colourful" for me is a
build tool like ninja which in turn runs compile commands
that produce coloured error messages. But nullfilter -- ninja |
less -R looks ugly because of those overprinted progress
reports. Solution: use this filter instead, noprogress
-- ninja | less -R.
Intercepts SGR escape sequences in the terminal output – the ones that turn on and off terminal attributes like bold and underline, and also select the foreground and background colour. Then rewrite the set of attributes, and re-emit a different SGR sequence.
In principle, this could be used for lots of special effects: converting between bold and brighter colours, getting rid of blinking text if you find it annoying, etc. My actual aim is to use it for remapping the foreground and background colours to improve readability, especially for people with anomalous or impaired colour vision who find that the more usual tools for that (such as terminal programs that enforce a minimum contrast) aren't suited to their own vision.
As of 2025-05-10 this is new and experimental, and the only way to choose what remapping rules you want is to edit the source code and recompile. If you run it unmodified, you get a demonstration remapping which just permutes the red, green and blue channels, to prove it's doing something.
(I don't yet know if this tool will have long-term usefulness, or if it will only be useful for initial experimentation and then some useful colour remapping rules will end up being done some other way. So I haven't put in any serious effort to allow user-defined remapping rules without recompilation, like embedding a language interpreter or a DSL or having a plugin system, or even a set of built-in remapping options. I'm waiting to see what if anything might be needed.)
Note that by default this tool works by
converting all colour-setting control sequences into ones
that specify 24-bit RGB colour. So if your terminal
doesn't support 24-bit colour, then it won't work at all.
Apparently this is true for the default Mac Terminal.app
(at least, as of version 2.14 on macOS 15.4.1); iTerm works
better.
Filter all non-ASCII characters out of the terminal input. There's one particular very old application I interact with which will outright crash if it receives any input byte with the high bit set.
This tool tries to translate some characters into good ASCII
approximations. In particular, Unicode typographical quotes and dashes
are converted back into ASCII ones, so if you paste English text out
of a nicely typographed web page into your terminal, it will come out
with a sensible ASCII rendition of the same punctuation. The range of
Unicode characters handled nicely in this way is quite small, and
anything too difficult will just generate a fallback such
as <U+1234>. But that's still good enough to
fulfill the primary purpose of the tool: not crashing the
wrapped application.
Convert input sequences of the form ESC x (which PuTTY will generate if you press Alt+x, for any letter key x) into the byte value that is the character 'x' with the high bit set.
I used to use this for playing Nethack, which expects that high-bit-set representation of Alt+letter keystrokes. PuTTY doesn't have a mode built in to make Alt set the high bit, because it's so difficult to work out what should happen in all the hard cases, and I was sure users would complain about edge cases. (What if the character value has bit 7 set already? What if it's a Unicode code point above 255?) But this tool has a much more limited scope: it's only for making Nethack work properly, so I know it only needs to handle that small set of sequences.
Terminate the subprocess and exit if there's been no input or output for a specified length of time.
Prevent a session from being disconnected by idlewrapper
or anything like it, by manufacturing extra input periodically if
there hasn't been any other input. You'd probably only use this to
wrap an application for which there's a keystroke you know is always
safe to send.
Strip the high bit off all terminal output, in the expectation that it was a serial-line parity scheme that something failed to take off already before writing text to the terminal. Also, optionally, put a parity bit back on to terminal input.
I can't even remember what I wanted this for in the first place. I wrote it as recently as 2016, and I assume serial ports must have been involved somewhere, but I have no idea what I actually needed to fix in this area. The commit message doesn't remind me what application I was wrapping!
My own tool similar to both script
and ttyrec: it records a copy of the terminal output to a
file, in a format of your choice: plain text
(like script), or the ttyrec format that includes
timestamps, or the similar nh-recorder format.
I wrote my own one of these instead of just using script
and/or ttyrec for several reasons:
script
uses the -c "shell command" style. I generally find
the adverbial style useful more often.
My own tool similar to luit: you tell it what character
set the outer terminal speaks (default UTF-8) and what character set
you want the inner terminal to work in, and it does character-set
translation in both directions. So if you have an old-fashioned
program that expects some single-byte charset like ISO 8859-1 or even
CP437, this kind of thing is the easiest way to get it to work,
because you can wrap just that application instead of reconfiguring
your whole terminal.
My version of this tool uses my own character-set translation library
(a spinoff of PuTTY's), which supports a few lesser-known character
sets that I've occasionally wanted to use. Also I find it convenient
that the outer and inner character sets are specified by name on the
command line, instead of luit's approach of deducing them
from environment variables like LC_CTYPE. These aren't
very good reasons to do a huge amount of work writing a new
version of the same program from scratch – but since I already had
this filter framework and the character set library, it
wasn't a huge amount of work to glue them together, and
seemed just about worth it to me.
There's one other tool built from the same code base as all of these filters. It's different from the rest, because it's not a filter that runs a subprocess in a pty. It just reuses some of the infrastructure of the filters to do a different job.
linktty creates two pseudo-terminals, puts them
into raw mode, and copies data between them in both directions. So if
two unrelated processes each opens one of the two ptys for both
reading and writing, then they can talk to each other, as if they were
speaking via a serial line.
I use this for communicating between gdb
and gdbserver, when I need to debug a full-screen
terminal program (so that it's convenient to run the debugger in a
separate terminal from the program itself). You
tell gdbserver to use one of the ptys,
and gdb to use the other, and each of them behaves as if
it were doing serial-port debugging to a remote machine, but really
they're talking to each other.
(The usual way people do this is by
making gdbserver listen on a network port, and
telling gdb to connect to it. But this is a huge security
hazard, because the network connection has no authentication, so
anyone else can connect to the same port and then
tell gdbserver to run arbitrary code under your user id!
For debugging on the same machine you can restrict the network socket
to localhost, but that still doesn't take account of the fact that
Unix machines are multi-user. An AF_UNIX socket would be
the right way to fix this, but as far as I know gdb doesn't support
that. This security issue is known –
even documented
in the gdb manual – but that seems to be all they've done about it,
because they just say "don't run this while connected to any public
network", which rather overlooks the fact that these days pretty
much everything is connected to a public network one way or
another.)
For convenience, linktty creates a symlink to each
terminal, so that you can give each of gdb
and gdbserver a fixed pathname, even if the names of the
actual ptys differ between runs. For example, you might run commands
like these (in three separate shell sessions):
linktty /home/username/tty-for-gdb /home/username/tty-for-gdbserver gdbserver /home/username/tty-for-gdbserver program-to-debug [args] gdb -ex 'target extended-remote /home/username/tty-for-gdb'
and then you don't have to worry about the real names of the
pty devices, you just use the symlinks. The next time you run these
commands, linktty will probably get a different pair of
ptys, and none of these processes will care.
These tools have a public git repository, but no release
archives (or numbered releases at all). To download the code, run
git clone https://git.tartarus.org/simon/filter.gitAlternatively, you can browse the repository on the web.
If you want to use this setup to write a filter of your own, the
easiest thing to do is to make a copy of nulltrans.c, and
start from there. That source file contains comments describing the
API expected by the framework, and you can look at all the existing
filter programs for examples of each feature in use.
I'm not promising much maintenance of this set of tools, but if you do find a bug, you're welcome to report it to anakin@pobox.com.
You might find it helpful to read this article before reporting a bug.