chronicle-gdbserver

What is it?

chronicle-gdbserver is an implementation of gdb's remote protocol which talks to a database written by chronicle-recorder. What this means is that you can run the program you want to debug under chronicle, which will write a complete database of everything it does. Then you can use chronicle-gdbserver to link gdb with that database, and use all gdb's standard facilities to examine what the program did during its instrumented run. In particular, recent gdb has support for reverse debugging -- so you can step or run *backwards* through the program. Particularly handy is the ability to use watchpoints in conjunction with running backwards to find out where a particular corrupted variable or memory location was originally written.

How stable is it?

Still rather experimental. It seems to work and I think I've implemented the bits of the protocol gdb needs, but there may still be some corner cases which don't work. Also some things seem to cause gdb to ask the stub to do enormous amounts of work reading memory and thus gdb appears to hang (ctrl-c often leaves things in a broken state). I haven't really tested it with any seriously sized piece of code either, so it might have some nasty performance issues.

What are the dependencies?

You'll need at least:

plus the bits and pieces whose installation instructions are given below.

How do I get it?

This is a bit complicated because chronicle-gdbserver is really just a small bit of code which glues together much larger bits of code written by other (and generally smarter) people than me.

These instructions assume we're going to install everything into $HOME/chronicle/. I don't recommend allowing things to install into /usr/local/ because this means your system versions of things like gdb and valgrind are liable to be preempted by development or elderly binaries. So under this scheme $HOME/chronicle/bin will have various binaries used internally; a separate $HOME/chronicle/publicbin will contain only wrapper scripts. The latter is safe to put on your PATH.

1: Get and build chronicle-recorder

I used to recommend getting upstream chronicle here. However since that is based on valgrind 3.3 it won't build with recent versions of glibc. Also it has a couple of bugs that need manually patching. So instead I suggest you download the version of chronicle which I forward-ported to valgrind 3.5:

I'm afraid you'll need a fairly recent version of bzr for this repository. The build notes for chronicle-recorder might be worth looking at if this short version is too brief for you.

(If you still want to use 3.3 here's how to do it:

...end of remarks about 3.3.)

2. Get chroniquery

chroniquery is a python binding for the chronicle query agent and some useful utilities. Get the latest code from bzr like this: 

 bzr branch http://www.visophyte.org/rev_control/bzr/chroniquery/trunk/ chroniquery
 mkdir ~/chronicle/python
 cp chroniquery/python/*.py ~/chronicle/python/

3. Get distorm

distorm is an x86 disassembler with some python bindings. Chroniquery uses it, but in fact chronicle-gdbserver doesn't need it. So you can probably skip this step; I document it just in case (and in case you want to play with chroniquery).

Download and unpack the tarball. Make sure you have the 'python-dev' or equivalent package installed. Build distorm with: 

  cd distorm64/build/linux && make

Don't trust 'instpython.sh', it is barking. Instead just copy the resulting libdistorm64.so into ~/chronicle/python/

4. Get gdb

You need a gdb which is new enough to support reverse step/continue. (Technically it should all work with an older gdb, but if you don't have the ability to go backwards then you're missing 90% of the point.) This means gdb 7.0, recently released. Get it from the GNU website Unpack and compile in the usual way: 

  tar xvjf gdb-7.0.tar.bz2
  mkdir build
  cd build
  ../gdb-7.0/configure --prefix=$HOME/chronicle
  make
  make install

5. Get chronicle-gdbserver

Grab the current sources from bzr: 

bzr branch http://www.chiark.greenend.org.uk/~pmaydell/bzr/chronicle-gdbserver
If you do a make install in the resulting chronicle-gdbserver it will copy its python script into ~/chronicle/python/ and also copy some wrapper scripts into ~/chronicle/publicbin. You should put the latter directory onto your PATH.

Once you've done that, if you have expect installed then you can try running the test suite with make test.

How do I use it?

Put ~/chronicle/publicbin on your PATH if it's not there already (or give absolute paths to chronicle and chronicle-gdbserver).

First you need to run the program under test using chronicle. Chroniquery comes with a couple of handy test programs so we'll use one of those as our example: 

 g++ -g -Wall -o fancy fancy.cc
 chronicle ./fancy
When the program exits you should have a fancy.db chronicle database file. (If you want to override the default name for the database you can set the environment variable CHRONICLE_DB when running chronicle.)

The easiest way to run chronicle-gdbserver is via the chronicle-gdb wrapper script: 

 chronicle-gdb ./fancy
This will start chronicle-gdbserver, and also run gdb and tell it to connect to that chronicle-gdbserver. If all has gone well then gdb should tell you it's sat at main(). When gdb exits then chronicle-gdbserver will also exit.

gdb works as you'd expect. The manual it should have installed into ~/chronicle/info/ has more detail about the reverse-step commands, but a quick summary is that 'rs' steps backwards and 'rc' continues backwards. Watchpoints and breakpoints are set in the usual way and will be hit in either direction.

The other wrapper script is chronicle-gdbserver, which starts the server standalone. You can then connect to it with gdb 'by hand'. This is mostly useful if you want to turn on the exceedingly verbose logging: 

 chronicle-gdbserver -X fancy
Leave that running, and start gdb in another window: 
 ~/chronicle/bin/gdb

At gdb's command prompt type: 

 file fancy
 target remote localhost:8181

Known issues

This is my TODO list:

Acknowledgements

This is definitely a case of 'standing on the shoulders of giants'.

Feedback

Feel free to email pmaydell@chiark.greenend.org.uk with comments or bug reports. In particular, chronicle-gdbserver is my first Python script so it's probably full of hopelessly unidiomatic code which I'd be interested to have pointed out to me.