--- /dev/null
+*.pyc
+cache
+output
+++ /dev/null
-Hello!
-
-<p>New year, new blog. I've had a
-<a href="http://www.livejournal.com/users/cjwatson/">LiveJournal</a> for a
-while, but don't write very much in it, and many of its readers wouldn't be
-interested in me talking about Debian and such anyway. I think the best
-solution is for me to keep technical posts here.</p>
+++ /dev/null
-Killer apps: bzr shelve
-
-<p>Working on free software has made me fairly revision control
-system-agnostic; I can't afford to get too wedded to any one system because
-as soon as I do somebody will invent something new and I'll have to convert
-again, so I just work with whatever other people on the same project are
-using. Even CVS doesn't make a lot of difference to the way I work as long
-as I'm working online and have cvsps handy. And of course I usually don't
-bother with revision control if I'm just tweaking somebody else's Debian
-source package a bit (in which case I just use debdiff for paranoia).</p>
-
-<p>Using bzr at work, though, I think I just found my killer app in Michael
-Ellerman's <a href="http://bazaar.canonical.com/BzrShelveExample">shelve</a>
-plugin. My working style generally involves alternating between doing lots
-and lots of stuff in the one working copy and (after testing) going through
-and committing it in logical chunks. This is fine if everything's in
-separate files (most revision control systems let you commit just some
-files), but if several of the chunks are in the one file then I'm reduced to
-saving diffs and manually editing out the bits I don't want to commit yet,
-which is obviously pretty tedious and error-prone.</p>
-
-<p><kbd>bzr shelve</kbd> presents each diff hunk in your working copy to you
-in turn and asks you whether you want to keep it. If you say no, that hunk
-gets unapplied and goes into a "shelf", where <kbd>bzr unshelve</kbd> can
-later reapply it. In the meantime commits act as though the shelved hunks
-didn't exist. This doesn't help if you want to defer only one of two
-immediately adjacent changes that end up in the same hunk, of course, but it
-vastly reduces the scale of the problem.</p>
-
-<p>I suppose it would be easy enough to write a shelve-a-like for any other
-system; it's just that I haven't seen it for any other system yet. If
-working with systems that lack it really starts to annoy me, I may have to
-rip out the guts of shelve and figure out how to make it generic.</p>
+++ /dev/null
-Unix tools: sponge
-
-<p>Joey
-<a href="http://kitenet.net/~joey/blog/entry/unix_tools_vidir-2006-02-05-21-33.html">writes</a>
-about the lack of new tools that fit into the Unix philosophy. My favourite
-of such things I've written is
-<a href="http://riva.ucam.org/svn/cjwatson/bin/sponge">sponge</a>. It
-addresses the problem of editing files in-place with Unix tools, namely that
-if you just redirect output to the file you're trying to edit then the
-redirection takes effect (clobbering the contents of the file) before the
-first command in the pipeline gets round to reading from the file. Switches
-like <kbd>sed -i</kbd> and <kbd>perl -i</kbd> work around this, but not
-every command you might want to use in a pipeline has such an option, and
-you can't use that approach with multiple-command pipelines anyway.</p>
-
-<p>I normally use sponge a bit like this:</p>
-
-<pre>
-sed '...' file | grep '...' | sponge file
-</pre>
-
-<p>Since it's so trivial I imagine lots of other people have written
-something similar (another common name for it seems to be inplace; my name
-indicates soaking up all the input and then squeezing it all out again); but
-I do keep meaning to try to get a rewritten version into coreutils at some
-point.</p>
+++ /dev/null
-Keysigning public service announcement
-
-<p>If your key has so many UIDs and such a combinatorially exploded number
-of signatures on it that it takes gpg minutes just to start up in --edit-key
-mode, then I probably won't bother signing it. HTH, HAND.</p>
+++ /dev/null
-Encodings in man-db
-
-<p>I've spent some quality upstream time lately with man-db. Specifically,
-I've been upgrading its locale support. I recently published a pre-release,
-<a href="http://people.debian.org/~cjwatson/man-db/man-db-2.5.0-pre2.tar.gz">
-man-db 2.5.0-pre2</a>, mainly for translators, but other people may be
-interested in having a look at it as well. I hope to release 2.5.0 quite
-soon so that all of this can land in Debian.</p>
-
-<p>Firstly, man-db now supports creating and using databases for per-locale
-hierarchies of manual pages, not just English. This means that
-<a href="http://bugs.debian.org/29448">apropos and whatis can now display
-information about localised manual pages</a>.</p>
-
-<p>Secondly, I've been working on the transition to UTF-8 manual pages. Now,
-modulo some hacks, groff can't yet deal with Unicode input; some possible
-input characters are reserved for its internal use which makes full 32-bit
-input difficult to do properly until that's fixed. However, with a few
-exceptions, manual pages generally just need the subset of Unicode that
-corresponds to their language's usual legacy character set, so for now it's
-good enough to just recode on the fly from UTF-8 to some appropriate 8-bit
-character set and use groff's support for that.</p>
-
-<p>man-db has actually supported doing this kind of thing for a while, but
-it's been difficult to use since it only applies to
-<code>/usr/share/man/ll_CC.UTF-8/</code> directories, while manual pages
-usually aren't country-specific. So, man-db 2.5.0 supports using
-<code>/usr/share/man/ll.UTF-8/</code> instead, which is a bit more
-appropriate. Also, following a
-<a href="http://lists.debian.org/debian-mentors/2007/09/msg00245.html">
-discussion with Adam Borowski</a>, man-db can now try decoding manual pages
-as UTF-8 and fall back to 8-bit encodings even in directories without an
-explicit encoding tag; if this fails for some reason, you can put a
-<kbd>'\" -*- coding: UTF-8 -*-</kbd> line at the top of the page.</p>
-
-<p>I'm still debating whether Debian policy should recommend installing
-UTF-8 manual pages in <code>/usr/share/man/ll.UTF-8/</code> or just in
-<code>/usr/share/man/ll/</code>. Initially I was very strongly in favour of
-an encoding declaration, but now that man-db can do a pretty good job of
-guesswork I'm coming round to Adam Borowski's position that people should be
-able to forget about character sets with UTF-8. Opinions here would be
-welcome. One thing I haven't moved on is that any design that assumes that
-the encoding of manual pages on the filesystem has anything to do with the
-user's locale is demonstrably incorrect and broken; I'm not going to use
-<code>LC_CTYPE</code> for anything except output. However, maybe "UTF-8 or
-the usual legacy encoding provided that the latter is not typically confused
-for the former" is a good enough specification, and that still has the
-desirable property of not requiring a flag day. I'll try to come down from
-the fence before unleashing this code on the world.</p>
+++ /dev/null
-UTF-8 manual pages
-
-<p>See
-<a href="http://www.chiark.greenend.org.uk/ucgi/~cjwatson/blosxom/2007-09-17-man-db-encodings.html">
-Encodings in man-db</a> for context.</p>
-
-<p>Yesterday, I uploaded
-<a href="http://lists.debian.org/debian-devel-changes/2008/01/msg02665.html">
-man-db 2.5.1-1</a> to unstable. With this version, not only is it possible
-to install manual pages in UTF-8 (as with 2.5.0, although with fewer bugs),
-but it's also possible to ask man to produce a version of an arbitrary page
-in the encoding of your choice, and have it guess the source encoding for
-you fairly reliably. This finally provides enough support to have debhelper
-<a href="http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=462937">
-automatically recode manual pages to UTF-8</a>.</p>
-
-<p>It'll probably take a little while to shake out the corner-case bugs, but
-I'm generally pretty happy with this. Once the new man-db and debhelper land
-in testing, I'll send a note to debian-devel-announce and push harder on my
-<a href="http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=440420">policy
-amendment</a>.</p>
-
-<p>Considering the historical state of man-db when it comes to localisation,
-and all of the dependencies and general yak-shaving that had to be tackled
-to get here, this represents the end of probably several hundred hours of
-work, so I'm pretty happy that this is out the door. The only remaining step
-is to add UTF-8 input support to groff, which fortunately Brian M. Carlson
-is
-<a href="http://lists.gnu.org/archive/html/groff/2007-11/msg00018.html">working</a>
-<a href="http://lists.gnu.org/archive/html/groff/2008-01/msg00004.html">on</a>.
-After that, we can reasonably claim to have dragged manual pages kicking and
-screaming into the 21st century.</p>
+++ /dev/null
-Don't use sshkeygen.com to generate keys!
-
-<p>To my horror, I recently saw <a href="http://www.sshkeygen.com/">this
-online SSH key generator</a>.</p>
-
-<p>I hope nobody reading this needs to be told why this is a bad idea.
-However, in case you do, here are a few reasons:</p>
-
-<ul>
- <li>
- Every SSH implementation I know of - certainly all the major ones - that
- support public key authentication also provide a key generation utility.
- Even aside from all the good reasons not to, there is simply no reason why
- you should need to use a web-based tool in the first place.
- </li>
- <li>
- How can you trust the person running this site? Without implying that I
- know he or she is untrustworthy (I don't), and with the best will in the
- world, it's a big Internet with a lot of nasty people on it. Do you really want somebody you don't know in a position to keep a copy of all your
- private keys?
- </li>
- <li>
- Even if the person is trustworthy, the server running sshkeygen.com is now
- a giant blinking target. If lots of people use it, there is every
- incentive in the world for the bad guys to try to take control of it so
- that they can keep a copy of all your private keys. (Or, as we know from
- recent bitter experience, they can just give out keys from a limited set
- and it will probably take a couple of years before anyone notices ...)
- </li>
- <li>
- The front page of sshkeygen.com says that the keys are escrowed. The plain
- English meaning of this would be that the operator of that site keeps a
- copy of the private key, to be held in trust in case (presumably) you lose
- it and need to retrieve it. Normally this sort of thing depends on a legal
- trust relationship, perhaps linked to a contract. What does it mean here?
- Is it just a buzzword? If it isn't, then this just makes sshkeygen.com
- even more of a target.
- </li>
- <li>
- sshkeygen.com delivers keys to you over unencrypted HTTP. Yes, this is on
- its <a href="http://www.sshkeygen.com/about.php">to-do list</a>. That
- isn't really an excuse.
- </li>
- <li>
- Even if keys were delivered over HTTPS, that still relies on people
- diligently checking the authenticity of the certificate. A self-signature
- (as suggested as an alternative in the to-do list) would be impossible to
- check with any reliability; and will people who have trouble with
- non-web-based key generation software really be able or inclined to
- confirm the signature chain? Browsers typically don't enforce this very
- strictly, or if they do they provide fairly simple ways to bypass the
- enforcement, simply because so many sites have broken or poorly-signed SSL
- certificates, and keeping up with all the CAs is pretty hard work too.
- </li>
- <li>
- Furthermore, delivering private keys over HTTPS makes that SSL certificate
- a single giant blinking target. Might it be compromised? How would you
- tell? What servers would need to be compromised in order to get a copy of
- the private SSL key?
- </li>
- <li>
- Sure, Debian is in an awkward position here given the recent OpenSSL
- random number generation vulnerability. However, how do you know that
- sshkeygen.com is running on a system that doesn't suffer from this? (As it
- happens, I have checked, and it doesn't appear to suffer from this
- vulnerability - but most people won't check and won't know how to check.)
- </li>
-</ul>
-
-<p>I <em>think</em> this is probably being done in innocent seriousness
-(although I kind of hope it's a joke in poor taste), and have e-mailed the
-contact address offering to explain why it's a bad idea.</p>
+++ /dev/null
-Python SIGPIPE handling
-
-<p><a href="http://www.enricozini.org/2009/debian/python-pipes/">Enrico</a>
-writes about creating pipelines with Python's <code>subprocess</code>
-module, and notes that you need to take care to close stdout in non-final
-subprocesses so that subprocesses get <code>SIGPIPE</code> correctly. This
-is correct as far as it goes (and true in any language, although there's a
-<a href="http://bugs.python.org/issue1615376">Python bug report requesting
-that <code>subprocess</code> be able to do this itself</a>), but there's an
-additional gotcha with Python that you missed.</p>
-
-<p>Python ignores <code>SIGPIPE</code> on startup, because it prefers to
-check every write and raise an <code>IOError</code> exception rather than
-taking the signal. This is all well and good for Python itself, but most
-Unix subprocesses don't expect to work this way. Thus, when you are creating
-subprocesses from Python, it is <strong>very important</strong> to set
-<code>SIGPIPE</code> back to the default action. Before I realised this was
-necessary, I wrote code that caused serious data loss due to a child process
-carrying on out of control after its parent process died!</p>
-
-<pre>
-import signal
-import subprocess
-
-def subprocess_setup():
- # Python installs a SIGPIPE handler by default. This is usually not what
- # non-Python subprocesses expect.
- signal.signal(signal.SIGPIPE, signal.SIG_DFL)
-
-subprocess.Popen(command, preexec_fn=subprocess_setup)
-</pre>
-
-<p>I filed a <a href="http://bugs.python.org/issue1652">patch</a> a while
-back to add a <code>restore_sigpipe</code> option to
-<code>subprocess.Popen</code>, which would take care of this. As I say in
-that bug report, in a future release I think this ought to be made the
-default, as it's very easy to get things dangerously wrong right now.</p>
+++ /dev/null
-man-db: 'man -K'
-
-<p>I recently implemented <code>man -K</code> (full-text search over all
-manual pages) in <a href="http://man-db.nongnu.org/">man-db</a>. This was
-inspired by a similar feature in Federico Lucifredi's
-<a href="http://primates.ximian.com/~flucifredi/man/">man</a> package
-(formerly maintained by Andries Brouwer). I think I did a much better job of
-it, though. The man package just forks grep for every manual page; man-db
-takes advantage of the pipeline library I wrote for it a while back and does
-it entirely in-process (decompression requires a fork but no exec, while the
-man package has to exec gunzip as well).</p>
-
-<p>The upshot is that, with a hot cache, man-db takes around 40 seconds to
-search all manual pages on my laptop; the man package (also with a hot
-cache) takes around five minutes, and interactive performance goes down the
-drain while it's doing it since it's spawning subprocesses like crazy. If I
-limit to a single section, the disparity is closer to 3x than 10x, but it's
-still very noticeable. It's interesting how much good libraries can do to
-help guide efficient approaches to problems.</p>
-
-<p>Of course, a proper full-text search engine would be much better still, but
-that's a project for some other time ...</p>
+++ /dev/null
-Keysigning bits
-
-<p>If you're generating one of these shiny new RSA keys, do please remember
-to <a href="http://ekaia.org/blog/2009/05/10/creating-new-gpgkey/">generate
-an encryption subkey too</a> if you expect people to sign it - at least your
-more obscure UIDs. I'm not going to mail unencrypted signatures around
-unless I have some out-of-band knowledge that the e-mail address actually
-belongs to the person I met.</p>
-
-<p>I generated a new 4096-bit RSA key myself at DebConf (baa!), and have
-just published a
-<a href="http://www.chiark.greenend.org.uk/~cjwatson/key-transition">key
-transition document</a>. Please consider signing my new key if you signed my
-old one.</p>
+++ /dev/null
-Pipeline library
-
-<p>When I took over <a href="http://man-db.nongnu.org/">man-db</a> in 2001, one of the major problems that became evident after maintaining it for a while was the way it handled subprocesses. The nature of man and friends means that it spends a lot of time calling sequences of programs such as <code>zsoelim < input-file | tbl | nroff -mandoc -Tutf8</code>. Back then, it was using C library facilities such as <code>system</code> and <code>popen</code> for all this, and I had to deal with several bugs where those functions were being called with untrusted input as arguments without properly escaping metacharacters. Of course it was possible to chase around every such call inserting appropriate escaping functions, but this was always bound to be error-prone and one of the tasks that rapidly became important to me was arranging to start subprocesses in a way that was fundamentally immune to this kind of bug.</p>
-
-<p>In higher-level languages, there are usually standard constructs which are safer than just passing a command line to the shell. For example, in Perl you can use <code>system([$command, $arg1, $arg2, ...])</code> to invoke a program with arguments without the interference of the shell, and <code>perlipc(1)</code> describes various facilities for connecting them together. In Python, the <a href="http://docs.python.org/library/subprocess.html">subprocess</a> module allows you to create pipelines easily and safely (as long as you remember the <a href="http://www.chiark.greenend.org.uk/ucgi/~cjwatson/blosxom/2009-07-02-python-sigpipe.html">SIGPIPE gotcha</a>). C has the <code>fork</code> and <code>execve</code> primitives, but assembling these to construct full-blown pipelines correctly is difficult and error-prone, so many programmers don't bother and use the simple but unsafe library facilities instead.</p>
-
-<p>I wrote a couple of thousand lines of library code in man-db to address this problem, loosely and now quite distantly based on code in <a href="http://www.gnu.org/software/groff/">groff</a>. In the following examples, function names starting with <code>command_</code>, <code>pipeline_</code>, or <code>decompress_</code> are real functions in the library, while any other function names are pseudocode.</p>
-
-<p>Constructing the simplified example pipeline from my first paragraph using this library looks like this:</p>
-
-<pre>
-pipeline *p;
-int status;
-
-p = pipeline_new ();
-p->want_infile = "input-file";
-pipeline_command_args (p, "zsoelim", NULL);
-pipeline_command_args (p, "tbl", NULL);
-pipeline_command_args (p, "nroff", "-mandoc", "-Tutf8", NULL);
-pipeline_start (p);
-status = pipeline_wait (p);
-pipeline_free (p);
-</pre>
-
-<p>You might want to construct a command more dynamically:</p>
-
-<pre>
-command *manconv = command_new_args ("manconv", "-f", from_code,
- "-t", "UTF-8", NULL);
-if (quiet)
- command_arg (manconv, "-q");
-pipeline_command (p, manconv);
-</pre>
-
-<p>Perhaps you want an environment variable set only while running a certain command:</p>
-
-<pre>
-command *less = command_new ("less");
-command_setenv (less, "LESSCHARSET", lesscharset);
-</pre>
-
-<p>You might find yourself needing to pass the output of one pipeline to several other pipelines, in a "tee" arrangement:</p>
-
-<pre>
-pipeline *source, *sink1, *sink2;
-
-source = make_source ();
-sink1 = make_sink1 ();
-sink2 = make_sink2 ();
-pipeline_connect (source, sink1, sink2, NULL);
-/* Pump data among these pipelines until there's nothing left. */
-pipeline_pump (source, sink1, sink2, NULL);
-pipeline_free (sink2);
-pipeline_free (sink1);
-pipeline_free (source);
-</pre>
-
-<p>Maybe one of your commands is actually an in-process function, rather than an external program:</p>
-
-<pre>
-command *inproc = command_new_function ("in-process", &func, NULL, NULL);
-pipeline_command (p, inproc);
-</pre>
-
-<p>Sometimes your program needs to consume the output of a pipeline, rather than sending it all to some other subprocess:</p>
-
-<pre>
-pipeline *p = make_pipeline ();
-const char *line;
-
-line = pipeline_peekline (p);
-if (!strstr (line, "coding: UTF-8"))
- printf ("Unicode text follows:\n");
-while (line = pipeline_readline (p))
- printf (" %s", line);
-pipeline_free (p);
-</pre>
-
-<p>man-db deals with compressed files a lot, so I wrote an add-on library for opening compressed files (which is somewhat man-db-specific, but the implementation wasn't difficult given the underlying library):</p>
-
-<pre>
-pipeline *decomp_file = decompress_open (compressed_filename);
-pipeline *decomp_stdin = decompress_fdopen (fileno (stdin));
-</pre>
-
-<p>This library has been in production in man-db for over five years now. The very careful signal handling code has been reviewed independently and the whole thing has been run through multiple static analysis tools, although I would always welcome more review; in particular I have no idea what it would take to make it safe for use in threaded programs since I generally avoid threading wherever possible. There have been a handful of bugs, which I've fixed promptly, and I've added various new features to support particular requirements of man-db (though in as general a way as possible). Every so often I see somebody asking about subprocess handling in C, and I wonder if I should split this library out into a standalone package so that it can be used elsewhere. Web searches for things like "pipeline library" and "libpipeline" don't reveal anything that's a particularly close match for what I have. The licensing would be GPLv2 or later; this isn't likely to be negotiable since some of the original code wasn't mine and in any case I don't feel particularly bad about <a href="http://www.gnu.org/licenses/why-not-lgpl.html">giving an advantage to GPLed programs</a>. For more details on the interface, the <a href="http://bazaar.launchpad.net/~cjwatson/man-db/trunk/annotate/head%3A/lib/pipeline.h">header file</a> is well-commented.</p>
-
-<p>Is there enough interest in this to make the effort of producing a separate library package worthwhile? As well as the general effort of creating a new package, I'd need to do some work to disentangle it from a few bits and pieces specific to man-db. If you maintain a specific package that could use this and you're interested, please contact me with details, mentioning any extensions you think you'd need. I intentionally haven't enabled comments on my blog for various reasons, but you can e-mail me at cjwatson at debian.org or man-db-devel at nongnu.org.</p>
+++ /dev/null
-libpipeline 1.0.0 released
-
-<p>In my <a href="http://www.chiark.greenend.org.uk/ucgi/~cjwatson/blosxom/2010-10-03-pipeline-library.html">previous post</a>, I described the pipeline library from man-db and asked whether people were interested in a standalone release of it. Several people expressed interest, and so I've now released <a href="http://libpipeline.nongnu.org/">libpipeline</a> version 1.0.0. It's in the Debian NEW queue, and <a href="https://launchpad.net/~cjwatson/+archive/ppa">my PPA</a> contains packages of it for Ubuntu lucid and maverick.</p>
-
-<p>I gave a lightning talk on this at UDS in Orlando, and my <a href="http://libpipeline.nongnu.org/libpipeline-lightning-talk.odp">slides</a> are available. I hope there'll be a video at some point which I can link to.</p>
-
-<p>Thanks to Scott James Remnant for code review (some time back), Ian Jackson for an extensive design review, and Kees Cook and Matthias Klose for helpful conversations.</p>
+++ /dev/null
-man-db on Fedora
-
-<p>I just found out by chance that <a href="http://fedoraproject.org/">Fedora</a> 14 switched from their old man package to <a href="http://man-db.nongnu.org/">man-db</a>. This is great news: it should now be the beginning of the end of the divergence of man implementations that happened way back in the mid-1990s, when two different people took John W. Eaton's man package and developed it in different directions without being aware of each other's existence. For a while it looked as though man-db was stuck on just the Debian family and openSUSE, but a number of distributions have switched over in the last few years. As of now, the only remaining major distribution not using man-db is Gentoo, and they have a <a href="http://bugs.gentoo.org/show_bug.cgi?id=284822">bug for switching</a> which I think should be unblocked fairly soon.</p>
-
-<p>In some ways man-db's package name didn't help it; people thought that the main difference was that man-db had a database backend stuck around apropos. These days, the database is one of the least important parts of man-db as far as I'm concerned. Other ways in which it's very significantly superior to anything man could do without years of equivalent effort include correct encoding support, robust child process handling, and use of more modern development facilities (dear catgets: you belong to a previous millennium, so please go away). I'm glad that Fedora has recognised this.</p>
+++ /dev/null
-libpipeline 1.1.0 released
-
-<p>I've released <a href="http://libpipeline.nongnu.org/">libpipeline 1.1.0</a>, and uploaded it to Debian unstable. The changes are mostly just to add a few occasionally useful interfaces:</p>
-
-<ul>
- <li>Add <code>pipecmd_exec</code> to execute a single command, replacing the current process; this is analogous to <code>execvp</code>.</li>
- <li>Add <code>pipecmd_clearenv</code> to clear a command's environment; this is analogous to <code>clearenv</code>.</li>
- <li>Add <code>pipecmd_get_nargs</code> to get the number of arguments to a command.</li>
-</ul>
-
-<p>The shared library actually ends up being a few kilobytes smaller on Debian than 1.0.0, probably because I tweaked the set of Gnulib modules I'm using.</p>
+++ /dev/null
-man-db 2.6.0
-
-<p>I've released man-db 2.6.0 (<a href="http://lists.nongnu.org/archive/html/man-db-announce/2011-04/msg00000.html">announcement</a>, <a href="http://bazaar.launchpad.net/~cjwatson/man-db/trunk/view/1344/NEWS">NEWS</a>, <a href="http://bazaar.launchpad.net/~cjwatson/man-db/trunk/view/1344/ChangeLog">ChangeLog</a>), and uploaded it to Debian unstable. Ubuntu is rapidly approaching beta freeze so I'm not going to try to cram this into 11.04; it'll be in 11.10.</p>
+++ /dev/null
-libpipeline 1.2.1 released
-
-<p>I've released <a href="http://libpipeline.nongnu.org/">libpipeline 1.2.1</a>, and uploaded it to Debian unstable. This is a bug-fix release:</p>
-
-<ul>
- <li>Retry reads and writes on <code>EINTR</code>.</li>
- <li>Fix opening of output files requested by <code>pipeline_want_outfile</code>; these are now created if they do not already exist, and truncated if they do.</li>
- <li><code><pipeline.h></code> is now wrapped in <code>extern "C"</code> when used in a C++ compilation unit.</li>
-</ul>
+++ /dev/null
-OpenSSH 6.0p1
-
-<p>OpenSSH 6.0p1 was <a href="http://www.openssh.com/txt/release-6.0">released</a> a little while back; this weekend I belatedly got round to uploading packages of it to Debian unstable and Ubuntu quantal.</p>
-
-<p>I was a bit delayed by needing to put together an <a href="https://bugzilla.mindrot.org/show_bug.cgi?id=2011">improvement to privsep sandbox selection</a> that particularly matters in the context of distributions. One of the experts on seccomp_filter has commented favourably on it, but I haven't yet had a comment from upstream themselves, so I may need to refine this depending on what they say.</p>
-
-<p>(This is a good example of how it matters that software is often not built on the system that it's going to run on, and in particular that the kernel version is rather likely to be different. Where possible it's always best to detect kernel capabilities at run-time rather than at build-time.)</p>
-
-<p>I didn't make it very clear in the changelog, but using the new seccomp_filter sandbox currently requires <code>UsePrivilegeSeparation sandbox</code> in sshd_config as well as a capable kernel. I won't change the default here in advance of upstream, who still consider privsep sandboxing experimental.</p>
+++ /dev/null
-Testing wanted: GRUB 2.02~beta2 Debian/Ubuntu packages
-
-<p>This is mostly a repost of my <a href="https://lists.ubuntu.com/archives/ubuntu-devel/2014-January/037978.html">ubuntu-devel mail</a> for a wider audience, but see below for some additions.</p>
-
-<p>I'd like to upgrade to GRUB 2.02 for Ubuntu 14.04; it's currently in beta. This represents a year and a half of upstream development, and contains many new features, which you can see in the <a href="http://git.savannah.gnu.org/gitweb/?p=grub.git;a=blob;f=NEWS">NEWS</a> file.</p>
-
-<p>Obviously I want to be very careful with substantial upgrades to the default boot loader. So, I've put this in trusty-proposed, and filed a <a href="https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1269992">blocking bug</a> to ensure that it doesn't reach trusty proper until it's had a reasonable amount of manual testing. If you are already using trusty and have some time to try this out, it would be very helpful to me. I suggest that you only attempt this if you're comfortable driving <code>apt-get</code> directly and recovering from errors at that level, and if you're willing to spend time working with me on narrowing down any problems that arise.</p>
-
-<p>Please ensure that you have rescue media to hand before starting testing. The simplest way to upgrade is to enable trusty-proposed, upgrade ONLY packages whose names start with "grub" (e.g. use <code>apt-get dist-upgrade</code> to show the full list, say no to the upgrade, and then pass all the relevant package names to <code>apt-get install</code>), and then (very important!) disable trusty-proposed again. Provided that there were no errors in this process, you should be safe to reboot. If there were errors, you should be able to downgrade back to 2.00-22 (or 1.27+2.00-22 in the case of grub-efi-amd64-signed).</p>
-
-<p>Please report your experiences (positive and negative) with this upgrade in the <a href="https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1269992">tracking bug</a>. I'm particularly interested in systems that are complex in any way: UEFI Secure Boot, non-trivial disk setups, manual configuration, that kind of thing. If any of the problems you see are also ones you saw with earlier versions of GRUB, please identify those clearly, as I want to prioritise handling regressions over anything else. I've assigned myself to that bug to ensure that messages to it are filtered directly into my inbox.</p>
-
-<p>I'll add a couple of things that weren't in my ubuntu-devel mail. Firstly, this is all in Debian experimental as well (I do all the work in Debian and sync it across, so the grub2 source package in Ubuntu is a verbatim copy of the one in Debian these days). There are some configuration differences applied at build time, but a large fraction of test cases will apply equally well to both. I don't have a definite schedule for pushing this into jessie yet - I only just finished getting 2.00 in place there, and the release schedule gives me a bit more time - but I certainly want to ship jessie with 2.02 or newer, and any test feedback would be welcome. It's probably best to just e-mail feedback to me directly for now, or to the pkg-grub-devel list.</p>
-
-<p>Secondly, a couple of news sites have picked this up and run it as "Canonical intends to ship Ubuntu 14.04 LTS with a beta version of GRUB". This isn't in fact my intent at all. I'm doing this now because I think GRUB 2.02 will be ready in non-beta form in time for Ubuntu 14.04, and indeed that putting it in our development release will help to stabilise it; I'm an upstream GRUB developer too and I find the exposure of widely-used packages very helpful in that context. It will certainly be much easier to upgrade to a beta now and a final release later than it would be to try to jump from 2.00 to 2.02 in a month or two's time.</p>
-
-<p>Even if there's some unforeseen delay and 2.02 isn't released in time, though, I think nearly three months of stabilisation is still plenty to yield a boot loader that I'm comfortable with shipping in an LTS. I've been backporting a lot of changes to 2.00 and even 1.99, and, as ever for an actively-developed codebase, it gets harder and harder over time (in particular, I've spent longer than I'd like hunting down and backporting fixes for non-512-byte sector disks). While I can still manage it, I don't want to be supporting 2.00 for five more years after upstream has moved on; I don't think that would be in anyone's best interests. And I definitely want some of the new features which aren't sensibly backportable, such as several of the new platforms (ARM, ARM64, Xen) and various networking improvements; I can imagine a number of our users being interested in things like optional signature verification of files GRUB reads from disk, improved Mac support, and the TrueCrypt ISO loader, just to name a few. This should be a much stronger base for five-year support.</p>
+++ /dev/null
-Porting GHC: A Tale of Two Architectures
-
-<p>We had <a href="https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2013-December/014795.html">some</a> <a href="https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2014-March/014907.html">requests</a> to get <a href="http://www.haskell.org/ghc/">GHC</a> (the Glasgow Haskell Compiler) up and running on two new Ubuntu architectures: <acronym title="64-bit ARM, a.k.a. aarch64">arm64</acronym>, added in 13.10, and <acronym title="little-endian 64-bit PowerPC">ppc64el</acronym>, added in 14.04. This has been something of a saga, and has involved rather more late-night hacking than is probably good for me.</p>
-
-<h2>Book the First: Recalled to a life of strange build systems</h2>
-
-<p>You might not know it from the sheer bulk of uploads I do sometimes, but I actually don't speak a word of Haskell and it's not very high up my list of things to learn. But I am a pretty experienced build engineer, and I enjoy porting things to new architectures: I'm firmly of the belief that breadth of architecture support is a good way to shake out certain categories of issues in code, that it's worth doing aggressively across an entire distribution, and that, even if you don't think you need something now, new requirements have a habit of coming along when you least expect them and you might as well be prepared in advance. Furthermore, it annoys me when we have excessive noise in our <a href="http://qa.ubuntuwire.com/ftbfs/">build failure</a> and <a href="https://wiki.ubuntu.com/ProposedMigration">proposed-migration</a> output and I often put bits and pieces of spare time into gardening miscellaneous problems there, and at one point there was a lot of Haskell stuff on the list and it got a bit annoying to have to keep sending patches rather than just fixing things myself, and ... well, I ended up as probably the only non-Haskell-programmer on the Debian Haskell team and found myself fixing problems there in my free time. Life is a bit weird sometimes.</p>
-
-<p>Bootstrapping packages on a new architecture is a bit of a black art that only a fairly small number of relatively bitter and twisted people know very much about. Doing it in Ubuntu is specifically painful because we've always forbidden direct binary uploads: all binaries have to come from a build daemon. Compilers in particular often tend to be written in the language they compile, and it's not uncommon for them to build-depend on themselves: that is, you need a previous version of the compiler to build the compiler, stretching back to the dawn of time where somebody put things together with a big magnet or something. So how do you get started on a new architecture? Well, what we do in this case is we construct a binary somehow (usually involving cross-compilation) and insert it as a build-dependency for a proper build in Launchpad. The ability to do this is restricted to a small group of Canonical employees, partly because it's very easy to make mistakes and partly because things like the classic "<a href="http://cm.bell-labs.com/who/ken/trust.html">Reflections on Trusting Trust</a>" are in the backs of our minds somewhere. We have an iron rule for our own sanity that the injected build-dependencies must themselves have been built from the unmodified source package in Ubuntu, although there can be source modifications further back in the chain. Fortunately, we don't need to do this very often, but it does mean that as somebody who can do it I feel an obligation to try and unblock other people where I can.</p>
-
-<p>As far as constructing those build-dependencies goes, sometimes we look for binaries built by other distributions (particularly Debian), and that's pretty straightforward. In this case, though, these two architectures are pretty new and the Debian ports are only just getting going, and as far as I can tell none of the other distributions with active arm64 or ppc64el ports (or trivial name variants) has got as far as porting GHC yet. Well, OK. This was somewhere around the Christmas holidays and I had some time. Muggins here cracks his knuckles and decides to have a go at bootstrapping it from scratch. It can't be that hard, right? Not to mention that it was a blocker for over 600 entries on that build failure list I mentioned, which is definitely enough to make me sit up and take notice; we'd even had the odd customer request for it.</p>
-
-<p>Several attempts later and I was starting to doubt my sanity, not least for trying in the first place. We ship GHC 7.6, and upgrading to 7.8 is not a project I'd like to tackle until the much more experienced Haskell folks in Debian have switched to it in unstable. The <a href="https://ghc.haskell.org/trac/ghc/wiki/Building/Porting">porting documentation for 7.6</a> has bitrotted more or less beyond usability, and the <a href="https://ghc.haskell.org/trac/ghc/wiki/CrossCompilation">corresponding documentation for 7.8</a> really isn't backportable to 7.6. I tried building 7.8 for ppc64el anyway, picking that on the basis that we had quicker hardware for it and didn't seem likely to be particularly more arduous than arm64 (ho ho), and I even got to the point of having a cross-built stage2 compiler (stage1, in the cross-building case, is a GHC binary that runs on your starting architecture and generates code for your target architecture) that I could copy over to a ppc64el box and try to use as the base for a fully-native build, but it segfaulted incomprehensibly just after spawning any child process. Compilers tend to do rather a lot, especially when they're built to use GCC to generate object code, so this was a pretty serious problem, and it resisted analysis. I poked at it for a while but didn't get anywhere, and I had other things to do so declared it a write-off and gave up.</p>
-
-<h2>Book the Second: The golden thread of progress</h2>
-
-<p>In March, another mailing list conversation prodded me into finding a <a href="https://ghcarm.wordpress.com/2014/01/18/unregisterised-ghc-head-build-for-arm64-platform/">blog entry by Karel Gardas</a> on building GHC for arm64. This was enough to be worth another look, and indeed it turned out that (with some help from Karel in private mail) I was able to cross-build a compiler that actually worked and could be used to run a fully-native build that also worked. Of course this was 7.8, since as I mentioned cross-building 7.6 is unrealistically difficult unless you're considerably more of an expert on GHC's labyrinthine build system than I am. OK, no problem, right? Getting a GHC at all is the hard bit, and 7.8 must be at least as capable as 7.6, so it should be able to build 7.6 easily enough ...</p>
-
-<p>Not so much. What I'd missed here was that compiler engineers generally only care very much about building the compiler with <em>older</em> versions of itself, and if the language in question has any kind of deprecation cycle then the compiler itself is likely to be behind on various things compared to more typical code since it has to be buildable with older versions. This means that the removal of some deprecated interfaces from 7.8 posed a problem, as did some changes in certain <acronym title="primitive operations">primops</acronym> that had gained an associated compatibility layer in 7.8 but nobody had gone back to put the corresponding compatibility layer into 7.6. GHC supports running Haskell code through the C preprocessor, and there's a <code>__GLASGOW_HASKELL__</code> definition with the compiler's version number, so this was just a slog tracking down changes in git and adding <code>#ifdef</code>-guarded code that coped with the newer compiler (remembering that stage1 will be built with 7.8 and stage2 with stage1, i.e. 7.6, from the same source tree). More inscrutably, GHC has its own packaging system called Cabal which is also used by the compiler build process to determine which subpackages to build and how to link them against each other, and some crucial subpackages weren't being built: it looked like it was stuck on picking versions from "stage0" (i.e. the initial compiler used as an input to the whole process) when it should have been building its own. Eventually I figured out that this was because GHC's use of its packaging system hadn't anticipated this case, and was selecting the higher version of the <code>ghc</code> package itself from stage0 rather than the version it was about to build for itself, and thus never actually tried to build most of the compiler. Editing <code>ghc_stage1_DEPS</code> in <code>ghc/stage1/package-data.mk</code> after its initial generation sorted this out. One late night building round and round in circles for a while until I had something stable, and a Debian source upload to add basic support for the architecture name (and other changes which were a bit over the top in retrospect: I didn't need to touch the embedded copy of libffi, as we build with the system one), and I was able to feed this all into Launchpad and watch the builders munch away very satisfyingly at the Haskell library stack for a while.</p>
-
-<p>This was all interesting, and finally all that work was actually paying off in terms of getting to watch a slew of several hundred build failures vanish from arm64 (the final count was something like 640, I think). The fly in the ointment was that ppc64el was still blocked, as the problem there wasn't building 7.6, it was getting a working 7.8. But now I really did have other much more urgent things to do, so I figured I just wouldn't get to this by release time and stuck it on the figurative shelf.</p>
-
-<h2>Book the Third: The track of a bug</h2>
-
-<p>Then, last Friday, I cleared out my urgent pile and thought I'd have another quick look. (I get a bit obsessive about things like this that smell of "interesting intellectual puzzle".) slyfox on the #ghc IRC channel gave me some general debugging advice and, particularly usefully, a reduced example program that I could use to debug just the process-spawning problem without having to wade through noise from running the rest of the compiler. I reproduced the same problem there, and then found that the program crashed earlier (in <code>stg_ap_0_fast</code>, part of the run-time system) if I compiled it with <code>+RTS -Da -RTS</code>. I nailed it down to a small enough region of assembly that I could see all of the assembly, the source code, and an intermediate representation or two from the compiler, and then started meditating on what makes ppc64el special.</p>
-
-<p>You see, the vast majority of porting bugs come down to what I might call gross properties of the architecture. You have things like whether it's 32-bit or 64-bit, big-endian or little-endian, whether <code>char</code> is signed or unsigned, that sort of thing. There's a <a href="https://wiki.debian.org/ArchitectureSpecificsMemo">big table</a> on the Debian wiki that handily summarises most of the important ones. Sometimes you have to deal with distribution-specific things like whether GL or GLES is used; often, especially for new variants of existing architectures, you have to cope with foolish configure scripts that think they can guess certain things from the architecture name and get it wrong (assuming that <code>powerpc*</code> means big-endian, for instance). We often have to update <code>config.guess</code> and <code>config.sub</code>, and on ppc64el we have the additional hassle of updating libtool macros too. But I've done a lot of this stuff and I'd accounted for everything I could think of. ppc64el is actually a lot like amd64 in terms of many of these porting-relevant properties, and not even that far off arm64 which I'd just successfully ported GHC to, so I couldn't be dealing with anything particularly obvious. There was some hand-written assembly which certainly could have been problematic, but I'd carefully checked that this wasn't being used by the "unregisterised" (no specialised machine dependencies, so relatively easy to port but not well-optimised) build I was using. A problem around spawning processes suggested a problem with <code>SIGCHLD</code> handling, but I ruled that out by slowing down the first child process that it spawned and using <code>strace</code> to confirm that <code>SIGSEGV</code> was the first signal received. What on earth was the problem?</p>
-
-<p>From some painstaking gdb work, one thing I eventually noticed was that <code>stg_ap_0_fast</code>'s local stack appeared to be being corrupted by a function call, specifically a call to the colourfully-named <code>debugBelch</code>. Now, when IBM's toolchain engineers were putting together ppc64el based on ppc64, they took the opportunity to fix a number of problems with their ABI: there's an <a href="https://bugs.openjdk.java.net/browse/JDK-8035647">OpenJDK bug</a> with a handy list of references. One of the things I noticed there was that there were some <a href="http://gcc.gnu.org/ml/gcc-patches/2013-11/msg01149.html">stack allocation optimisations</a> in the new ABI, which affected functions that don't call any vararg functions and don't call any functions that take enough parameters that some of them have to be passed on the stack rather than in registers. <code>debugBelch</code> takes varargs: hmm. Now, the calling code isn't quite in C as such, but in a related dialect called "Cmm", a variant of C-- (yes, minus), that GHC uses to help bridge the gap between the functional world and its code generation, and which is compiled down to C by GHC. When importing C functions into Cmm, GHC generates prototypes for them, but it doesn't do enough parsing to work out the true prototype; instead, they all just get something like <code>extern StgFunPtr f(void);</code>. In most architectures you can get away with this, because the arguments get passed in the usual calling convention anyway and it all works out, but on ppc64el this means that the caller doesn't generate enough stack space and then the callee tries to save its varargs onto the stack in an area that in fact belongs to the caller, and suddenly everything goes south. Things were starting to make sense.</p>
-
-<p>Now, <code>debugBelch</code> is only used in optional debugging code; but <code>runInteractiveProcess</code> (the function associated with the initial round of failures) takes no fewer than twelve arguments, plenty to force some of them onto the stack. I poked around the GCC patch for this ABI change a bit and determined that it only optimised away the stack allocation if it had a full prototype for all the callees, so I guessed that changing those prototypes to <code>extern StgFunPtr f();</code> might work: it's still technically wrong, not least because omitting the parameter list is an obsolescent feature in C11, but it's at least just omitting information about the parameter list rather than actively lying about it. I tweaked that and ran the cross-build from scratch again. Lo and behold, suddenly I had a working compiler, and I could go through the same build-7.6-using-7.8 procedure as with arm64, much more quickly this time now that I knew what I was doing. One <a href="https://ghc.haskell.org/trac/ghc/ticket/8965">upstream bug</a>, one Debian upload, and several bootstrapping builds later, and GHC was up and running on another architecture in Launchpad. Success!</p>
-
-<h2>Epilogue</h2>
-
-<p>There's still more to do. I gather there may be a Google Summer of Code project in Linaro to write proper native code generation for GHC on arm64: this would make things a good deal faster, but also enable GHCi (the interpreter) and Template Haskell, and thus clear quite a few more build failures. Since there's already native code generation for ppc64 in GHC, getting it going for ppc64el would probably only be a couple of days' work at this point. But these are niceties by comparison, and I'm more than happy with what I got working for 14.04.</p>
-
-<p>The upshot of all of this is that I may be the first non-Haskell-programmer to ever port GHC to two entirely new architectures. I'm not sure if I gain much from that personally aside from a lot of lost sleep and being considered extremely strange. It has, however, been by far the most challenging set of packages I've ported, and a fascinating trip through some odd corners of build systems and undefined behaviour that I don't normally need to touch.</p>
--- /dev/null
+PY?=python
+PELICAN?=pelican
+PELICANOPTS=
+
+BASEDIR=$(CURDIR)
+INPUTDIR=$(BASEDIR)/content
+OUTPUTDIR=$(BASEDIR)/output
+CONFFILE=$(BASEDIR)/pelicanconf.py
+PUBLISHCONF=$(BASEDIR)/publishconf.py
+
+SSH_HOST=chiark-external
+SSH_PORT=22
+SSH_USER=cjwatson
+SSH_TARGET_DIR=public-html/blog/
+
+DEBUG ?= 0
+ifeq ($(DEBUG), 1)
+ PELICANOPTS += -D
+endif
+
+help:
+ @echo 'Makefile for a pelican Web site '
+ @echo ' '
+ @echo 'Usage: '
+ @echo ' make html (re)generate the web site '
+ @echo ' make clean remove the generated files '
+ @echo ' make regenerate regenerate files upon modification '
+ @echo ' make publish generate using production settings '
+ @echo ' make serve [PORT=8000] serve site at http://localhost:8000'
+ @echo ' make devserver [PORT=8000] start/restart develop_server.sh '
+ @echo ' make stopserver stop local server '
+ @echo ' make ssh_upload upload the web site via SSH '
+ @echo ' make rsync_upload upload the web site via rsync+ssh '
+ @echo ' '
+ @echo 'Set the DEBUG variable to 1 to enable debugging, e.g. make DEBUG=1 html'
+ @echo ' '
+
+html:
+ $(PELICAN) $(INPUTDIR) -o $(OUTPUTDIR) -s $(CONFFILE) $(PELICANOPTS)
+
+clean:
+ [ ! -d $(OUTPUTDIR) ] || rm -rf $(OUTPUTDIR)
+
+regenerate:
+ $(PELICAN) -r $(INPUTDIR) -o $(OUTPUTDIR) -s $(CONFFILE) $(PELICANOPTS)
+
+serve:
+ifdef PORT
+ cd $(OUTPUTDIR) && $(PY) -m pelican.server $(PORT)
+else
+ cd $(OUTPUTDIR) && $(PY) -m pelican.server
+endif
+
+devserver:
+ifdef PORT
+ $(BASEDIR)/develop_server.sh restart $(PORT)
+else
+ $(BASEDIR)/develop_server.sh restart
+endif
+
+stopserver:
+ kill -9 `cat pelican.pid`
+ kill -9 `cat srv.pid`
+ @echo 'Stopped Pelican and SimpleHTTPServer processes running in background.'
+
+publish:
+ $(PELICAN) $(INPUTDIR) -o $(OUTPUTDIR) -s $(PUBLISHCONF) $(PELICANOPTS)
+
+ssh_upload: publish
+ scp -P $(SSH_PORT) -r $(OUTPUTDIR)/* $(SSH_USER)@$(SSH_HOST):$(SSH_TARGET_DIR)
+
+rsync_upload: publish
+ rsync -e "ssh -p $(SSH_PORT)" -P -rvzc --delete $(OUTPUTDIR)/ $(SSH_USER)@$(SSH_HOST):$(SSH_TARGET_DIR) --cvs-exclude
+
+.PHONY: html help clean regenerate serve devserver publish ssh_upload rsync_upload
--- /dev/null
+#! /usr/bin/perl
+use warnings;
+use strict;
+use lib '/home/cjwatson/perl/lib/perl5';
+
+use Mojolicious::Lite;
+
+our $target_base = "http://www.chiark.greenend.org.uk/~cjwatson/blog";
+
+my %redirections = (
+ '2006-01-03-hello' => 'hello',
+ '2006-01-09-bzr-shelve' => 'bzr-shelve',
+ '2006-02-06-sponge' => 'sponge',
+ '2007-07-04-keysigning-psa' => 'keysigning-psa',
+ '2007-09-17-man-db-encodings' => 'man-db-encodings',
+ '2008-01-29-utf-8-manual-pages' => 'utf-8-manual-pages',
+ '2008-06-23-ssh-keygen' => 'ssh-keygen',
+ '2009-07-02-python-sigpipe' => 'python-sigpipe',
+ '2009-07-14-man-db-K' => 'man-db-K',
+ '2009-07-31-keysigning-bits' => 'keysigning-bits',
+ '2010-10-03-pipeline-library' => 'pipeline-library',
+ '2010-10-29-libpipeline-released' => 'libpipeline-released',
+ '2010-12-02-man-db-on-fedora' => 'man-db-on-fedora',
+ '2010-12-11-libpipeline-1.1.0-released' => 'libpipeline-1.1.0-released',
+ '2011-04-09-man-db-2.6.0' => 'man-db-2.6.0',
+ '2012-03-02-libpipeline-1.2.1-released' => 'libpipeline-1.2.1-released',
+ '2012-05-27-openssh-6.0p1' => 'openssh-6.0p1',
+ '2014-01-18-testing-wanted-grub-2.02-beta2' => 'testing-wanted-grub-2.02-beta2',
+ '2014-04-15-porting-ghc-a-tale-of-two-architectures' => 'porting-ghc-a-tale-of-two-architectures',
+ 'debian/2006-01-03-openssh-iutf8' => 'openssh-iutf8',
+ 'debian/2006-01-27-debconf-cdebconf-coinstallable' => 'debconf-cdebconf-coinstallable',
+ 'debian/2006-12-23-moving-conffiles' => 'moving-conffiles',
+ 'debian/2007-11-29-safe-upgrade' => 'safe-upgrade',
+ 'debian/2008-06-23-reply-perl-is-strange' => 'reply-perl-is-strange',
+ 'debian/2010-02-21-catching-up' => 'catching-up',
+ 'debian/2010-03-03-debhelper-statistics' => 'debhelper-statistics',
+ 'debian/2010-03-22-parted-2.2-transition' => 'parted-2.2-transition',
+ 'debian/2010-03-25-thoughts-on-3.0-quilt-format' => 'thoughts-on-3.0-quilt-format',
+ 'debian/2010-06-04-hacking-on-grub2' => 'hacking-on-grub2',
+ 'debian/2010-06-21-grub2-boot-problems' => 'grub2-boot-problems',
+ 'debian/2010-07-02-grub2-with-luck' => 'grub2-with-luck',
+ 'debian/2010-07-10-debhelper-statistics-redux' => 'debhelper-statistics-redux',
+ 'debian/2010-08-28-windows-applications-making-grub2-unbootable' => 'windows-applications-making-grub2-unbootable',
+ 'summerofcode/2006/2006-05-25-gsoc-d-i-hurd-started' => 'gsoc-d-i-hurd-started',
+ 'summerofcode/2006/2006-05-25-gsoc-ubiquity-migration-started' => 'gsoc-ubiquity-migration-started',
+ 'ubuntu/2006-01-03-single-stage-installer' => 'single-stage-installer',
+ 'ubuntu/2008-01-31-vim-lpbug-omnicomplete' => 'vim-lpbug-omnicomplete',
+ 'ubuntu/2008-04-12-desktop-automount-pain' => 'desktop-automount-pain',
+ 'ubuntu/2008-10-27-totem-bbc-plugin' => 'totem-bbc-plugin',
+ 'ubuntu/2009-02-27-bug-triage-rants' => 'bug-triage-rants',
+ 'ubuntu/2009-03-05-bug-triage-redux' => 'bug-triage-redux',
+ 'ubuntu/2009-05-28-code_swarm' => 'code_swarm',
+ 'ubuntu/2009-11-13-tissue-of-lies' => 'tissue-of-lies',
+ 'ubuntu/2010-05-10-openssh-5.5p1-for-lucid' => 'openssh-5.5p1-for-lucid',
+ 'ubuntu/2010-12-06-ntp-synchronisation-problems' => 'ntp-synchronisation-problems',
+ 'ubuntu/2011-03-14-wubi-bug-693671' => 'wubi-bug-693671',
+ 'ubuntu/2011-10-06-brainstorm-review' => 'brainstorm-review',
+ 'ubuntu/2011-10-24-quality-in-12-04' => 'quality-in-12-04',
+ 'ubuntu/2012-01-29-apt-resolver-bugs' => 'apt-resolver-bugs',
+ 'ubuntu/2012-10-26-automatic-installability-checking' => 'automatic-installability-checking',
+ 'ubuntu/2014-10-26-moving-on-but-not-too-far' => 'moving-on-but-not-too-far',
+);
+
+any '/' => sub {
+ my $c = shift;
+ return $c->redirect_to("$target_base/");
+};
+
+any '/*path' => sub {
+ my $c = shift;
+ my $path = $c->stash('path');
+ if (exists $redirections{$path}) {
+ return $c->redirect_to("$target_base/$redirections{$path}.html");
+ }
+ if ($path =~ /(.*)\.html/ and exists $redirections{$1}) {
+ return $c->redirect_to("$target_base/$redirections{$1}.html");
+ }
+ return $c->reply->not_found;
+};
+
+app->start;
+
+__DATA__
+
+@@ not_found.html.ep
+<!DOCTYPE html>
+<html>
+ <head><title>Entry not found</title></head>
+ <body><p>Entry not found, but you may be able to find it by starting from my <%= link_to blog => "$main::target_base/" %>.</p></body>
+</html>
--- /dev/null
+/*
+ Name: Smashing HTML5
+ Date: July 2009
+ Description: Sample layout for HTML5 and CSS3 goodness.
+ Version: 1.0
+ License: MIT <http://opensource.org/licenses/MIT>
+ Licensed by: Smashing Media GmbH <http://www.smashingmagazine.com/>
+ Original author: Enrique RamÃrez <http://enrique-ramirez.com/>
+*/
+
+/* Imports */
+@import url("reset.css");
+@import url("pygment.css");
+@import url("typogrify.css");
+@import url(//fonts.googleapis.com/css?family=Yanone+Kaffeesatz&subset=latin);
+
+/***** Global *****/
+/* Body */
+body {
+ background: #F5F4EF;
+ color: #000305;
+ font-size: 87.5%; /* Base font size: 14px */
+ font-family: 'Trebuchet MS', Trebuchet, 'Lucida Sans Unicode', 'Lucida Grande', 'Lucida Sans', Arial, sans-serif;
+ line-height: 1.429;
+ margin: 0;
+ padding: 0;
+ text-align: left;
+}
+
+/* Headings */
+h1 {font-size: 2em }
+h2 {font-size: 1.571em} /* 22px */
+h3 {font-size: 1.429em} /* 20px */
+h4 {font-size: 1.286em} /* 18px */
+h5 {font-size: 1.143em} /* 16px */
+h6 {font-size: 1em} /* 14px */
+
+h1, h2, h3, h4, h5, h6 {
+ font-weight: 400;
+ line-height: 1.1;
+ margin-bottom: .8em;
+ font-family: 'Yanone Kaffeesatz', arial, serif;
+}
+
+h3, h4, h5, h6 { margin-top: .8em; }
+
+hr { border: 2px solid #EEEEEE; }
+
+/* Anchors */
+a {outline: 0;}
+a img {border: 0px; text-decoration: none;}
+a:link, a:visited {
+ color: #C74350;
+ padding: 0 1px;
+ text-decoration: underline;
+}
+a:hover, a:active {
+ background-color: #C74350;
+ color: #fff;
+ text-decoration: none;
+ text-shadow: 1px 1px 1px #333;
+}
+
+h1 a:hover {
+ background-color: inherit
+}
+
+/* Paragraphs */
+div.line-block,
+p { margin-top: 1em;
+ margin-bottom: 1em;}
+
+strong, b {font-weight: bold;}
+em, i {font-style: italic;}
+
+/* Lists */
+ul {
+ list-style: outside disc;
+ margin: 0em 0 0 1.5em;
+}
+
+ol {
+ list-style: outside decimal;
+ margin: 0em 0 0 1.5em;
+}
+
+li { margin-top: 0.5em;}
+
+/*
+.post-info {
+ float:right;
+ margin:10px;
+ padding:5px;
+}
+
+.post-info p{
+ margin-top: 1px;
+ margin-bottom: 1px;
+}
+*/
+
+.readmore { float: right }
+
+dl {margin: 0 0 1.5em 0;}
+dt {font-weight: bold;}
+dd {margin-left: 1.5em;}
+
+pre{background-color: rgb(238, 238, 238); padding: 10px; margin: 10px; overflow: auto;}
+
+/* Quotes */
+blockquote {
+ margin: 20px;
+ font-style: italic;
+}
+cite {}
+
+q {}
+
+div.note {
+ float: right;
+ margin: 5px;
+ font-size: 85%;
+ max-width: 300px;
+}
+
+/* Tables */
+table {margin: .5em auto 1.5em auto; width: 98%;}
+
+ /* Thead */
+ thead th {padding: .5em .4em; text-align: left;}
+ thead td {}
+
+ /* Tbody */
+ tbody td {padding: .5em .4em;}
+ tbody th {}
+
+ tbody .alt td {}
+ tbody .alt th {}
+
+ /* Tfoot */
+ tfoot th {}
+ tfoot td {}
+
+/* HTML5 tags */
+header, section, footer,
+aside, nav, article, figure {
+ display: block;
+}
+
+abbr, acronym {
+ border-bottom: .1em dotted;
+}
+
+/***** Layout *****/
+.body {clear: both; margin: 0 auto;}
+img.right, figure.right {float: right; margin: 0 0 2em 2em;}
+img.left, figure.left {float: left; margin: 0 2em 2em 0;}
+
+/*
+ Header
+*****************/
+#banner {
+ margin: 0 auto;
+ padding: 2.5em 0 0 0;
+}
+
+ /* Banner */
+ #banner h1 {font-size: 3.571em; line-height: 0;}
+ #banner h1 a:link, #banner h1 a:visited {
+ color: #000305;
+ display: block;
+ font-weight: bold;
+ margin: 0 0 .6em .2em;
+ text-decoration: none;
+ }
+ #banner h1 a:hover, #banner h1 a:active {
+ background: none;
+ color: #C74350;
+ text-shadow: none;
+ }
+
+ #banner h1 strong {font-size: 0.36em; font-weight: normal;}
+
+ /* Main Nav */
+ #banner nav {
+ background: #000305;
+ font-size: 1.143em;
+ height: 40px;
+ line-height: 30px;
+ margin: 0 20px 2em 20px;
+ padding: 0;
+ text-align: center;
+
+ border-radius: 5px;
+ -moz-border-radius: 5px;
+ -webkit-border-radius: 5px;
+ }
+
+ #banner nav ul {list-style: none; margin: 0 auto;}
+ #banner nav li {float: left; display: inline; margin: 0;}
+
+ #banner nav a:link, #banner nav a:visited {
+ color: #fff;
+ display: inline-block;
+ height: 30px;
+ padding: 5px 1.5em;
+ text-decoration: none;
+ }
+ #banner nav a:hover, #banner nav a:active,
+ #banner nav .active a:link, #banner nav .active a:visited {
+ background: #C74451;
+ color: #fff;
+ text-shadow: none !important;
+ }
+
+ #banner nav li:first-child a {
+ border-top-left-radius: 5px;
+ -moz-border-radius-topleft: 5px;
+ -webkit-border-top-left-radius: 5px;
+
+ border-bottom-left-radius: 5px;
+ -moz-border-radius-bottomleft: 5px;
+ -webkit-border-bottom-left-radius: 5px;
+ }
+
+/*
+ Featured
+*****************/
+.featured {
+ background: #fff;
+ margin-bottom: 2em;
+ overflow: hidden;
+ padding: 20px;
+
+ border-radius: 10px;
+ -moz-border-radius: 10px;
+ -webkit-border-radius: 10px;
+}
+
+.featured figure {
+ border: 2px solid #eee;
+ float: right;
+ margin: 0.786em 2em 0 5em;
+ width: 248px;
+}
+.featured figure img {display: block; float: right;}
+
+.featured h2 {color: #C74451; font-size: 1.714em; margin-bottom: 0.333em;}
+.featured h3 {font-size: 1.429em; margin-bottom: .5em;}
+
+.featured h3 a:link, .featured h3 a:visited {color: #000305; text-decoration: none;}
+.featured h3 a:hover, .featured h3 a:active {color: #fff;}
+
+/*
+ Body
+*****************/
+#content {
+ background: #fff;
+ margin-bottom: 2em;
+ overflow: hidden;
+ padding: 20px 20px;
+
+ border-radius: 10px;
+ -moz-border-radius: 10px;
+ -webkit-border-radius: 10px;
+}
+
+.paginator {
+ padding: 20px;
+}
+
+/*
+ Extras
+*****************/
+#extras {margin: 0 auto 3em auto; overflow: hidden;}
+
+#extras ul {list-style: none; margin: 0;}
+#extras li {border-bottom: 1px solid #fff;}
+#extras h2 {
+ color: #C74350;
+ font-size: 1.429em;
+ margin-bottom: .25em;
+ padding: 0 3px;
+}
+
+#extras a:link, #extras a:visited {
+ color: #444;
+ display: block;
+ border-bottom: 1px solid #F4E3E3;
+ text-decoration: none;
+ padding: .3em .25em;
+}
+
+#extras a:hover, #extras a:active {color: #fff;}
+
+ /* Blogroll */
+ #extras .blogroll {
+ float: left;
+ width: 615px;
+ }
+
+ #extras .blogroll li {float: left; margin: 0 20px 0 0; width: 185px;}
+
+ /* Social */
+ #extras .social {
+ float: right;
+ width: 175px;
+ }
+
+ #extras div[class='social'] a {
+ background-repeat: no-repeat;
+ background-position: 3px 6px;
+ padding-left: 25px;
+ }
+
+ /* Icons */
+ .social a[href*='plus.google.com'] {background-image: url('../images/icons/google-plus.png');}
+ .social a[type$='atom+xml'], .social a[type$='rss+xml'] {background-image: url('../images/icons/rss.png');}
+ .social a[href*='twitter.com'] {background-image: url('../images/icons/twitter.png');}
+
+/*
+ About
+*****************/
+#about {
+ background: #fff;
+ font-style: normal;
+ margin-bottom: 2em;
+ overflow: hidden;
+ padding: 20px;
+ text-align: left;
+
+ border-radius: 10px;
+ -moz-border-radius: 10px;
+ -webkit-border-radius: 10px;
+}
+
+#about .primary {float: left; width: 165px;}
+#about .primary strong {color: #C64350; display: block; font-size: 1.286em;}
+#about .photo {float: left; margin: 5px 20px;}
+
+#about .url:link, #about .url:visited {text-decoration: none;}
+
+#about .bio {float: right; width: 500px;}
+
+/*
+ Footer
+*****************/
+#contentinfo {padding: 0 20px 2em; text-align: right;}
+
+/***** Sections *****/
+/* Blog */
+.hentry {
+ display: block;
+ clear: both;
+ border-bottom: 1px solid #eee;
+ padding: 1.5em 0;
+}
+li:last-child .hentry, #content > .hentry {border: 0; margin: 0;}
+#content > .hentry {padding: 1em 0;}
+.hentry img{display : none ;}
+.entry-title {font-size: 3em; margin-bottom: 10px; margin-top: 0;}
+.entry-title a:link, .entry-title a:visited {text-decoration: none; color: #333;}
+.entry-title a:visited {background-color: #fff;}
+
+.hentry .post-info * {font-style: normal;}
+
+ /* Content */
+ .hentry footer {margin-bottom: 2em;}
+ .hentry footer address {display: inline;}
+ #posts-list footer address {display: block;}
+
+ /* Blog Index */
+ #posts-list {list-style: none; margin: 0;}
+ #posts-list .hentry {padding-left: 10px; position: relative;}
+
+ #posts-list footer {
+ left: 10px;
+ position: relative;
+ float: left;
+ top: 0.5em;
+ width: 190px;
+ }
+
+ /* About the Author */
+ #about-author {
+ background: #f9f9f9;
+ clear: both;
+ font-style: normal;
+ margin: 2em 0;
+ padding: 10px 20px 15px 20px;
+
+ border-radius: 5px;
+ -moz-border-radius: 5px;
+ -webkit-border-radius: 5px;
+ }
+
+ #about-author strong {
+ color: #C64350;
+ clear: both;
+ display: block;
+ font-size: 1.429em;
+ }
+
+ #about-author .photo {border: 1px solid #ddd; float: left; margin: 5px 1em 0 0;}
+
+ /* Comments */
+ #comments-list {list-style: none; margin: 0 1em;}
+ #comments-list blockquote {
+ background: #f8f8f8;
+ clear: both;
+ font-style: normal;
+ margin: 0;
+ padding: 15px 20px;
+
+ border-radius: 5px;
+ -moz-border-radius: 5px;
+ -webkit-border-radius: 5px;
+ }
+ #comments-list footer {color: #888; padding: .5em 1em 0 0; text-align: right;}
+
+ #comments-list li:nth-child(2n) blockquote {background: #F5f5f5;}
+
+ /* Add a Comment */
+ #add-comment label {clear: left; float: left; text-align: left; width: 150px;}
+ #add-comment input[type='text'],
+ #add-comment input[type='email'],
+ #add-comment input[type='url'] {float: left; width: 200px;}
+
+ #add-comment textarea {float: left; height: 150px; width: 495px;}
+
+ #add-comment p.req {clear: both; margin: 0 .5em 1em 0; text-align: right;}
+
+ #add-comment input[type='submit'] {float: right; margin: 0 .5em;}
+ #add-comment * {margin-bottom: .5em;}
--- /dev/null
+.hll {
+background-color:#eee;
+}
+.c {
+color:#408090;
+font-style:italic;
+}
+.err {
+border:1px solid #FF0000;
+}
+.k {
+color:#007020;
+font-weight:bold;
+}
+.o {
+color:#666666;
+}
+.cm {
+color:#408090;
+font-style:italic;
+}
+.cp {
+color:#007020;
+}
+.c1 {
+color:#408090;
+font-style:italic;
+}
+.cs {
+background-color:#FFF0F0;
+color:#408090;
+}
+.gd {
+color:#A00000;
+}
+.ge {
+font-style:italic;
+}
+.gr {
+color:#FF0000;
+}
+.gh {
+color:#000080;
+font-weight:bold;
+}
+.gi {
+color:#00A000;
+}
+.go {
+color:#303030;
+}
+.gp {
+color:#C65D09;
+font-weight:bold;
+}
+.gs {
+font-weight:bold;
+}
+.gu {
+color:#800080;
+font-weight:bold;
+}
+.gt {
+color:#0040D0;
+}
+.kc {
+color:#007020;
+font-weight:bold;
+}
+.kd {
+color:#007020;
+font-weight:bold;
+}
+.kn {
+color:#007020;
+font-weight:bold;
+}
+.kp {
+color:#007020;
+}
+.kr {
+color:#007020;
+font-weight:bold;
+}
+.kt {
+color:#902000;
+}
+.m {
+color:#208050;
+}
+.s {
+color:#4070A0;
+}
+.na {
+color:#4070A0;
+}
+.nb {
+color:#007020;
+}
+.nc {
+color:#0E84B5;
+font-weight:bold;
+}
+.no {
+color:#60ADD5;
+}
+.nd {
+color:#555555;
+font-weight:bold;
+}
+.ni {
+color:#D55537;
+font-weight:bold;
+}
+.ne {
+color:#007020;
+}
+.nf {
+color:#06287E;
+}
+.nl {
+color:#002070;
+font-weight:bold;
+}
+.nn {
+color:#0E84B5;
+font-weight:bold;
+}
+.nt {
+color:#062873;
+font-weight:bold;
+}
+.nv {
+color:#BB60D5;
+}
+.ow {
+color:#007020;
+font-weight:bold;
+}
+.w {
+color:#BBBBBB;
+}
+.mf {
+color:#208050;
+}
+.mh {
+color:#208050;
+}
+.mi {
+color:#208050;
+}
+.mo {
+color:#208050;
+}
+.sb {
+color:#4070A0;
+}
+.sc {
+color:#4070A0;
+}
+.sd {
+color:#4070A0;
+font-style:italic;
+}
+.s2 {
+color:#4070A0;
+}
+.se {
+color:#4070A0;
+font-weight:bold;
+}
+.sh {
+color:#4070A0;
+}
+.si {
+color:#70A0D0;
+font-style:italic;
+}
+.sx {
+color:#C65D09;
+}
+.sr {
+color:#235388;
+}
+.s1 {
+color:#4070A0;
+}
+.ss {
+color:#517918;
+}
+.bp {
+color:#007020;
+}
+.vc {
+color:#BB60D5;
+}
+.vg {
+color:#BB60D5;
+}
+.vi {
+color:#BB60D5;
+}
+.il {
+color:#208050;
+}
--- /dev/null
+/*
+ Name: Reset Stylesheet
+ Description: Resets browser's default CSS
+ Author: Eric Meyer
+ Author URI: http://meyerweb.com/eric/tools/css/reset/
+*/
+
+/* v1.0 | 20080212 */
+html, body, div, span, applet, object, iframe,
+h1, h2, h3, h4, h5, h6, p, blockquote, pre,
+a, abbr, acronym, address, big, cite, code,
+del, dfn, em, font, img, ins, kbd, q, s, samp,
+small, strike, strong, sub, sup, tt, var,
+b, u, i, center,
+dl, dt, dd, ol, ul, li,
+fieldset, form, label, legend,
+table, caption, tbody, tfoot, thead, tr, th, td {
+ background: transparent;
+ border: 0;
+ font-size: 100%;
+ margin: 0;
+ outline: 0;
+ padding: 0;
+ vertical-align: baseline;
+}
+
+body {line-height: 1;}
+
+ol, ul {list-style: none;}
+
+blockquote, q {quotes: none;}
+
+blockquote:before, blockquote:after,
+q:before, q:after {
+ content: '';
+ content: none;
+}
+
+/* remember to define focus styles! */
+:focus {
+ outline: 0;
+}
+
+/* remember to highlight inserts somehow! */
+ins {text-decoration: none;}
+del {text-decoration: line-through;}
+
+/* tables still need 'cellspacing="0"' in the markup */
+table {
+ border-collapse: collapse;
+ border-spacing: 0;
+}
\ No newline at end of file
--- /dev/null
+.caps {font-size:.92em;}
+.amp {color:#666; font-size:1.05em;font-family:"Warnock Pro", "Goudy Old Style","Palatino","Book Antiqua",serif; font-style:italic;}
+.dquo {margin-left:-.38em;}
--- /dev/null
+@import url("main.css");
+
+body {
+ font:1.3em/1.3 "Hoefler Text","Georgia",Georgia,serif,sans-serif;
+}
+
+.post-info{
+ display: none;
+}
+
+#banner nav {
+ display: none;
+ -moz-border-radius: 0px;
+ margin-bottom: 20px;
+ overflow: hidden;
+ font-size: 1em;
+ background: #F5F4EF;
+}
+
+#banner nav ul{
+ padding-right: 50px;
+}
+
+#banner nav li{
+ float: right;
+ color: #000;
+}
+
+#banner nav li a {
+ color: #000;
+}
+
+#banner h1 {
+ margin-bottom: -18px;
+}
+
+#featured, #extras {
+ padding: 50px;
+}
+
+#featured {
+ padding-top: 20px;
+}
+
+#extras {
+ padding-top: 0px;
+ padding-bottom: 0px;
+}
--- /dev/null
+{% extends "base.html" %}
+{% block content %}
+<section id="content" class="body">
+<h1>Archives for {{ SITENAME }}</h1>
+
+<dl>
+{% for article in dates %}
+ <dt>{{ article.locale_date }}</dt>
+ <dd><a href="{{ SITEURL }}/{{ article.url }}">{{ article.title }}</a></dd>
+{% endfor %}
+</dl>
+</section>
+{% endblock %}
--- /dev/null
+{% extends "base.html" %}
+{% block title %}{{ article.title|striptags }}{% endblock %}
+{% block content %}
+<section id="content" class="body">
+ <article>
+ <header>
+ <h1 class="entry-title">
+ <a href="{{ SITEURL }}/{{ article.url }}" rel="bookmark"
+ title="Permalink to {{ article.title|striptags }}">{{ article.title }}</a></h1>
+ {% include 'twitter.html' %}
+ </header>
+
+ <div class="entry-content">
+ {% include 'article_infos.html' %}
+ {{ article.content }}
+ </div><!-- /.entry-content -->
+ {% if DISQUS_SITENAME and SITEURL and article.status != "draft" %}
+ <div class="comments">
+ <h2>Comments !</h2>
+ <div id="disqus_thread"></div>
+ <script type="text/javascript">
+ var disqus_shortname = '{{ DISQUS_SITENAME }}';
+ var disqus_identifier = '{{ article.url }}';
+ var disqus_url = '{{ SITEURL }}/{{ article.url }}';
+ (function() {
+ var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
+ dsq.src = '//{{ DISQUS_SITENAME }}.disqus.com/embed.js';
+ (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
+ })();
+ </script>
+ <noscript>Please enable JavaScript to view the comments.</noscript>
+ </div>
+ {% endif %}
+
+ </article>
+</section>
+{% endblock %}
--- /dev/null
+<div class="post-info">
+ <p>
+ <abbr class="published" title="{{ article.date.isoformat() }}">{{ article.locale_date }}</abbr>
+ {% if article.modified %}
+ (updated <abbr class="modified" title="{{ article.modified.isoformat() }}">{{ article.locale_modified }}</abbr>)
+ {% endif %}
+
+ {% if article.authors %}
+ by {% for author in article.authors %}
+ <a class="url fn" href="{{ SITEURL }}/{{ author.url }}">{{ author }}</a>
+ {% endfor %}
+ {% endif %}
+
+ •
+ <a href="{{ SITEURL }}/{{ article.category.url }}">{{ article.category }}</a>
+{% include 'taglist.html' %}
+{% import 'translations.html' as translations with context %}
+{{ translations.translations_for(article) }}
+</div><!-- /.post-info -->
--- /dev/null
+{% extends "index.html" %}
+{% block title %}{{ SITENAME }} - {{ author }}{% endblock %}
--- /dev/null
+{% extends "base.html" %}
+
+{% block title %}{{ SITENAME }} - Authors{% endblock %}
+
+{% block content %}
+
+<section id="content" class="body">
+ <h1>Authors on {{ SITENAME }}</h1>
+ <ul>
+ {% for author, articles in authors|sort %}
+ <li><a href="{{ SITEURL }}/{{ author.url }}">{{ author }}</a> ({{ articles|count }})</li>
+ {% endfor %}
+ </ul>
+</section>
+
+{% endblock %}
--- /dev/null
+<!DOCTYPE html>
+<html lang="{{ DEFAULT_LANG }}">
+<head>
+ <meta charset="utf-8" />
+ <title>{% block title %}{{ SITENAME }}{%endblock%}</title>
+ <link rel="stylesheet" href="{{ SITEURL }}/{{ THEME_STATIC_DIR }}/css/{{ CSS_FILE }}" />
+ {% if FEED_ALL_ATOM %}
+ <link href="{{ FEED_DOMAIN }}/{{ FEED_ALL_ATOM }}" type="application/atom+xml" rel="alternate" title="{{ SITENAME }} Atom Feed" />
+ {% endif %}
+ {% if FEED_ALL_RSS %}
+ <link href="{{ FEED_DOMAIN }}/{{ FEED_ALL_RSS }}" type="application/rss+xml" rel="alternate" title="{{ SITENAME }} RSS Feed" />
+ {% endif %}
+
+ <!--[if IE]>
+ <script src="http://html5shiv.googlecode.com/svn/trunk/html5.js"></script>
+ <![endif]-->
+</head>
+
+<body id="index" class="home">
+ <header id="banner" class="body">
+ <h1><a href="{{ SITEURL }}/">{{ SITENAME }} {% if SITESUBTITLE %} <strong>{{ SITESUBTITLE }}</strong>{% endif %}</a></h1>
+ <nav><ul>
+ {% for title, link in MENUITEMS %}
+ <li><a href="{{ link }}">{{ title }}</a></li>
+ {% endfor %}
+ {% if DISPLAY_PAGES_ON_MENU -%}
+ {% for pg in PAGES %}
+ <li{% if pg == page %} class="active"{% endif %}><a href="{{ SITEURL }}/{{ pg.url }}">{{ pg.title }}</a></li>
+ {% endfor %}
+ {% endif %}
+ {% if DISPLAY_CATEGORIES_ON_MENU -%}
+ {% for cat, null in categories %}
+ <li{% if cat == category %} class="active"{% endif %}><a href="{{ SITEURL }}/{{ cat.url }}">{{ cat }}</a></li>
+ {% endfor %}
+ {% endif %}
+ </ul></nav>
+ </header><!-- /#banner -->
+ {% block content %}
+ {% endblock %}
+ <section id="extras" class="body">
+ {% if LINKS %}
+ <div class="blogroll">
+ <h2>blogroll</h2>
+ <ul>
+ {% for name, link in LINKS %}
+ <li><a href="{{ link }}">{{ name }}</a></li>
+ {% endfor %}
+ </ul>
+ </div><!-- /.blogroll -->
+ {% endif %}
+ {% if SOCIAL or FEED_ALL_ATOM or FEED_ALL_RSS %}
+ <div class="social">
+ <h2>social</h2>
+ <ul>
+ {% if FEED_ALL_ATOM %}
+ <li><a href="{{ FEED_DOMAIN }}/{{ FEED_ALL_ATOM }}" type="application/atom+xml" rel="alternate">atom feed</a></li>
+ {% endif %}
+ {% if FEED_ALL_RSS %}
+ <li><a href="{{ FEED_DOMAIN }}/{{ FEED_ALL_RSS }}" type="application/rss+xml" rel="alternate">rss feed</a></li>
+ {% endif %}
+
+ {% for name, link in SOCIAL %}
+ <li><a href="{{ link }}">{{ name }}</a></li>
+ {% endfor %}
+ </ul>
+ </div><!-- /.social -->
+ {% endif %}
+ </section><!-- /#extras -->
+
+ <footer id="contentinfo" class="body">
+ <p>Powered by <a href="http://getpelican.com/">Pelican</a>. The theme is based on one by <a href="http://coding.smashingmagazine.com/2009/08/04/designing-a-html-5-layout-from-scratch/">Smashing Magazine</a>, thanks!</p>
+ </footer><!-- /#contentinfo -->
+
+{% include 'disqus_script.html' %}
+</body>
+</html>
--- /dev/null
+{% extends "index.html" %}
+{% block title %}{{ SITENAME }} - {{ category }}{% endblock %}
--- /dev/null
+{% if DISQUS_SITENAME %}<p>There are <a href="{{ SITEURL }}/{{ article.url }}#disqus_thread">comments</a>.</p>{% endif %}
--- /dev/null
+{% if DISQUS_SITENAME %}
+<script type="text/javascript">
+ var disqus_shortname = '{{ DISQUS_SITENAME }}';
+ (function () {
+ var s = document.createElement('script'); s.async = true;
+ s.type = 'text/javascript';
+ s.src = '//' + disqus_shortname + '.disqus.com/count.js';
+ (document.getElementsByTagName('HEAD')[0] || document.getElementsByTagName('BODY')[0]).appendChild(s);
+ }());
+</script>
+{% endif %}
--- /dev/null
+{% extends "base.html" %}
+{% block content_title %}{% endblock %}
+{% block content %}
+{% if articles %}
+ {% for article in articles_page.object_list %}
+
+ <aside class="featured body">
+ <article>
+ <h1 class="entry-title"><a href="{{ SITEURL }}/{{ article.url }}">{{ article.title }}</a></h1>
+ {% include 'article_infos.html' %}{{ article.content }}{% include 'comments.html' %}
+ </article>
+ {% if loop.length == 1 %}
+ {% include 'pagination.html' %}
+ {% endif %}
+ </aside><!-- /.featured -->
+ {% if loop.last %}
+ {% if articles_page.has_previous() or loop.length > 1 %}
+ <div class="body">
+ {% include 'pagination.html' %}
+ </div>
+ {% endif %}
+ {% endif %}
+ {% endfor %}
+{% else %}
+<section id="content" class="body">
+<h2>Pages</h2>
+ {% for page in PAGES %}
+ <li><a href="{{ SITEURL }}/{{ page.url }}">{{ page.title }}</a></li>
+ {% endfor %}
+</section>
+{% endif %}
+{% endblock content %}
--- /dev/null
+{% extends "base.html" %}
+{% block title %}{{ page.title }}{% endblock %}
+{% block content %}
+<section id="content" class="body">
+ <h1 class="entry-title">{{ page.title }}</h1>
+ {% import 'translations.html' as translations with context %}
+ {{ translations.translations_for(page) }}
+ {% if PDF_PROCESSOR %}<a href="{{ SITEURL }}/pdf/{{ page.slug }}.pdf">get
+ the pdf</a>{% endif %}
+ {{ page.content }}
+</section>
+{% endblock %}
--- /dev/null
+{% extends "base.html" %}
+{% block content %}
+<section id="content" class="body">
+<h1>Archives for {{ period | reverse | join(' ') }}</h1>
+
+<dl>
+{% for article in dates %}
+ <dt>{{ article.locale_date }}</dt>
+ <dd><a href="{{ SITEURL }}/{{ article.url }}">{{ article.title }}</a></dd>
+{% endfor %}
+</dl>
+</section>
+{% endblock %}
--- /dev/null
+{% extends "index.html" %}
+{% block title %}{{ SITENAME }} - {{ tag }}{% endblock %}
--- /dev/null
+{% if article.tags %}• {% for tag in article.tags %}<a href="{{ SITEURL }}/{{ tag.url }}">{{ tag | escape }}</a> {% endfor %}<br />{% endif %}
--- /dev/null
+{% extends "base.html" %}
+
+{% block title %}{{ SITENAME }} - Tags{% endblock %}
+
+{% block content %}
+
+<section id="content" class="body">
+ <h1>Tags for {{ SITENAME }}</h1>
+ <ul>
+ {% for tag, articles in tags|sort %}
+ <li><a href="{{ SITEURL }}/{{ tag.url }}">{{ tag }}</a> ({{ articles|count }})</li>
+ {% endfor %}
+ </ul>
+</section>
+
+{% endblock %}
--- /dev/null
+{% macro translations_for(article) %}
+{% if article.translations %}
+Translations:
+ {% for translation in article.translations %}
+ <a href="{{ SITEURL }}/{{ translation.url }}">{{ translation.lang }}</a>
+ {% endfor %}
+{% endif %}
+{% endmacro %}
--- /dev/null
+{% if TWITTER_USERNAME %}
+<a href="http://twitter.com/share" class="twitter-share-button" data-count="horizontal" data-via="{{TWITTER_USERNAME}}">Tweet</a><script type="text/javascript" src="//platform.twitter.com/widgets.js"></script>
+{% endif %}
--- /dev/null
+Title: APT resolver bugs
+Slug: apt-resolver-bugs
+Date: 2012-01-30 10:54:25 +0000
+Category: ubuntu
+Tags: apt, ubuntu, planet-debian, planet-ubuntu
+
+I've managed to go for eleven years working on Debian and nearly eight on
+Ubuntu without ever needing to teach myself how APT's resolver works. I get
+the impression that there's a certain mystique about it in general
+(alternatively, I'm just the last person to figure this out). Recently,
+though, I had a couple of Ubuntu upgrade bugs to fix that turned out to be
+bugs in the resolver, and I thought it might be interesting to walk through
+the process of fixing them based on the `Debug::pkgProblemResolver=true` log
+files.
+
+## Breakage with Breaks
+
+The first was [Ubuntu bug #922485](https://bugs.launchpad.net/bugs/922485)
+([apt.log](https://launchpadlibrarian.net/91187038/apt.log)). To understand
+the log, you first need to know that APT makes up to ten passes of the
+resolver to attempt to fix broken dependencies by upgrading, removing, or
+holding back packages; if there are still broken packages after this point,
+it's generally because it's got itself stuck in some kind of loop, and it
+bails out rather than carrying on forever. The current pass number is shown
+in each "Investigating" log entry, so they start with "Investigating (0)"
+and carry on up to at most "Investigating (9)". Any packages that you see
+still being investigated on the tenth pass are probably something to do with
+whatever's going wrong.
+
+In this case, most packages have been resolved by the end of the fourth
+pass, but `xserver-xorg-core` is causing some trouble. (Not a particular
+surprise, as it's an important package with lots of relationships.) We can
+see that each breakage is:
+
+ Broken xserver-xorg-core:i386 Breaks on xserver-xorg-video-6 [ i386 ] < none > ( none )
+
+This is a
+[`Breaks`](http://www.debian.org/doc/debian-policy/ch-relationships.html#s-breaks)
+(a relatively new package relationship type introduced a few years ago as a
+sort of weaker form of `Conflicts`) on a virtual package, which means that
+in order to unpack `xserver-xorg-core` each package that provides
+`xserver-xorg-video-6` must be deconfigured. Much like `Conflicts`, APT
+responds to this by upgrading providing packages to versions that don't
+provide the offending virtual package if it can, and otherwise removing
+them. We can see it doing just that in the log (some lines omitted):
+
+ Investigating (0) xserver-xorg-core [ i386 ] < 2:1.7.6-2ubuntu7.10 -> 2:1.11.3-0ubuntu8 > ( x11 )
+ Fixing xserver-xorg-core:i386 via remove of xserver-xorg-video-tseng:i386
+ Investigating (1) xserver-xorg-core [ i386 ] < 2:1.7.6-2ubuntu7.10 -> 2:1.11.3-0ubuntu8 > ( x11 )
+ Fixing xserver-xorg-core:i386 via remove of xserver-xorg-video-i740:i386
+ Investigating (2) xserver-xorg-core [ i386 ] < 2:1.7.6-2ubuntu7.10 -> 2:1.11.3-0ubuntu8 > ( x11 )
+ Fixing xserver-xorg-core:i386 via remove of xserver-xorg-video-nv:i386
+
+OK, so that makes sense - presumably upgrading those packages didn't help at
+the time. But look at the pass numbers. Rather than just fixing all the
+packages that provide `xserver-xorg-video-6` in a single pass, which it
+would be perfectly able to do, it only fixes one per pass. This means that
+if a package `Breaks` a virtual package which is provided by more than ten
+installed packages, the resolver will fail to handle that situation. On
+inspection of the code, this was being handled correctly for `Conflicts` by
+carrying on through the list of possible targets for the dependency relation
+in that case, but apparently when `Breaks` support was implemented in APT
+this case was overlooked. The fix is to carry on through the list of
+possible targets for any "negative" dependency relation, not just
+`Conflicts`, and I've filed a patch as [Debian
+bug #657695](http://bugs.debian.org/657695).
+
+## My cup overfloweth
+
+The second bug I looked at was [Ubuntu
+bug #917173](https://bugs.launchpad.net/bugs/917173)
+([apt.log](https://launchpadlibrarian.net/90202820/apt.log)). Just as in
+the previous case, we can see the resolver "running out of time" by reaching
+the end of the tenth pass with some dependencies still broken. This one is
+a lot less obvious, though. The last few entries clearly indicate that the
+resolver is stuck in a loop:
+
+ Investigating (8) dpkg [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( admin )
+ Broken dpkg:i386 Breaks on dpkg-dev [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( utils ) (< 1.15.8)
+ Considering dpkg-dev:i386 29 as a solution to dpkg:i386 7205
+ Upgrading dpkg-dev:i386 due to Breaks field in dpkg:i386
+ Investigating (8) dpkg-dev [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( utils )
+ Broken dpkg-dev:i386 Depends on libdpkg-perl [ i386 ] < none -> 1.16.1.2ubuntu5 > ( perl ) (= 1.16.1.2ubuntu5)
+ Considering libdpkg-perl:i386 12 as a solution to dpkg-dev:i386 29
+ Holding Back dpkg-dev:i386 rather than change libdpkg-perl:i386
+ Investigating (9) dpkg [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( admin )
+ Broken dpkg:i386 Breaks on dpkg-dev [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( utils ) (< 1.15.8)
+ Considering dpkg-dev:i386 29 as a solution to dpkg:i386 7205
+ Upgrading dpkg-dev:i386 due to Breaks field in dpkg:i386
+ Investigating (9) dpkg-dev [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( utils )
+ Broken dpkg-dev:i386 Depends on libdpkg-perl [ i386 ] < none -> 1.16.1.2ubuntu5 > ( perl ) (= 1.16.1.2ubuntu5)
+ Considering libdpkg-perl:i386 12 as a solution to dpkg-dev:i386 29
+ Holding Back dpkg-dev:i386 rather than change libdpkg-perl:i386
+
+The new version of `dpkg` requires upgrading `dpkg-dev`, but it can't
+because of something wrong with `libdpkg-perl`. Following the breadcrumb
+trail back through the log, we find:
+
+ Investigating (1) libdpkg-perl [ i386 ] < none -> 1.16.1.2ubuntu5 > ( perl )
+ Broken libdpkg-perl:i386 Depends on perl [ i386 ] < 5.10.1-8ubuntu2.1 -> 5.14.2-6ubuntu1 > ( perl )
+ Considering perl:i386 1472 as a solution to libdpkg-perl:i386 12
+ Holding Back libdpkg-perl:i386 rather than change perl:i386
+
+ Investigating (1) perl [ i386 ] < 5.10.1-8ubuntu2.1 -> 5.14.2-6ubuntu1 > ( perl )
+ Broken perl:i386 Depends on perl-base [ i386 ] < 5.10.1-8ubuntu2.1 -> 5.14.2-6ubuntu1 > ( perl ) (= 5.14.2-6ubuntu1)
+ Considering perl-base:i386 5806 as a solution to perl:i386 1472
+ Removing perl:i386 rather than change perl-base:i386
+
+ Investigating (1) perl-base [ i386 ] < 5.10.1-8ubuntu2.1 -> 5.14.2-6ubuntu1 > ( perl )
+ Broken perl-base:i386 PreDepends on libc6 [ i386 ] < 2.11.1-0ubuntu7.8 -> 2.13-24ubuntu2 > ( libs ) (>= 2.11)
+ Considering libc6:i386 -17473 as a solution to perl-base:i386 5806
+ Added libc6:i386 to the remove list
+
+ Investigating (0) libc6 [ i386 ] < 2.11.1-0ubuntu7.8 -> 2.13-24ubuntu2 > ( libs )
+ Broken libc6:i386 Depends on libc-bin [ i386 ] < 2.11.1-0ubuntu7.8 -> 2.13-24ubuntu2 > ( libs ) (= 2.11.1-0ubuntu7.8)
+ Considering libc-bin:i386 10358 as a solution to libc6:i386 -17473
+ Removing libc6:i386 rather than change libc-bin:i386
+
+So ultimately the problem is something to do with libc6; but what? [As
+Steve Langasek said in the
+bug](https://bugs.launchpad.net/ubuntu/+source/apt/+bug/917173/comments/10),
+libc6's dependencies have been very carefully structured, and surely we
+would have seen some hint of it elsewhere if they were wrong. At this point
+ideally I wanted to break out GDB or at the very least experiment a bit with
+`apt-get`, but due to some tedious local problems I hadn't been able to
+restore the `apt-clone` state file for this bug onto my system so that I
+could attack it directly. So I fell back on the last refuge of the
+frustrated debugger and sat and thought about it for a bit.
+
+Eventually I noticed something. The numbers after the package names in the
+third line of each of these log entries are "scores": roughly, the more
+important a package is, the higher its score should be. The function that
+calculates these is `pkgProblemResolver::MakeScores()` in
+[apt-pkg/algorithms.cc](http://bazaar.launchpad.net/+branch/apt/view/1951/apt-pkg/algorithms.cc).
+Reading this, I noticed that the various values added up to make each score
+are almost all provably positive, for example:
+
+ :::c
+ Scores[I->ID] += abs(OldScores[D.ParentPkg()->ID]);
+
+The only exceptions are an initial -1 or -2 points for `Priority: optional`
+or `Priority: extra` packages respectively, or some values that could
+theoretically be configured to be negative but weren't in this case. OK.
+So how come `libc6` has such a huge negative score of -17473, when one would
+normally expect it to be an extremely powerful package with a large positive
+score?
+
+Oh. This is computer programming, not mathematics ... and each score is
+stored in a `signed short`, so in a sufficiently large upgrade all those
+bonus points add up to something larger than 32767 and everything goes
+haywire. Bingo. Make it an `int` instead - the number of installed
+packages is going to be on the order of tens of thousands at most, so it's
+not as though it'll make a substantial difference to the amount of memory
+used - and chances are everything will be fine. I've filed a patch as
+[Debian bug #657732](http://bugs.debian.org/657732).
+
+I'd expected this to be a pretty challenging pair of bugs. While I
+certainly haven't lost any respect for the APT maintainers for dealing with
+this stuff regularly, it wasn't as bad as I thought. I'd expected to have
+to figure out how to retune some slightly out-of-balance heuristics and not
+really know whether I'd broken anything else in the process; but in the end
+both patches were very straightforward.
--- /dev/null
+Title: Automatic installability checking
+Slug: automatic-installability-checking
+Date: 2012-10-26 10:18:26 +0100
+Modified: 2012-10-26 10:20:07 +0100
+Category: ubuntu
+Tags: ubuntu, planet-debian, planet-ubuntu
+
+I've just finished deploying automatic installability checking for Ubuntu's
+development release, which is more or less equivalent to the way that
+uploads are promoted from Debian unstable to testing. See [my ubuntu-devel
+post](https://lists.ubuntu.com/archives/ubuntu-devel/2012-October/036043.html)
+and [my ubuntu-devel-announce
+post](https://lists.ubuntu.com/archives/ubuntu-devel-announce/2012-October/000989.html)
+for details. This now means that we'll be opening the archive for general
+development once glibc 2.16 packages are ready.
+
+I'm very excited about this because it's something I've wanted to do for a
+long, long time. In fact, back in 2004 when I had my very first telephone
+conversation with a certain spaceman about this crazy Debian-based project
+he wanted me to work on, I remember talking about Debian's testing migration
+system and some ways I thought it could be improved. I don't remember the
+details of that conversation any more and what I just deployed may well bear
+very little resemblance to it, but it should transform the extent to which
+our development release is continuously usable.
+
+The next step is to hook in [autopkgtest](http://dep.debian.net/deps/dep8/)
+results. This will allow us to do a degree of automatic testing of
+reverse-dependencies when we upgrade low-level libraries.
--- /dev/null
+Title: Top ideas on Ubuntu Brainstorm (August 2011)
+Slug: brainstorm-review
+Date: 2011-10-06 16:58:51 +0100
+Category: ubuntu
+Tags: ubuntu, planet-debian, planet-ubuntu
+
+The Ubuntu Technical Board conducts a regular review of the most popular [Ubuntu Brainstorm](http://brainstorm.ubuntu.com/) ideas (previous reviews conducted by [Matt Zimmerman](http://mdzlog.alcor.net/2010/12/10/ubuntu-brainstorm-top-10-for-december-2010/) and [Martin Pitt](http://www.piware.de/2011/04/top-ideas-on-ubuntu-brainstorm-march-2011/)). This time it was my turn. Apologies for the late arrival of this review.
+
+## Contact lens in the Unity Dash ([#27584](http://brainstorm.ubuntu.com/idea/27584/))
+
+Unity supports [Lenses](https://wiki.ubuntu.com/Unity/Lenses), which provide a consistent way for users to quickly search for information via the Dash. Current lenses include Applications, Files, and Music, but a number of people have asked for contacts to be accessible using the same interface.
+
+While Canonical's DX team isn't currently working on this for Ubuntu 11.10 or 12.04, we'd love somebody who's interested in this to get involved. Allison Randal [explains how to get started](http://allisonrandal.com/2011/09/27/contacts-lens/), including some skeleton example code and several useful links.
+
+## Displaying Ubuntu version information ([#27460](http://brainstorm.ubuntu.com/idea/27460/))
+
+Several people have asked for it to be more obvious what Ubuntu version they're running, as well as other general information about their system.
+
+John Lea, user experience architect on the Unity team, responds that in Ubuntu 11.10 the new LightDM greeter shows the Ubuntu version number, making that basic information very easily visible. For more detail, System Settings -> System Info provides a simple summary.
+
+## Volume adjustments for headphone use ([#27275](http://brainstorm.ubuntu.com/idea/27275/))
+
+People often find that they need to adjust their sound volume when plugging in or removing headphones. It seems as though the computer ought to be able to remember this kind of thing and do it automatically; after all, a major goal of Ubuntu is to make the desktop Just Work.
+
+David Henningson, a member of Canonical's OEM Services group and an Ubuntu audio developer, [responds](http://voices.canonical.com/david.henningsson/2011/09/29/independent-volume-for-headphones-and-speakers/) on his blog with a summary of how PulseAudio jack detection has improved matters in Ubuntu 11.10, and what's left to do:
+
+> The good news: in the upcoming Ubuntu Oneiric (11.10), this is actually
+> working. The bad news: it isn't working for everyone.
+
+## Making it easier to find software to handle a file ([#28148](http://brainstorm.ubuntu.com/idea/28148/))
+
+Ubuntu is not always as helpful as it could be when you don't have the right software installed to handle a particular file.
+
+Michael Vogt, one of the developers of the Ubuntu Software Center, responded to this. It seems that most of the pieces to make this work nicely are in place, but there are a few more bits of glue required:
+
+> Thanks a lot for this suggestion. I like the idea and it's something that
+> software-center itself supports now. In the coming version 5.0 we will
+> offer to "sort by top-rated" (based on the ratings&reviews data). It's
+> also possible to search for an application based on its mime data. To
+> search for a mime-type, you can enter "mime:text/html" or "mime:audio/ogg"
+> into the search field. What is needed however is better integration into
+> the file manager nautilus. I will make sure this gets attention at the
+> next developer meeting and filed
+> [bug #860536](https://launchpad.net/bugs/860536) about it.
+>
+> In nautilus, there is now a button called "Find applications online"
+> available as an option when opening an unknown file or when the user
+> selects "open with...other application" in the context menu. But that
+> will not use the data from software-center.
+
+## Show pop-up alert on low battery ([#28037](http://brainstorm.ubuntu.com/idea/28037/))
+
+Some users have reported on Brainstorm that they are not alerted frequently enough when their laptop's battery is low, as they clearly ought to be.
+
+This is an odd one, because there are already several power alert levels and this has been working well for us for some time. Nevertheless, enough people have voted for this idea that there must be something behind it, perhaps a bug that only affects certain systems. Martin Pitt, technical lead of the Ubuntu desktop team, has [responded](http://brainstorm.ubuntu.com/idea/28037/) directly to the Brainstorm idea with a description of the current system and how to file a bug when it does not work as intended.
--- /dev/null
+Title: Bug triage rants
+Slug: bug-triage-rants
+Date: 2009-03-02 14:51:37 +0000
+Category: ubuntu
+Tags: ubuntu, planet-debian, planet-ubuntu
+
+I hate to say this, but often when somebody does lots of bug triage on a
+package I work on, I find it to be a net loss for me. I end up having to go
+through all the things that were changed, correct a bunch of them,
+occasionally pacify angry bug submitters, and all the rest of it, and often
+the benefits are minimal at best.
+
+I would very much like this not to be the case. Bug triage is supposed to
+help developers be more efficient, and I think most people who do bug triage
+are generally well-intentioned and eager to help. Accordingly, here is a
+series of mini-rants intended to have educational value.
+
+ * **Bugs are not like fruit.**
+
+ Fruit goes bad if you leave it too long. By and large, bugs don't,
+ especially if they're on software that doesn't change very much. There
+ is no reason why a bug filed against a package in Ubuntu 4.10 where the
+ relevant code hasn't changed much since shouldn't still be perfectly
+ valid. Even if it isn't, it deserves proper consideration.
+
+ My biggest single annoyance with bug triage is people coming around and
+ asking if bugs are still valid when they haven't put any effort into
+ reproducing them themselves. This annoys bug submitters too; every so
+ often somebody replies and says "didn't you even bother to check?".
+ This gives a very bad impression of us as a project - wouldn't it be
+ better if we looked as if we knew what we were talking about? There is
+ a good reason to do this kind of check, of course: random undiagnosed
+ crash reports and the like may well go away due to related changes, and
+ it is occasionally worth checking. But if the bug is already
+ well-understood and/or well-described, you should just go and check
+ whether it's still there rather than asking.
+
+ As I understand it, the intended workflow is that people file bugs, then
+ if they aren't clear enough bug triagers work with the submitter to
+ gather information until they are, then they're passed to developers for
+ further work. We seem to have added an extra step wherein submitters
+ must periodically give their bug a health-check, and if they don't then
+ it gets closed as being out of date. In a small minority of cases this
+ is useful; in most cases, frankly, it makes us look a bit clueless. Can
+ we please stop doing this? The more we waste people's time doing this,
+ the less likely it is that they'll bother to respond to us, and this
+ might help our statistics but doesn't help the project as a whole.
+
+ I know that there's a problem with bug count. I think every project of
+ non-trivial size has that problem. But, honestly, the right answer is
+ to *fix more bugs* - and, personally, I would be able to spend more time
+ doing that if I weren't often running around trying to make sure that
+ bugs I care about aren't getting overenthusiastically closed just
+ because somebody thinks they've been lying around too long.
+
+ There is a good way to expire bugs like this, of course. It goes
+ something like this: "I've read through your bug and tried to reproduce
+ it with a current release, but I'm afraid I can't do so. Are you still
+ experiencing it? If not, then I think it might have been fixed by [this
+ change I found in the package's history that seems to be related]." You
+ can't do this *en masse*, but you'll get a much better response from
+ submitters, you'll learn more doing it, and in the process of doing the
+ necessary investigation of each bug you'll find that there are many
+ cases you don't have to ask about at all.
+
+ * **Wishlist bugs are not intrinsically bad.**
+
+ There are certainly cases where something is far too broad or vague for
+ a bug report; but there are also plenty of cases, probably far more,
+ where the wish in question is a relatively small change to the program,
+ or doesn't need any more sophisticated tracking, and a wishlist bug is
+ just right. If you don't know the program very well, it may be
+ difficult to tell whether a wishlist bug is appropriate or not; in that
+ case, just leave the bug alone.
+
+ Please, for the love of all that's holy, don't close wishlist bugs
+ saying that people should use Brainstorm or write a specification
+ instead! If you don't want to see wishlist bugs in your statistics,
+ just filter them out; it's quite easy to do. Even worse, don't tell
+ people that something probably isn't a good idea when you aren't
+ familiar with the software; people who have gone to the effort of
+ writing up their idea for us deserve a response from somebody who knows
+ the software well. I've encountered cases where friends of mine
+ submitted a bug report (sometimes even at my request) and then a triager
+ told them it was a bad idea and closed their bug. This sort of thing
+ puts people off Ubuntu.
+
+ Specifications are software design documents. As such, they are best
+ written by software designers. People who tell other people to go and
+ write a specification may not realise that as a result of doing this for
+ three years it's now essentially impossible to find anything in the
+ specification system! The intent was never that every user of Ubuntu
+ would need to write a specification to get anything changed;
+ specifications are used by developers to document the results of
+ discussions and write up plans. They are not a straightforward
+ alternative to wishlist bugs, nor do they turn out to work very well as
+ what many formal processes call "requirements documents"; the process of
+ refining the latter in the context of Ubuntu might involve wishlist
+ bugs, mailing list threads, wiki pages, private discussions with
+ developers, or things of that nature, and probably shouldn't involve
+ creating a specification until the requirements-gathering process is
+ well underway.
+
+ * **Closing a bug is taking an item off somebody's to-do list.**
+
+ You wouldn't go up to a colleague's whiteboard and take an eraser to it
+ unless you were sure that was OK, would you? Yet people seem to do that
+ all the time with bugs. It's OK when the bug is really just like a
+ support request - "help, it crashed, what do I do?" - and either you're
+ pretty sure it's user error or there's just no way to get enough
+ information to fix it. But once the initial triage process is done, now
+ it's on somebody's to-do list.
+
+ This is closely related to ...
+
+ * **If a developer has accepted it, leave it alone.**
+
+ Every so often I find that there's a bug that I have accepted by way of
+ a bug comment or setting to Triaged or whatever, or even a bug that I
+ filed on a package I work on as a reminder to myself, and somebody comes
+ along and asks for more information or asks if we can still reproduce it
+ or something. The hit rate on this kind of thing is extraordinarily
+ low. There's a good chance that the developer went and verified the bug
+ against the code, and in that case it certainly doesn't need more
+ information (or they would have asked for it) and it probably isn't
+ going to go away without anyone noticing.
+
+ In most other free software projects, developers file bug reports
+ themselves as a reminder about things that need to be done, and people
+ leave them alone unless they're intending to help with the fix. In
+ Ubuntu, developers also have to spend time making sure that those to-do
+ items don't get expired. Nobody is helped by this.
+
+ [launchpad-gm-scripts](https://launchpad.net/launchpad-gm-scripts)
+ includes a Greasemonkey script called `lp_karma_suffix`, which can help
+ you to identify developers without having to spend lots of time clicking
+ around.
+
+ * **Check whether the package is being actively worked on.**
+
+ Some packages are actively worked on in Ubuntu; some aren't (e.g. we
+ just sync packages from Debian, or they're basically orphaned, or
+ whatever). It's worth checking which is which before doing any kind of
+ extensive triage work. If it's being actively worked on, why not go and
+ talk to the developer(s) in question first? It's only polite, and it
+ will probably help you to do a better job.
--- /dev/null
+Title: Bug triage, redux
+Slug: bug-triage-redux
+Date: 2009-03-05 11:04:02 +0000
+Category: ubuntu
+Tags: ubuntu, planet-debian, planet-ubuntu
+
+I've been a bit surprised by the strong positive response to my [previous
+post](http://www.chiark.greenend.org.uk/ucgi/~cjwatson/blosxom/ubuntu/2009-02-27-bug-triage-rants.html).
+People generally seemed to think it was quite non-ranty; maybe I should
+clean the rust off my flamethrower. :-) My hope was that I'd be able to
+persuade people to change some practices, so I guess that's a good thing.
+
+Of course, there are many very smart people doing bug triage very well, and
+I don't want to impugn their fine work. Like its medical namesake, bug
+triage is a skilled discipline. While it's often repetitive, and there are
+lots of people showing up with similar symptoms, a triage nurse can really
+make a difference by spotting urgent cases, cleaning up some of the initial
+blood, and referring the patient quickly to a doctor for attention. Or, if
+a pattern of cases suddenly appears, a triage nurse might be able to warn of
+an incipient epidemic. [Note: I have no medical experience, so please
+excuse me if I'm talking crap here. :-)] The bug triagers who do this well
+are an absolute godsend; especially when they respond to repetitive tasks
+with tremendously useful pieces of automation like
+[bughelper](https://launchpad.net/bughelper). The cases I have trouble with
+are more like somebody showing up untrained, going through everyone in the
+waiting room, and telling each of them that they just need to go home, get
+some rest, and stop complaining so much. Sometimes of course they'll be
+right, but without taking the time to understand the problem they're
+probably going to do more harm than good.
+
+Ian Jackson reminded me that it's worth mentioning the purpose of bug
+reports on free software: namely, **to improve the software**. The GNU
+Project has some [advice to
+maintainers](http://www.gnu.org/prep/maintain/maintain.html#Mail) on this.
+I think sometimes we stray into regarding bug reports more like support
+tickets. In that case it would be appropriate to focus on resolving each
+case as quickly as possible, if necessary by means of a workaround rather
+than by a software change, and only bother the developers when necessary.
+This is the wrong way to look at bug reports, though. The reason that we
+needed to set up a bug triage community in Ubuntu was that we had a
+relatively low developer-to-package ratio and a very high user-to-developer
+ratio, and we were getting a lot of bug reports that weren't fleshed out
+enough for a developer to investigate them without spending a lot of time in
+back-and-forth with the reporter, so a number of people volunteered to take
+care of the initial back-and-forth so that good clear bug reports could be
+handed over to developers. This is all well and good, and indeed I
+encouraged it because I was personally finding myself unable to keep up with
+incoming bugs and actually fix anything at the same time. Somewhere along
+the way, though, some people got the impression that what we wanted was a
+first-line support firewall to try to defend developers from users, which of
+course naturally leads to ideas such as closing wishlist bugs containing
+ideas because obviously those important developers wouldn't want to be
+bothered by them, and closing old bugs because clearly they must just be
+getting in developers' way. Let me be clear about this now: I absolutely
+appreciate help getting bug reports into a state where I can deal with them
+efficiently, but **I do not want to be defended from my users**! I don't
+have a basis from which to state that all developers feel the same way, but
+my guess is that most do.
+
+[Antti-Juhani
+Kaijanaho](http://antti-juhani.kaijanaho.fi/newblog/archives/471) said he'd
+experienced most of these problems in Debian. I hadn't actually intended my
+post to go to Planet Debian - I'd forgotten that the "ubuntu" category on my
+blog goes there too, which generally I see as a feature, but if I'd
+remembered that I would have been a little clearer that I was talking about
+Ubuntu bug triage. If I had been talking about Debian bug triage I'd
+probably have emphasised different things. Nevertheless, it's interesting
+that at least one Debian (and non-Ubuntu) developer had experienced similar
+problems.
+
+[Justin Dugger](http://jldugger.livejournal.com/25994.html) mentions a
+practice of marking duplicate bugs invalid that he has problems with. I
+agree that this is suboptimal and try not to do it myself. That said, this
+is not something I object to to the same extent. Given that the purpose of
+bugs is to improve the software, the real goal is to be able to spend more
+time fixing bugs, not to get bugs into the ideal state when the underlying
+problem has already been solved. If it's a choice between somebody having
+to spend time tracking down the exact duplicate bug number versus fixing
+another bug, I know which I'd take. Obviously, when doing this, it's worth
+apologising that you weren't able to find the original bug number, and
+explaining what the user can do if they believe that you're mistaken
+(particularly if it's a bug that's believed to be fixed); the stock text
+people often use for this doesn't seem informative enough to me.
+
+Sebastien Bacher commented that preferred bug triage practices differ among
+teams: for instance, the Ubuntu desktop team deals with packages that are
+very much to the forefront of users' attention and so get a lot of duplicate
+bugs. Indeed - and bug triagers who are working closely with the desktop
+team on this are almost certainly doing things the way the developers on the
+desktop team prefer, so I have no problem with that. The best advice I can
+give bug triagers is that their ultimate aim is to help developers, and so
+they should figure out which developers they need to work with and **go and
+talk to them**! That way, rather than duplicating work or being
+counterproductive, they can tailor their work to be most effective.
+Everybody wins.
--- /dev/null
+Title: Killer apps: bzr shelve
+Slug: bzr-shelve
+Date: 2006-01-09 16:47:43 +0000
+Tags: bzr, planet-debian, planet-ubuntu
+
+Working on free software has made me fairly revision control
+system-agnostic; I can't afford to get too wedded to any one system because
+as soon as I do somebody will invent something new and I'll have to convert
+again, so I just work with whatever other people on the same project are
+using. Even CVS doesn't make a lot of difference to the way I work as long
+as I'm working online and have cvsps handy. And of course I usually don't
+bother with revision control if I'm just tweaking somebody else's Debian
+source package a bit (in which case I just use debdiff for paranoia).
+
+Using bzr at work, though, I think I just found my killer app in Michael
+Ellerman's [shelve](http://bazaar.canonical.com/BzrShelveExample) plugin.
+My working style generally involves alternating between doing lots and lots
+of stuff in the one working copy and (after testing) going through and
+committing it in logical chunks. This is fine if everything's in separate
+files (most revision control systems let you commit just some files), but if
+several of the chunks are in the one file then I'm reduced to saving diffs
+and manually editing out the bits I don't want to commit yet, which is
+obviously pretty tedious and error-prone.
+
+`bzr shelve` presents each diff hunk in your working copy to you in turn and
+asks you whether you want to keep it. If you say no, that hunk gets
+unapplied and goes into a "shelf", where `bzr unshelve` can later reapply
+it. In the meantime commits act as though the shelved hunks didn't exist.
+This doesn't help if you want to defer only one of two immediately adjacent
+changes that end up in the same hunk, of course, but it vastly reduces the
+scale of the problem.
+
+I suppose it would be easy enough to write a shelve-a-like for any other
+system; it's just that I haven't seen it for any other system yet. If
+working with systems that lack it really starts to annoy me, I may have to
+rip out the guts of shelve and figure out how to make it generic.
--- /dev/null
+Title: Catching up
+Slug: catching-up
+Date: 2010-02-21 20:04:55 +0000
+Tags: man-db, planet-debian, planet-ubuntu
+
+I did a bit of catching up on my Debian backlog over the last week or so.
+Among the things I got round to:
+
+ * I released man-db 2.5.7. This was mostly an "I've been meaning to do
+ this for ages" kind of thing to reduce the bug list a bit, closing ten
+ Debian bugs, but there were a few interesting things in there as well,
+ such as always saving cat pages in UTF-8 and recoding to the user's
+ locale at display time (long overdue), adjusting the search order for
+ localised manual pages by request of quite a few non-native English
+ speakers to prefer a page in the right section over a page in the right
+ language, and a cute gimmick to make things like `man /usr/bin/time`
+ display the appropriate manual page rather than the text of the
+ executable. See the [NEWS
+ file](http://bazaar.launchpad.net/~cjwatson/man-db/trunk/annotate/head%3A/NEWS)
+ for more details.
+ * binfmt-support now [installs cleanly on non-Linux
+ systems](http://bugs.debian.org/565109), even if it doesn't do anything
+ useful yet.
+ * I fixed a couple of [shell](http://bugs.debian.org/256226)
+ [bugs](http://bugs.debian.org/547750) in groff.
+ * halibut now [complies with the Debian Vim
+ policy](http://bugs.debian.org/464821), even though I can't say I
+ entirely agree with it in this case.
+ * I fixed a [really odd build failure in
+ troffcvt](http://lists.debian.org/debian-devel-changes/2010/02/msg02219.html).
+ Yay imake, or something.
+ * All Debian patches to putty are now upstream, or will be once I upload a
+ new snapshot. Thanks to Simon Tatham and Jacob Nevins.
+ * I did a few bits and pieces of packaging cleanup with an eye on my
+ [DDPO](http://qa.debian.org/developer.php) list, and added some watch
+ files where they were missing.
+ * Responded to an offer to take over icoutils maintenance.
+
+So nothing really earth-shaking, and as ever [openssh could use some
+attention](http://lists.debian.org/debian-ssh/2010/01/msg00017.html), but I
+feel a bit better about my backlog now. I do still have a [critical bug in
+makepasswd](http://bugs.debian.org/564559) to fix, and a sponsored upload of
+parrot; those are the next two things on my to-do list.
--- /dev/null
+Title: code_swarm video of Ubuntu uploads
+Slug: code_swarm
+Date: 2009-05-28 20:29:55 +0000
+Modified: 2009-05-28 20:32:12 +0000
+Category: ubuntu
+Tags: ubuntu, planet-debian, planet-ubuntu
+
+Joey Hess posted a
+[draft](http://lists.debian.org/debian-boot/2009/05/msg00265.html) of a
+[code\_swarm](http://code.google.com/p/codeswarm/) video for d-i a couple of
+weeks ago, which reminded me that I've been meaning to do something similar
+for Ubuntu for a while now as it's just about our archive's fifth birthday.
+I have a more or less complete archive of all our -changes mailing lists
+locally (I think I'm missing some of the very early ones, before the end of
+July 2004; let me know if you were one of the very early Canonical employees
+and have a record of these), and with the aid of
+[launchpadlib](https://help.launchpad.net/API/launchpadlib) it's fairly easy
+to map all the e-mail addresses into Launchpad user names, massage out some
+of the more obvious duplicates, and then treat the stream of uploads as if
+it were a stream of commits.
+
+If you haven't seen code\_swarm before, each dot represents an upload, and
+the dots "swarm" around their corresponding committers' names; more active
+committers have larger swarms of dots and brighter names. I assigned a
+colour to each of our archive components (uploads aren't really at the C
+code vs. Python code vs. translations vs. whatever kind of granularity that
+you see in other code\_swarm videos), which mostly means that people who
+predominantly upload to main are in roughly an Ubuntu tan colour, people who
+predominantly upload to universe are coloured bluish, and people with a good
+mixture tend to come out coloured green. If I get a bit more time I may try
+to figure out enough about video editing software to add some captions.
+
+Here's the [video](http://people.ubuntu.com/~cjwatson/ubuntu-uploads.ogv)
+(194 MB).
--- /dev/null
+Title: debconf/cdebconf coinstallability
+Slug: debconf-cdebconf-coinstallable
+Date: 2006-01-27 02:55:06 +0000
+Tags: debconf, planet-debian, planet-ubuntu
+
+[Joey](http://kitenet.net/~joey/blog/) has been
+[campaigning](http://lists.debian.org/debian-devel/2005/08/msg00136.html)
+for a while to get everything in the archive changed to depend on `debconf |
+debconf-2.0` or similar rather than just `debconf`, in order that we can
+start rolling out `cdebconf` as its replacement. Like most jobs that
+involve touching the bulk of the archive, this looks set to take quite a
+while, as the [list of bugs](http://bugs.debian.org/328498) should
+indicate.</p>
+
+Recently it occurred to me that we didn't necessarily have to do it that way
+round. In a bout of late-night hacking while staying awake to look after a
+sick child (he seems mostly OK now, although the rushed trip to the hospital
+earlier was a bit on the nerve-wracking side), I've shuffled things around
+in the cdebconf package so that it no longer has any file conflicts with
+debconf or debconf-doc, and changed the debconf confmodule to fire up the
+cdebconf frontend rather than its own if the `DEBCONF_USE_CDEBCONF`
+environment variable is non-empty. (The details of this may change before
+it actually gets uploaded, as I'd like to get Joey to look it over and
+approve it first.) This allows you to install cdebconf, set that
+environment variable, and play around with cdebconf with relative ease; when
+we come to switch to cdebconf for real, instead of a huge conflicting mess
+that apt will probably have trouble resolving, it'll just be a matter of
+changing a couple of lines in `/usr/share/debconf/confmodule`.</p>
+
+Of course, don't expect cdebconf to be a complete working replacement for
+debconf just yet; if you try using it for a dist-upgrade run it'll fall
+over. Due to its d-i heritage, it doesn't yet load templates automatically;
+that has to be done by hand. Frontend names differ from debconf's, which
+will need some migration code. At the moment it can only handle UTF-8
+templates, which are mandated in the installer but only optional in the rest
+of the system. It doesn't have all of debconf's rich array of database
+modules. I haven't adapted the Perl or Python confmodules yet. The list
+goes on. However, I think we at least stand a chance of getting a handle on
+the problem now.
+
+(I'll post this article to debian-devel once the changes have been reviewed
+and uploaded.)
--- /dev/null
+Title: debhelper statistics, redux
+Slug: debhelper-statistics-redux
+Date: 2010-07-10 23:40:20 +0100
+Modified: 2010-07-10 23:42:45 +0100
+Tags: planet-debian, planet-ubuntu
+
+Apropos of [my previous post]({filename}/debhelper-statistics.md), I see
+that dh has now overtaken CDBS as the most popular rules helper system of
+its kind in Debian unstable, and shows no particular sign of slowing its
+rate of uptake any time soon. The resolution of the graph is such that you
+can't see it yet, but dh drew dead level with CDBS on Thursday, and today
+3836 packages are using dh as opposed to 3823 using CDBS.
+
+
--- /dev/null
+Title: debhelper statistics
+Slug: debhelper-statistics
+Date: 2010-03-03 00:22:19 +0000
+Tags: planet-debian, planet-ubuntu
+
+I don't know if anyone else has been tracking this recently, but a while
+back I got curious about the relative proportions of dh(1) and CDBS in the
+archive, and started running some daily analysis on the Lintian lab.
+Apologies for my poor graphing abilities, but the graph is here
+(occasionally updated):
+
+
+
+Although dh is still a bit behind CDBS, the steady upward trend is quite
+striking - it looks set to break 20% soon, up from under 13% in September -
+compared with CDBS which has been sitting within half a percentage point of
+25% the whole time.
+
+Incidentally, was that an ftpmaster trying to sign his name in the graph
+over Christmas or something? :-)
--- /dev/null
+Title: Desktop automounting pain
+Slug: desktop-automount-pain
+Date: 2008-04-12 07:46:12 +0000
+Category: ubuntu
+Tags: planet-ubuntu
+
+Ubuntu's live CD installer, Ubiquity, needs to suppress desktop automounting
+while it's doing partitioning and generally messing about with mount points,
+otherwise its temporary mount points end up busy on unmount due to some
+smart-arse desktop component that decides to open a window for it.
+
+To date, it employs the following methods, each of which was sufficient at
+the time:
+
+ * Set the `/desktop/gnome/volume_manager/automount_drives` and
+ `/desktop/gnome/volume_manager/automount_media` gconf keys to `false`.
+ * Tell `kded` to unload its `medianotifier` module, and load it again just
+ before the installer exits.
+ * Set the `/apps/nautilus/desktop/volumes_visible` gconf key to `false`.
+ * Set the `AutomountDrives` and `AutomountMedia` keys in
+ `$HOME/.config/Thunar/volmanrc` to `FALSE`.
+ * Set the `/apps/nautilus/preferences/media_automount` and
+ `/apps/nautilus/preferences/media_automount_open` gconf keys to `false`.
+ * The entire installer is run under `hal-lock --interface
+ org.freedesktop.Hal.Device.Storage --exclusive`.
+ * Set the `/apps/nautilus/preferences/media_autorun_never` gconf key to
+ `true` ([experimental, but apparently now required since nautilus uses
+ the gio volume
+ monitor](https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/210620)).
+
+This is getting **ridiculous**. Dear desktop implementors: please pick a
+configuration mechanism and stick to it, and provide backward compatibility
+if you can't. This is not a rocket-science concept.
+
+I rather liked the `hal-lock` mechanism; it was simple and involved minimal
+fuss. I had hoped that it might end up as a standard, but I guess that
+would be too easy.
--- /dev/null
+Title: GRUB 2 boot problems
+Slug: grub2-boot-problems
+Date: 2010-06-21 11:52:10 +0100
+Category: grub
+Tags: grub, planet-debian, planet-ubuntu
+
+(This is partly a repost of material I've posted to bug reports and to
+debian-release, put together with some more detail for a wider audience.)
+
+You could be forgiven for looking at the RC bug activity on
+[grub2](http://bugs.debian.org/src:grub2) over the last couple of days and
+thinking that it's all gone to hell in a handbasket with recent uploads. In
+fact, aside from an interesting case which turned out to be due to botched
+handling of the GRUB Legacy to GRUB 2 chainloading setup (which prompted me
+to fix three other RC bugs along the way), all the recent problems people
+have been having have been duplicates of one of these bugs which have
+existed essentially forever:
+
+ * [#554790 - grub-pc/install\_devices uses unstable device
+ names](http://bugs.debian.org/554790)
+ * [#583271 - device.map uses unstable device
+ names](http://bugs.debian.org/583271)
+
+When GRUB boots, its boot sector first loads its "core image", which is
+usually embedded in the gap between the boot sector and the first partition
+on the same disk as the boot sector. This core image then figures out where
+to find /boot/grub, and loads grub.cfg from it as well as more GRUB modules.
+
+The thing that tends to go wrong here is that the core image must be from
+the same version of GRUB as any modules it loads. /boot/grub/*.mod are
+updated only by grub-install, so this normally works OK. However, for
+various reasons (deliberate or accidental) some people install GRUB to
+multiple disks. In this case, grub-install might update /boot/grub/*.mod
+along with the core image on one disk, but your BIOS might actually be
+booting from a different disk. The effect of this will be that you'll have
+an old core image and new modules, which will probably blow up in any number
+of possible ways. Quite often, this problem lies dormant for a while
+because GRUB happens not to change in a way that causes incompatibility
+between the core image and modules, but then we get massive spikes of bug
+reports any time the interface does change. Since these bugs sometimes bite
+people upgrading from testing to unstable, they get interpreted as
+regressions from the version in testing even though that isn't strictly true
+(but it tends not to be very productive to argue this line; after all,
+people's computers suddenly don't boot!). Any problem that causes the core
+image to be installed to a disk other than the one actually being booted
+from, or not to be installed at all, will show up this way sooner or later.
+
+On 2010-06-10, there was a substantial upstream change to the handling of
+list iterators (to reduce core image size and make code clearer and faster)
+which introduced an incompatibility between old core images and newer
+modules. This caused a bunch of dormant problems to flare up again, and so
+there was a flood of reports of booting problems with 1.98+20100614-1 and
+newer, often described as "the unaligned pointer bug" due to how it happened
+to manifest this time round. In previous cases, GRUB reported undefined
+symbols on boot, but it's all essentially the same problem even though there
+are different symptoms.
+
+The confusing bit when handling bug reports is that not only are there
+different symptoms with the same cause, but there are also multiple causes
+for the same symptom! This takes a certain amount of untangling, especially
+when lots of people have thought "ooh, that bug looks a bit like mine" and
+jumped in with their own comments. Working through this was a worthwhile
+exercise, as it came up with an entirely new cause for a problem I thought
+was fairly well-understood (thanks to debugging assistance from Sedat
+Dilek). If you had set up GRUB 2 to be automatically chainloaded from GRUB
+Legacy (which happens automatically on upgrade from the latter to the
+former), never got round to running `upgrade-from-grub-legacy` once you
+confirmed it worked, and then later ran `grub-install` by hand for one
+reason or another, then the core image you installed by hand would never be
+updated and would eventually [fall over](http://bugs.debian.org/586143) the
+next time the core/modules interface changed. Fixing future cases of this
+was easy enough, but fixing existing cases involved figuring out how to
+detect whether an installed GRUB boot sector came from GRUB Legacy or GRUB
+2, which isn't as easy as you might think. Fortunately, it turns out that
+there are a limited number of jump offsets that have ever been used in the
+second byte of the boot sector, and none of the GRUB 2 values clash with the
+only value ever used in GRUB Legacy; so, if you still have
+`/boot/grub/stage2` et al on upgrade, we scan all disks for a GRUB 2 boot
+sector, and if we find one then we offer to complete the upgrade to GRUB 2.
+
+Unless anything new shows up, that just leaves the problems that were
+already understood. Today, I posted a [patch to generate stable device
+names in device.map by
+default](http://lists.gnu.org/archive/html/grub-devel/2010-06/msg00118.html).
+If this is accepted, then we can do something or other to fix up device.map
+on upgrade, switch over to `/dev/disk/by-id` names in
+`grub-pc/install_devices` at the same time, and that should take care of the
+vast majority of this kind of upgrade bug. I think at that point it should
+be feasible to get a new version into testing, and we should be down from 18
+RC bugs towards the end of last month to around 6. We can then start
+attacking things like the lack of support for mdadm 1.x metadata.
+
+Since my [last blog entry on GRUB 2]({filename}/hacking-on-grub2.md),
+improvements have included:
+
+ * Substantial work on `info grub`, with, among other things, new sections
+ on `/etc/default/grub` and on configuring authentication.
+ * A workaround for GRUB's inability to probe dm-crypt devices, thanks to
+ Marc Haber.
+ * Several build fixes for architectures I wasn't testing, and a fix for
+ broken nested partition handling on Debian GNU/kFreeBSD. I'm now testing
+ GNU/kFreeBSD locally.
+ * Rather less cruft in `fs.lst`, `partmap.lst`, and `video.lst`, which
+ should speed up booting a bit by e.g. avoiding unnecessary filesystem
+ probing.
+ * `upgrade-from-grub-legacy` actually now installs GRUB 2 to the boot
+ sector (!).
+ * Ask for confirmation if `grub-pc/install_devices` is left empty.
+
+The next upstream snapshot will bring several improvements to EFI video
+support, mainly thanks to Vladimir Serbinenko. I've been working on making
+`grub-install` actually work on UEFI systems as one of my goals for the next
+Ubuntu release, and I hope to get this landed in the not-too-distant future.
--- /dev/null
+Title: GRUB 2: With luck ...
+Slug: grub2-with-luck
+Date: 2010-07-02 22:27:35 +0100
+Category: grub
+Tags: grub, planet-debian, planet-ubuntu
+
+... this version, or something not too far away from it, might actually
+stand a chance of getting into testing.
+
+I've just uploaded grub2 1.98+20100702-1. The most significant set of
+changes in this release is that it switches `/boot/grub/device.map` and the
+`grub-pc/install_devices` debconf question over to stable device names under
+`/dev/disk/by-id` (on Linux kernels). The code implementing this is
+reasonably careful, and it should make it quite difficult for people to
+accidentally fail to upgrade their installed GRUB core image; I explained
+the problems that tends to cause in the [previous post in this
+series]({filename}/grub2-boot-problems.md). There will probably be a few
+small glitches I need to clear up, but I've given this much more extensive
+testing than usual so I hope I won't break too many people's computers
+(again).
+
+I did this work first in Ubuntu as one of my major goals for 10.04 LTS,
+which exposed a few problems that I wanted to fix before inflicting it on
+Debian as well (fixes for those are now under testing for 10.04.1). Most
+significantly, I felt it was necessary to start offering partitions in the
+select list for `grub-pc/install_devices`, but I went a bit overboard and
+offered all partitions in a giant list. This seemed like a good idea at the
+time, but it tended to confuse people into just selecting everything in the
+list, which in particular tended to make Windows unbootable! So I dialled
+that back a bit, and in the version I just merged it will only offer the
+partitions mounted on `/`, `/boot`, and `/boot/grub` (de-duplicating if
+necessary). This seems like a reasonable compromise between confusing
+people too much and forcing them to install only to MBRs.
+
+My next priority will be making whatever fixes are necessary to get this
+version into testing, since the problems with `/dev/mapper` symlinks in
+testing aren't getting any less urgent, and this is finally a version that
+shouldn't break for most people due to the kernel's switch to libata. I
+expect that I'll try to get mdadm 1.x metadata sorted out immediately after
+that.
+
+Other improvements since my last entry have included:
+
+ * Further documentation work. Thanks to Vladimir Serbinenko (and to Jordan
+ Uggla for hosting it temporarily), there's now an [HTML version of the
+ GRUB manual from trunk](http://www.gnu.org/software/grub/manual/) online,
+ which includes new sections on embedded configuration files, the various
+ GRUB image files, `device.map`, and (shortly) a summary of changes from
+ GRUB Legacy.
+ * Video improvements: among other things, UEFI systems whose firmware uses
+ the Graphics Output Protocol should now work rather better, and GRUB now
+ includes specific support for some cards often used with minimal firmware
+ support under emulation.
+ * A fix to handle large memory maps exposed by some UEFI firmware.
+ * Automatic configuration support for Fedora 13. You may need [os-prober
+ 1.39](http://packages.qa.debian.org/o/os-prober/news/20100628T171748Z.html)
+ from unstable as well.
+ * Automatic configuration support for Linux on Xen.
+ * Skip LVM snapshots rather than failing when they're present.
--- /dev/null
+Title: Google Summer of Code project started (Debian)
+Slug: gsoc-d-i-hurd-started
+Date: 2006-05-26 17:23:00 +0000
+Modified: 2006-05-26 20:11:52 +0000
+Category: summerofcode
+Tags: summerofcode-2006, planet-debian
+
+I'm mentoring [Matheus Morais](http://xsunblog.blogspot.com/) in the [Google
+Summer of Code](http://code.google.com/soc/), porting d-i to the Hurd. We've
+exchanged a few mails and he has in hand all the preliminary (but not yet
+functional; wouldn't want to make it too easy :-)) patches I've put together
+in the past. I think I should be reasonably well-placed to judge his
+progress.
+
+Best of luck, Matheus!
-Google Summer of Code project started (Ubuntu)
+Title: Google Summer of Code project started (Ubuntu)
+Slug: gsoc-ubiquity-migration-started
+Date: 2006-05-26 20:13:13 +0000
+Category: summerofcode
+Tags: summerofcode-2006, planet-ubuntu
-<p>More on the <a href="http://code.google.com/soc/">Google Summer of
-Code</a>: as well as the
-<a href="http://riva.ucam.org/~cjwatson/blog/summerofcode/2006/2006-05-25-gsoc-d-i-hurd-started.html">
-project I'm mentoring for Debian</a>, I'm mentoring Evan Dandrea (no blog
-yet?), writing a <a href="https://wiki.ubuntu.com/MigrationAssistance">
-migration assistant</a> for Ubiquity.</p>
+More on the [Google Summer of Code](http://code.google.com/soc/): as well as
+the [project I'm mentoring for Debian]({filename}/gsoc-d-i-hurd-started.md),
+I'm mentoring Evan Dandrea (no blog yet?), writing a [migration
+assistant](https://wiki.ubuntu.com/MigrationAssistance) for Ubiquity.
-<p>I haven't talked much about Ubiquity, mostly because I've been far too
-busy writing it. Ubuntu has needed an installer for its live CD for a while,
+I haven't talked much about Ubiquity, mostly because I've been far too busy
+writing it. Ubuntu has needed an installer for its live CD for a while,
partly because, well, loads of users want it, and partly because it will cut
Canonical's costs quite a lot if we only have to send out half the number of
CDs in shipit (apparently single-CD packaging is significantly cheaper than
maintaining two installer implementations from the ground up (live CDs are
nice, but they don't cut it for everyone). So when I was given the task of
doing a shiny live CD installer with a custom-designed UI, while I started
-with a more-or-less from-scratch implementation put together by <a
-href="http://www.guadalinex.org/">Guadalinex</a> (thanks!), I fairly quickly
+with a more-or-less from-scratch implementation put together by
+[Guadalinex](http://www.guadalinex.org/) (thanks!), I fairly quickly
diverged from that and morphed it into something that uses d-i code for as
much of the backend operation and logic as it sensibly can. I'd call it a
sort of debconf frontend except that its design is almost opposite to how a
particular question names. This had a lot of advantages in terms of being
fairly quick to write, although in the long term I think I might prefer
something closer to cdebconf plugins for the job; we'll see how things turn
-out.</p>
+out.
-<p>Evan wanted to write an extension to this to automatically migrate
-settings and documents from an existing Windows installation, which I think
-would be an absolutely excellent thing to have: automatic migration is a
-real killer feature in an installer. In fact, d-i already has a start at
-this, namely os-prober, which in conjunction with some bootloader installer
-code magically sets up boot menu entries for other operating systems on your
+Evan wanted to write an extension to this to automatically migrate settings
+and documents from an existing Windows installation, which I think would be
+an absolutely excellent thing to have: automatic migration is a real killer
+feature in an installer. In fact, d-i already has a start at this, namely
+os-prober, which in conjunction with some bootloader installer code
+magically sets up boot menu entries for other operating systems on your
disk. So, I suggested to Evan that he might want to put most of the clever
logic in a udeb, so that it can be used in d-i too, and to my relief he
seemed quite enthused by the idea. He's starting on the preliminary work now
-and I look forward to seeing his progress.</p>
+and I look forward to seeing his progress.
-<p>Best of luck, Evan!</p>
+Best of luck, Evan!
--- /dev/null
+Title: Hacking on grub2
+Slug: hacking-on-grub2
+Date: 2010-06-04 22:57:07 +0100
+Modified: 2010-06-04 23:00:54 +0100
+Category: grub
+Tags: grub, planet-debian, planet-ubuntu
+
+Various people observed in a [long thread on
+debian-devel](http://lists.debian.org/debian-devel/2010/05/msg00769.html)
+that the grub2 package was in a bit of a mess in terms of its
+release-critical bug count, and [Jordi](http://oskuro.net/blog) and
+[Stefano](http://upsilon.cc/~zack/blog/planet-debian/) both got in touch
+with me directly to gently point out that I probably ought to be doing
+something about it as one of the co-maintainers.
+
+Actually, I don't think grub2 was in quite as bad a state as its 18 RC bugs
+suggested. Of course every boot loader failure is critical to the person
+affected by it, not to mention that GRUB 2 offers more complex functionality
+than any other boot loader (e.g. LVM and RAID), and so it tends to
+accumulate RC bugs at rather a high rate. That said, we'd been neglecting
+its bug list for some time; [Robert](http://robertmh.wordpress.com/) and
+Felix have both been taking some time off, Jordi mostly only cared about
+PowerPC and can't do that any more due to hardware failure, and I hadn't
+been able to pick up the slack.
+
+Most of my projects at [work](http://www.ubuntu.com/) for the next while
+involve GRUB in one way or another, so I decided it was a perfectly
+reasonable use of work time to do something about this; I was going to need
+fully up-to-date snapshots anyway, and practically all the Debian grub2 bugs
+affect Ubuntu too. Thus, with the exception of some other little things
+like releasing the first Maverick alpha, I've spent pretty much the last
+week and a half solidly trying to get the grub2 package back into shape,
+with four uploads so far.
+
+The RC issues that remain are:
+
+ * `upgrade-from-grub-legacy` problems
+ ([#547944](http://bugs.debian.org/547944),
+ [#550477](http://bugs.debian.org/550477)):
+
+ I think this has just been traditionally undertested. I'm setting up a
+ KVM image now with GRUB Legacy which I can snapshot just before and
+ after running `upgrade-from-grub-legacy`, and I should be able to unpick
+ the bugs this way.
+
+ * LVM snapshots break GRUB's LVM module
+ ([#574863](http://bugs.debian.org/574863)):
+
+ [Sean](http://www.seanius.net/feeds/planet-debian/) has been working on
+ this and seems to be nearly there. Yay.
+
+ * RAID metadata version 1.x not supported
+ ([#492897](http://bugs.debian.org/492897)):
+
+ This became rather more of an issue recently since `mdadm` switched its
+ default from the old 0.90 format which GRUB understood. Felix put
+ together a branch implementing the hard parts of this a while back, and
+ I've been trying to finish it off. The hard bit is dealing with device
+ naming, especially as the new-format and rather more useful names under
+ `/dev/md/` don't show up during
+ [d-i](http://www.debian.org/devel/debian-installer) after creating RAID
+ volumes; I think this is because we always create them as `/dev/md0`
+ etc. It's looking tractable, though.
+
+ * Another odd problem probing RAID
+ ([#548648](http://bugs.debian.org/548648)):
+
+ Not sure about this one, and I'll need to work with Josip on it as soon
+ as I get a chance.
+
+ * Stable device naming [#554790](http://bugs.debian.org/554790)) and
+ consequential problems due to `grub-install` not being properly run
+ ([#557425](http://bugs.debian.org/557425) and many other sub-RC bugs):
+
+ Ubuntu's been carrying a patch to rearrange device presentation in the
+ postinst, which Robert OKed in principle ages ago and so I've been
+ intending to merge it for a while, but there are a few known problems
+ with it that I need to fix first. One known unfixable problem is that
+ it will have to ask some people which devices they want GRUB to be
+ installed on, even if they'd answered that question before: this will be
+ one-time, and it's because it recorded the answer using unstable device
+ names and so has in some sense forgotten. Simple cases (e.g.
+ single-disk) can be handled without needing to ask again, though.
+
+ * Alignment errors on SPARC ([#560823](http://bugs.debian.org/560823)):
+
+ I have no idea what's going on here, I'm afraid. I'll try to trace it,
+ but may have to downgrade it at some point since after all we don't
+ install GRUB by default on SPARC yet.
+
+ * Fonts not shown in gfxmenu ([#564844](http://bugs.debian.org/564844)):
+
+ Apparently fixed upstream, but I couldn't find the responsible commit so
+ I want to make sure I can get gfxmenu working before closing this.
+
+ * Sensitivity to out-of-date `device.map` files
+ ([#575076](http://bugs.debian.org/575076) and other sub-RC bugs):
+
+ We're trying to get rid of `device.map` in general. It was fine in the
+ 1990s but it's hopeless now. Unfortunately there are still a small
+ number of problems with running entirely without one, and one of my
+ patches to help is controversial upstream, so we probably won't get to
+ that for squeeze. In the meantime we'll probably just need some extra
+ sanity-checking and robustness in the event that there's an incorrect or
+ out-of-date `device.map` lying around, which we may just be able to do
+ in the maintainer scripts or something if necessary.
+
+ * Seriously weird failures to load initramfs
+ ([#582342](http://bugs.debian.org/582342)):
+
+ If anyone can produce a reproduction recipe for this, that would really
+ help me out. There are too many reports to discount as user error, but
+ I haven't seen this myself yet.
+
+ * Build failure on sparc (unfiled):
+
+ We've been discussing this upstream, but for the time being I'm just
+ going to stop building `grub-emu` on sparc as a workaround.
+
+If we can fix that lot, or even just the ones that are reasonably
+well-understood, I think we'll be in reasonable shape. I'd also like to
+make `grub-mkconfig` a bit more robust in the event that the root filesystem
+isn't one that GRUB understands ([#561855](http://bugs.debian.org/561855),
+[#562672](http://bugs.debian.org/562672)), and I'd quite like to write some
+more documentation.
+
+On the upside, progress has been good. We have multiple terminal support
+thanks to a new upstream snapshot
+([#506707](http://bugs.debian.org/506707)), `update-grub` runs much faster
+([#508834](http://bugs.debian.org/508834),
+[#574088](http://bugs.debian.org/574088)), we have DM-RAID support with a
+following wind ([#579919](http://bugs.debian.org/579919)), the new scheme
+with symlinks under `/dev/mapper/` works
+([#550704](http://bugs.debian.org/550704)), we have basic support for btrfs
+`/` as long as you have something GRUB understands properly on `/boot`
+([#540786](http://bugs.debian.org/540786)), we have full info documentation
+covering all the user-adjustable settings in `/etc/default/grub`, and a host
+of other smaller fixes. I'm hoping we can keep this up.
+
+If you'd like to help, contact me, especially if there's something
+particular that isn't being handled that you think you could work on. GRUB
+2 is actually quite a pleasant codebase to work on once you get used to its
+layout; it's certainly much easier to fix bugs in than GRUB Legacy ever was,
+as far as I'm concerned. Thanks to tools like `grub-probe` and
+`grub-fstest`, it's very often possible to fix problems without needing to
+reboot for anything other than a final sanity check (although KVM certainly
+helps), and you can often debug very substantial bits of the boot loader -
+the bits that actually go wrong - using standard tools such as `strace` and
+`gdb`. Upstream is helpful and I've been able to get many of the problems
+above fixed directly there. If you have a sound knowledge of C and a decent
+level of understanding of the environment a boot loader needs to operate in
+- or for that matter specialist knowledge of interesting device types - then
+you should be able to find something to do.
--- /dev/null
+Title: Hello!
+Slug: hello
+Date: 2006-01-03 13:11:31
+Tags: planet-debian, planet-ubuntu
+
+New year, new blog. I've had a
+[LiveJournal](http://www.livejournal.com/users/cjwatson/) for a while, but
+don't write very much in it, and many of its readers wouldn't be interested
+in me talking about Debian and such anyway. I think the best solution is
+for me to keep technical posts here.
--- /dev/null
+Title: Keysigning bits
+Slug: keysigning-bits
+Date: 2009-07-31 11:31:44 +0000
+Tags: keysigning, planet-debian, planet-ubuntu
+
+If you're generating one of these shiny new RSA keys, do please remember to
+[generate an encryption subkey
+too](http://ekaia.org/blog/2009/05/10/creating-new-gpgkey/) if you expect
+people to sign it - at least your more obscure UIDs. I'm not going to mail
+unencrypted signatures around unless I have some out-of-band knowledge that
+the e-mail address actually belongs to the person I met.
+
+I generated a new 4096-bit RSA key myself at DebConf (baa!), and have just
+published a [key transition
+document](http://www.chiark.greenend.org.uk/~cjwatson/key-transition).
+Please consider signing my new key if you signed my old one.
--- /dev/null
+Title: Keysigning public service announcement
+Slug: keysigning-psa
+Date: 2007-07-04 17:45:39 +0000
+Tags: keysigning, planet-debian, planet-ubuntu
+
+If your key has so many UIDs and such a combinatorially exploded number of
+signatures on it that it takes `gpg` minutes just to start up in
+`--edit-key` mode, then I probably won't bother signing it. HTH, HAND.
--- /dev/null
+Title: libpipeline 1.1.0 released
+Slug: libpipeline-1.1.0-released
+Date: 2010-12-11 15:47:39 +0000
+Modified: 2010-12-11 15:50:23 +0000
+Category: libpipeline
+Tags: libpipeline, planet-debian, planet-ubuntu
+
+I've released [libpipeline 1.1.0](http://libpipeline.nongnu.org/), and
+uploaded it to Debian unstable. The changes are mostly just to add a few
+occasionally useful interfaces:
+
+ * Add `pipecmd_exec` to execute a single command, replacing the current
+ process; this is analogous to `execvp`.
+ * Add `pipecmd_clearenv` to clear a command's environment; this is
+ analogous to `clearenv`.
+ * Add `pipecmd_get_nargs` to get the number of arguments to a command.
+
+The shared library actually ends up being a few kilobytes smaller on Debian
+than 1.0.0, probably because I tweaked the set of Gnulib modules I'm using.
--- /dev/null
+Title: libpipeline 1.2.1 released
+Slug: libpipeline-1.2.1-released
+Date: 2012-03-02 21:49:10 +0000
+Category: libpipeline
+Tags: libpipeline, planet-debian, planet-ubuntu
+
+I've released [libpipeline 1.2.1](http://libpipeline.nongnu.org/), and
+uploaded it to Debian unstable. This is a bug-fix release:
+
+ * Retry reads and writes on `EINTR`.
+ * Fix opening of output files requested by `pipeline_want_outfile`; these
+ are now created if they do not already exist, and truncated if they do.
+ * `<pipeline.h>` is now wrapped in `extern "C"` when used in a C++
+ compilation unit.
--- /dev/null
+Title: libpipeline 1.0.0 released
+Slug: libpipeline-released
+Date: 2010-10-29 21:23:26 +0100
+Category: libpipeline
+Tags: libpipeline, man-db, planet-debian, planet-ubuntu
+
+In my [previous post]({filename}/pipeline-library.md), I described the
+pipeline library from man-db and asked whether people were interested in a
+standalone release of it. Several people expressed interest, and so I've
+now released [libpipeline](http://libpipeline.nongnu.org/) version 1.0.0.
+It's in the Debian NEW queue, and [my
+PPA](https://launchpad.net/~cjwatson/+archive/ppa) contains packages of it
+for Ubuntu lucid and maverick.
+
+I gave a lightning talk on this at UDS in Orlando, and my
+[slides](http://libpipeline.nongnu.org/libpipeline-lightning-talk.odp) are
+available. I hope there'll be a video at some point which I can link to.
+
+Thanks to Scott James Remnant for code review (some time back), Ian Jackson
+for an extensive design review, and Kees Cook and Matthias Klose for helpful
+conversations.
--- /dev/null
+Title: man-db 2.6.0
+Slug: man-db-2.6.0
+Date: 2011-04-09 20:45:17 +0100
+Category: man-db
+Tags: man-db, planet-debian, planet-ubuntu
+
+I've released man-db 2.6.0
+([announcement](http://lists.nongnu.org/archive/html/man-db-announce/2011-04/msg00000.html),
+[NEWS](http://bazaar.launchpad.net/~cjwatson/man-db/trunk/view/1344/NEWS),
+[ChangeLog](http://bazaar.launchpad.net/~cjwatson/man-db/trunk/view/1344/ChangeLog)),
+and uploaded it to Debian unstable. Ubuntu is rapidly approaching beta
+freeze so I'm not going to try to cram this into 11.04; it'll be in 11.10.
--- /dev/null
+Title: man-db: 'man -K'
+Slug: man-db-K
+Date: 2009-07-14 15:36:45 +0000
+Category: man-db
+Tags: man-db, planet-debian, planet-ubuntu
+
+I recently implemented `man -K` (full-text search over all manual pages) in
+[man-db](http://man-db.nongnu.org/). This was inspired by a similar feature
+in Federico Lucifredi's [man](http://primates.ximian.com/~flucifredi/man/)
+package (formerly maintained by Andries Brouwer). I think I did a much
+better job of it, though. The man package just forks grep for every manual
+page; man-db takes advantage of the pipeline library I wrote for it a while
+back and does it entirely in-process (decompression requires a fork but no
+exec, while the man package has to exec gunzip as well).
+
+The upshot is that, with a hot cache, man-db takes around 40 seconds to
+search all manual pages on my laptop; the man package (also with a hot
+cache) takes around five minutes, and interactive performance goes down the
+drain while it's doing it since it's spawning subprocesses like crazy. If I
+limit to a single section, the disparity is closer to 3x than 10x, but it's
+still very noticeable. It's interesting how much good libraries can do to
+help guide efficient approaches to problems.
+
+Of course, a proper full-text search engine would be much better still, but
+that's a project for some other time ...
--- /dev/null
+Title: Encodings in man-db
+Slug: man-db-encodings
+Date: 2007-09-17 07:28:20 +0000
+Category: man-db
+Tags: man-db, planet-debian, planet-ubuntu
+
+I've spent some quality upstream time lately with man-db. Specifically,
+I've been upgrading its locale support. I recently published a pre-release,
+[man-db
+2.5.0-pre2](http://people.debian.org/~cjwatson/man-db/man-db-2.5.0-pre2.tar.gz)
+mainly for translators, but other people may be interested in having a look
+at it as well. I hope to release 2.5.0 quite soon so that all of this can
+land in Debian.
+
+Firstly, man-db now supports creating and using databases for per-locale
+hierarchies of manual pages, not just English. This means that [apropos and
+whatis can now display information about localised manual
+pages](http://bugs.debian.org/29448).
+
+Secondly, I've been working on the transition to UTF-8 manual pages. Now,
+modulo some hacks, groff can't yet deal with Unicode input; some possible
+input characters are reserved for its internal use which makes full 32-bit
+input difficult to do properly until that's fixed. However, with a few
+exceptions, manual pages generally just need the subset of Unicode that
+corresponds to their language's usual legacy character set, so for now it's
+good enough to just recode on the fly from UTF-8 to some appropriate 8-bit
+character set and use groff's support for that.
+
+man-db has actually supported doing this kind of thing for a while, but it's
+been difficult to use since it only applies to `/usr/share/man/ll_CC.UTF-8/`
+directories, while manual pages usually aren't country-specific. So, man-db
+2.5.0 supports using `/usr/share/man/ll.UTF-8/` instead, which is a bit more
+appropriate. Also, following a [discussion with Adam
+Borowski](http://lists.debian.org/debian-mentors/2007/09/msg00245.html),
+man-db can now try decoding manual pages as UTF-8 and fall back to 8-bit
+encodings even in directories without an explicit encoding tag; if this
+fails for some reason, you can put a `'\" -*- coding: UTF-8 -*-` line at the
+top of the page.
+
+I'm still debating whether Debian policy should recommend installing UTF-8
+manual pages in `/usr/share/man/ll.UTF-8/` or just in `/usr/share/man/ll/`.
+Initially I was very strongly in favour of an encoding declaration, but now
+that man-db can do a pretty good job of guesswork I'm coming round to Adam
+Borowski's position that people should be able to forget about character
+sets with UTF-8. Opinions here would be welcome. One thing I haven't moved
+on is that any design that assumes that the encoding of manual pages on the
+filesystem has anything to do with the user's locale is demonstrably
+incorrect and broken; I'm not going to use `LC_CTYPE` for anything except
+output. However, maybe "UTF-8 or the usual legacy encoding provided that
+the latter is not typically confused for the former" is a good enough
+specification, and that still has the desirable property of not requiring a
+flag day. I'll try to come down from the fence before unleashing this code
+on the world.
--- /dev/null
+Title: man-db on Fedora
+Slug: man-db-on-fedora
+Date: 2010-12-02 14:06:58 +0000
+Modified: 2010-12-02 14:09:22 +0000
+Category: man-db
+Tags: man-db, planet-debian, planet-ubuntu
+
+I just found out by chance that [Fedora](http://fedoraproject.org/) 14
+switched from their old man package to [man-db](http://man-db.nongnu.org/).
+This is great news: it should now be the beginning of the end of the
+divergence of man implementations that happened way back in the mid-1990s,
+when two different people took John W. Eaton's man package and developed it
+in different directions without being aware of each other's existence. For
+a while it looked as though man-db was stuck on just the Debian family and
+openSUSE, but a number of distributions have switched over in the last few
+years. As of now, the only remaining major distribution not using man-db is
+Gentoo, and they have a [bug for
+switching](http://bugs.gentoo.org/show_bug.cgi?id=284822) which I think
+should be unblocked fairly soon.
+
+In some ways man-db's package name didn't help it; people thought that the
+main difference was that man-db had a database backend stuck around apropos.
+These days, the database is one of the least important parts of man-db as
+far as I'm concerned. Other ways in which it's very significantly superior
+to anything man could do without years of equivalent effort include correct
+encoding support, robust child process handling, and use of more modern
+development facilities (dear catgets: you belong to a previous millennium,
+so please go away). I'm glad that Fedora has recognised this.
-Moving conffiles between packages, redux
+Title: Moving conffiles between packages, redux
+Slug: moving-conffiles
+Date: 2006-12-23 23:37:08 +0000
+Tags: dpkg, openssh, planet-debian, planet-ubuntu
I spent far too much of today cleaning up an upgrade bug to do with
conffiles, which I suspect also affects other packages that have attempted
to work around dpkg conffile prompts when moving conffiles between packages.
If you maintain such a package, please review your code to make sure that it
works properly when upgrading both with sarge's dpkg and with etch's dpkg.
-See <a href="http://lists.debian.org/debian-devel/2006/12/msg00647.html">my
-debian-devel post</a> for a full description.
+See [my debian-devel
+post](http://lists.debian.org/debian-devel/2006/12/msg00647.html)
+> for a full description.
--- /dev/null
+Title: Moving on, but not too far
+Slug: moving-on-but-not-too-far
+Date: 2014-10-26 18:54:34 -0400
+Category: ubuntu
+Tags: launchpad, ubuntu, planet-debian, planet-ubuntu
+
+The [Ubuntu Code of
+Conduct](http://www.ubuntu.com/about/about-ubuntu/conduct) says:
+
+> **Step down considerately**: When somebody leaves or disengages from the
+> project, we ask that they do so in a way that minimises disruption to the
+> project. They should tell people they are leaving and take the proper
+> steps to ensure that others can pick up where they left off.
+
+I've been working on Ubuntu for over ten years now, almost right from the
+very start; I'm Canonical's employee #17 due to working out a notice period
+in my previous job, but I was one of the founding group of developers. I
+occasionally tell the story that Mark originally hired me mainly to work on
+what later became Launchpad Bugs due to my experience maintaining the Debian
+bug tracking system, but then not long afterwards Jeff Waugh got in touch
+and said "hey Colin, would you mind just sorting out some installable CD
+images for us?". This is where you imagine one of those movie time-lapse
+clocks ... At some point it became fairly clear that I was working on
+Ubuntu, and the bug system work fell to other people. Then, when Matt
+Zimmerman could no longer manage the entire Ubuntu team in Canonical by
+himself, Scott James Remnant and I stepped up to help him out. I did that
+for a couple of years, starting the Foundations team in the process. As the
+team grew I found that my interests really lay in hands-on development
+rather than in management, so I switched over to being the technical lead
+for Foundations, and have made my home there ever since. Over the years
+this has given me the opportunity to do all sorts of things, particularly
+working on our installers and on the GRUB boot loader, leading the
+development work on many of our archive maintenance tools, instituting the
++1 maintenance effort and proposed-migration, and developing the Click
+package manager, and I've had the great pleasure of working with many
+exceptionally talented people.
+
+However. In recent months I've been feeling a general sense of malaise and
+what I've come to recognise with hindsight as the symptoms of approaching
+burnout. I've been working long hours for a long time, and while I can draw
+on a lot of experience by now, it's been getting harder to summon the
+enthusiasm and creativity to go with that. I have a wonderful wife, amazing
+children, and lovely friends, and I want to be able to spend a bit more time
+with them. After ten years doing the same kinds of things, I've accreted
+history with and responsibility for a lot of projects. One of the things I
+always loved about Foundations was that it's a broad church, covering a wide
+range of software and with a correspondingly wide range of opportunities;
+but, over time, this has made it difficult for me to focus on things that
+are important because there are so many areas where I might be called upon
+to help. I thought about simply stepping down from the technical lead
+position and remaining in the same team, but I decided that that wouldn't
+make enough of a difference to what matters to me. I need a clean break and
+an opportunity to reset my habits before I burn out for real.
+
+One of the things that has consistently held my interest through all of this
+has been making sure that the infrastructure for Ubuntu keeps running
+reliably and that other developers can work efficiently. As part of this,
+I've been able to do [a lot of
+work](https://dev.launchpad.net/Contributions#colin_watson) over the years
+on [Launchpad](https://launchpad.net/) where it was a good fit with my
+remit: this has included significant performance improvements to archive
+publishing, moving most archive administration operations from
+excessively-privileged command-line operations to the webservice, making
+build cancellation reliable across the board, and moving live filesystem
+building from an unscalable ad-hoc collection of machines into the Launchpad
+build farm. The Launchpad development team has generally welcomed help with
+open arms, and in fact I joined the [~launchpad
+team](https://launchpad.net/~launchpad) last year.
+
+So, the logical next step for me is to make this informal involvement
+permanent. As such, at the end of this year I will be moving from Ubuntu
+Foundations to the Launchpad engineering team.
+
+This doesn't mean me leaving Ubuntu. Within Canonical, Launchpad
+development is currently organised under the Continuous Integration team,
+which is part of Ubuntu Engineering. I'll still be around in more or less
+the usual places and available for people to ask me questions. But I will
+in general be trying to reduce my involvement in Ubuntu proper to things
+that are closely related to the operation of Launchpad, and a small number
+of low-effort things that I'm interested enough in to find free time for
+them. I still need to sort out a lot of details, but it'll very likely
+involve me handing over project leadership of Click, drastically reducing my
+involvement in the installer, and looking for at least some help with boot
+loader work, among others. I don't expect my Debian involvement to change,
+and I may well find myself more motivated there now that it won't be so
+closely linked with my day job, although it's possible that I will pare some
+things back that I was mostly doing on Ubuntu's behalf. If you ask me for
+help with something over the next few months, expect me to be more likely to
+direct you to other people or suggest ways you can help yourself out, so
+that I can start disentangling myself from my current web of projects.
+
+Please contact me sooner or later if you're interested in helping out with
+any of the things I'm visible in right now, and we can see what makes sense.
+I'm looking forward to this!
--- /dev/null
+Title: NTP synchronisation problems
+Slug: ntp-synchronisation-problems
+Date: 2010-12-06 12:58:29 +0000
+Modified: 2010-12-06 13:05:02 +0000
+Category: ubuntu
+Tags: ubuntu, planet-debian, planet-ubuntu
+
+The Ubuntu Technical Board is currently conducting a review of the top ten
+Brainstorm issues users have raised about Ubuntu, and Matt asked me to
+investigate and respond to [Idea #25301: Keeping the time accurate over the
+Internet by default](http://brainstorm.ubuntu.com/idea/25301/).
+
+My first reaction was "hey, that's odd - I thought we already did that?".
+We install the `ntpdate` package by default (although it's [deprecated
+upstream](http://www.eecis.udel.edu/~mills/ntp/html/ntpdate.html) in favour
+of other tools, but that shouldn't be important here). `ntpdate` is run
+from `/etc/network/if-up.d/ntpdate`, in other words every time you connect
+to a network, which should be acceptably frequent for most people, so it
+really ought to Just Work by default. But this is one of the top ten
+problems where users have gone to the trouble of proposing solutions on
+Brainstorm, so it couldn't be that simple. What was going on?
+
+I brought up a clean virtual machine with a development version of Natty
+(the current Ubuntu development version, which will eventually become
+11.04), and had a look in its logs: it was indeed synchronising its time
+from `ntp.ubuntu.com`, and I didn't think anything in that area had changed
+recently. On the other hand, I had occasionally noticed that my own laptop
+wasn't always synchronising its time quite right, but I'd put it down to
+local weirdness as my network isn't always very stable. Maybe this wasn't
+so local after all?
+
+So, I started tracing through the scripts to figure out what was going on.
+It turned out that I had an empty `/etc/ntp.conf` file on my laptop. The
+`/usr/sbin/ntpdate-debian` script assumed that that meant I had a full NTP
+server installed (I don't), and fetched the list of servers from it; since
+the file was empty, it ended up synchronising time from no servers, that is,
+not synchronising at all. I removed the file and all was well.
+
+That left the question of where that file came from. It didn't seem to be
+owned by any package; I was pretty sure I hadn't created it by hand either.
+I had a look through some bug reports, and soon found [ntpdate
+1:4.2.2.p4+dfsg-1ubuntu2 has a flawed configuration
+file](https://bugs.launchpad.net/bugs/83604). It turns out that
+`time-admin` (System -> Administration -> Time and Date) creates an empty
+`/etc/ntp.conf` file if you press the reload button (tooltip: "Synchronise
+now"), as part of an attempt to update NTP configuration. Aha!
+
+Once I knew where the problems were, it was easy to fix them. I've uploaded
+the following changes, which will be in the 11.04 release:
+
+ * Disregard empty `ntp.conf` files in `ntpdate-debian`.
+ * Remove an empty `/etc/ntp.conf` file on fresh installation of the `ntp`
+ package, so that it doesn't interfere with creating the normal
+ configuration file.
+ * Don't create the NTP configuration file in the `time-admin` backend if it
+ doesn't exist already.
+
+I've also sent these changes to
+[Debian](http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=606107) and
+[GNOME](https://bugzilla.gnome.org/show_bug.cgi?id=449267) as appropriate.
+
+There are still a few problems. The "Synchronise now" button doesn't work
+quite right in general
+([bug #90524](https://bugs.launchpad.net/bugs/90524)), and if your network
+doesn't allow time synchronisation from `ntp.ubuntu.com` then you'll have to
+change the value of `NTPSERVERS` in `/etc/default/ntpdate`. Furthermore,
+the `time-admin` interface is confusing and makes it seem as though the
+default is not to synchronise the time automatically; this interface is
+being [redesigned](https://wiki.ubuntu.com/TimeAndDate) at the moment, which
+should be a good opportunity to make it less confusing, and I will contact
+the designers to mention this problem. On the whole, though, I think that
+many fewer people should have this kind of problem in Ubuntu 11.04.
+
+It's always possible that I missed some other problem that breaks automatic
+time synchronisation for people. Please do file a bug report if it still
+doesn't work for you in 11.04, or contact me directly (cjwatson at
+ubuntu.com).
--- /dev/null
+Title: OpenSSH 5.5p1 for Lucid
+Slug: openssh-5.5p1-for-lucid
+Date: 2010-05-10 10:29:51 +0200
+Category: ubuntu
+Tags: openssh, ubuntu, planet-debian, planet-ubuntu
+
+For various reasons, I chose to leave Ubuntu 10.04 LTS using OpenSSH 5.3p1.
+The [new features in 5.4p1](http://www.openssh.org/txt/release-5.4) such as
+certificate authentication, the new smartcard handling, netcat mode, and
+tab-completion in sftp are great, but unfortunately it was available just a
+little bit too late for me to be able to land it for 10.04 LTS. I realise
+that many Lucid users want to make use of these features for one reason or
+another, though, so as a compromise here's a PPA containing [OpenSSH 5.5p1
+for Lucid](https://launchpad.net/~cjwatson/+archive/openssh).
+
+I intend to keep this up to date for as long as I reasonably can, and I'm
+happy to accept bug reports on it in the [usual
+place](https://bugs.launchpad.net/ubuntu/+source/openssh).
--- /dev/null
+Title: OpenSSH 6.0p1
+Slug: openssh-6.0p1
+Date: 2012-05-27 20:12:12 +0100
+Tags: openssh, planet-debian, planet-ubuntu
+
+OpenSSH 6.0p1 was [released](http://www.openssh.com/txt/release-6.0) a
+little while back; this weekend I belatedly got round to uploading packages
+of it to Debian unstable and Ubuntu quantal.
+
+I was a bit delayed by needing to put together an [improvement to privsep
+sandbox selection](https://bugzilla.mindrot.org/show_bug.cgi?id=2011) that
+particularly matters in the context of distributions. One of the experts on
+`seccomp_filter` has commented favourably on it, but I haven't yet had a
+comment from upstream themselves, so I may need to refine this depending on
+what they say.
+
+(This is a good example of how it matters that software is often not built
+on the system that it's going to run on, and in particular that the kernel
+version is rather likely to be different. Where possible it's always best
+to detect kernel capabilities at run-time rather than at build-time.)
+
+I didn't make it very clear in the changelog, but using the new
+`seccomp_filter` sandbox currently requires `UsePrivilegeSeparation sandbox`
+in `sshd_config` as well as a capable kernel. I won't change the default
+here in advance of upstream, who still consider privsep sandboxing
+experimental.
--- /dev/null
+Title: Forwarding bugs to the IETF
+Slug: openssh-iutf8
+Date: 2006-01-03 13:16:06 +0000
+Tags: openssh, planet-debian, planet-ubuntu
+
+Sometimes following up on a bug takes you a lot further than you expected.
+[Debian bug #337041](http://bugs.debian.org/337041) looked like it was going
+to be fairly straightforward once I upgraded coreutils to figure out what
+the [new IUTF8 flag](http://www.cl.cam.ac.uk/~mgk25/unicode.html#mod)
+actually did, since the SSH2 protocol already supports transferring termios
+flags around.
+
+Unfortunately, since IUTF8 is relatively new, it doesn't have a number
+assigned in the [draft connection
+protocol](http://www.ietf.org/internet-drafts/draft-ietf-secsh-connect-25.txt)
+Moreover, that Internet-Draft is in the last stages before becoming an RFC
+and can't be modified any more, and it doesn't include any facility for
+private-use extensions. D'oh. To add further complication, since IUTF8 is
+Linux-specific, it's not hard to imagine that some other OS might introduce
+something with the same name but subtly different semantics, and so the SSH
+protocols can't just defer to POSIX for the definition but instead have to
+spell out exactly what they mean.
+
+As a result of all of this, it looks like the best way to make progress
+might be for me to write an I-D myself that creates a channel extension to
+set or clear IUTF8, and attempt to enlist support from some upstream
+implementors. I didn't expect bug triage to lead me into the Internet
+standardisation process quite so quickly!
--- /dev/null
+Title: parted 2.2 transition
+Slug: parted-2.2-transition
+Date: 2010-03-22 01:12:48 +0000
+Tags: parted, planet-debian, planet-ubuntu
+
+I've started the [transition of parted 2.2 to
+unstable](http://lists.debian.org/debian-release/2010/03/msg00121.html).
+This is a major update needed for sensible support of newer hard disks with
+alignment requirements different from the archaic cylinder alignment
+tradition. I posted to debian-boot with a [summary of the partman changes
+involved](http://lists.debian.org/debian-boot/2010/03/msg00420.html).
--- /dev/null
+Title: Pipeline library
+Slug: pipeline-library
+Date: 2010-10-03 22:59:11 +0100
+Category: libpipeline
+Tags: libpipeline, man-db, planet-debian, planet-ubuntu
+
+When I took over [man-db](http://man-db.nongnu.org/) in 2001, one of the
+major problems that became evident after maintaining it for a while was the
+way it handled subprocesses. The nature of man and friends means that it
+spends a lot of time calling sequences of programs such as `zsoelim <
+input-file | tbl | nroff -mandoc -Tutf8`. Back then, it was using C library
+facilities such as `system` and `popen` for all this, and I had to deal with
+several bugs where those functions were being called with untrusted input as
+arguments without properly escaping metacharacters. Of course it was
+possible to chase around every such call inserting appropriate escaping
+functions, but this was always bound to be error-prone and one of the tasks
+that rapidly became important to me was arranging to start subprocesses in a
+way that was fundamentally immune to this kind of bug.
+
+In higher-level languages, there are usually standard constructs which are
+safer than just passing a command line to the shell. For example, in Perl
+you can use `system([$command, $arg1, $arg2, ...])` to invoke a program with
+arguments without the interference of the shell, and `perlipc(1)` describes
+various facilities for connecting them together. In Python, the
+[subprocess](http://docs.python.org/library/subprocess.html) module allows
+you to create pipelines easily and safely (as long as you remember the
+[SIGPIPE gotcha]({filename}/python-sigpipe.md)). C has the `fork` and
+`execve` primitives, but assembling these to construct full-blown pipelines
+correctly is difficult and error-prone, so many programmers don't bother and
+use the simple but unsafe library facilities instead.
+
+I wrote a couple of thousand lines of library code in man-db to address this
+problem, loosely and now quite distantly based on code in
+[groff](http://www.gnu.org/software/groff/). In the following examples,
+function names starting with `command_`, `pipeline_`, or `decompress_` are
+real functions in the library, while any other function names are
+pseudocode.
+
+Constructing the simplified example pipeline from my first paragraph using
+this library looks like this:
+
+ :::c
+ pipeline *p;
+ int status;
+
+ p = pipeline_new ();
+ p->want_infile = "input-file";
+ pipeline_command_args (p, "zsoelim", NULL);
+ pipeline_command_args (p, "tbl", NULL);
+ pipeline_command_args (p, "nroff", "-mandoc", "-Tutf8", NULL);
+ pipeline_start (p);
+ status = pipeline_wait (p);
+ pipeline_free (p);
+
+You might want to construct a command more dynamically:
+
+ :::c
+ command *manconv = command_new_args ("manconv", "-f", from_code,
+ "-t", "UTF-8", NULL);
+ if (quiet)
+ command_arg (manconv, "-q");
+ pipeline_command (p, manconv);
+
+Perhaps you want an environment variable set only while running a certain
+command:
+
+ :::c
+ command *less = command_new ("less");
+ command_setenv (less, "LESSCHARSET", lesscharset);
+
+You might find yourself needing to pass the output of one pipeline to
+several other pipelines, in a "tee" arrangement:
+
+ :::c
+ pipeline *source, *sink1, *sink2;
+
+ source = make_source ();
+ sink1 = make_sink1 ();
+ sink2 = make_sink2 ();
+ pipeline_connect (source, sink1, sink2, NULL);
+ /* Pump data among these pipelines until there's nothing left. */
+ pipeline_pump (source, sink1, sink2, NULL);
+ pipeline_free (sink2);
+ pipeline_free (sink1);
+ pipeline_free (source);
+
+Maybe one of your commands is actually an in-process function, rather than
+an external program:
+
+ :::c
+ command *inproc = command_new_function ("in-process", &func, NULL, NULL);
+ pipeline_command (p, inproc);
+
+Sometimes your program needs to consume the output of a pipeline, rather
+than sending it all to some other subprocess:
+
+ :::c
+ pipeline *p = make_pipeline ();
+ const char *line;
+
+ line = pipeline_peekline (p);
+ if (!strstr (line, "coding: UTF-8"))
+ printf ("Unicode text follows:\n");
+ while (line = pipeline_readline (p))
+ printf (" %s", line);
+ pipeline_free (p);
+
+man-db deals with compressed files a lot, so I wrote an add-on library for
+opening compressed files (which is somewhat man-db-specific, but the
+implementation wasn't difficult given the underlying library):
+
+ :::c
+ pipeline *decomp_file = decompress_open (compressed_filename);
+ pipeline *decomp_stdin = decompress_fdopen (fileno (stdin));
+
+This library has been in production in man-db for over five years now. The
+very careful signal handling code has been reviewed independently and the
+whole thing has been run through multiple static analysis tools, although I
+would always welcome more review; in particular I have no idea what it would
+take to make it safe for use in threaded programs since I generally avoid
+threading wherever possible. There have been a handful of bugs, which I've
+fixed promptly, and I've added various new features to support particular
+requirements of man-db (though in as general a way as possible). Every so
+often I see somebody asking about subprocess handling in C, and I wonder if
+I should split this library out into a standalone package so that it can be
+used elsewhere. Web searches for things like "pipeline library" and
+"libpipeline" don't reveal anything that's a particularly close match for
+what I have. The licensing would be GPLv2 or later; this isn't likely to be
+negotiable since some of the original code wasn't mine and in any case I
+don't feel particularly bad about [giving an advantage to GPLed
+programs](http://www.gnu.org/licenses/why-not-lgpl.html). For more details
+on the interface, the [header
+file](http://bazaar.launchpad.net/~cjwatson/man-db/trunk/annotate/head%3A/lib/pipeline.h)
+is well-commented.
+
+Is there enough interest in this to make the effort of producing a separate
+library package worthwhile? As well as the general effort of creating a new
+package, I'd need to do some work to disentangle it from a few bits and
+pieces specific to man-db. If you maintain a specific package that could
+use this and you're interested, please contact me with details, mentioning
+any extensions you think you'd need. I intentionally haven't enabled
+comments on my blog for various reasons, but you can e-mail me at cjwatson
+at debian.org or man-db-devel at nongnu.org.
--- /dev/null
+Title: Porting GHC: A Tale of Two Architectures
+Slug: porting-ghc-a-tale-of-two-architectures
+Date: 2014-04-15 02:36:01 +0100
+Tags: ubuntu, planet-debian, planet-ubuntu
+
+We had
+[some](https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2013-December/014795.html)
+[requests](https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2014-March/014907.html)
+to get [GHC](http://www.haskell.org/ghc/) (the Glasgow Haskell Compiler) up
+and running on two new Ubuntu architectures:
+<acronym title="64-bit ARM, a.k.a. aarch64">arm64</acronym>, added in 13.10,
+and <acronym title="little-endian 64-bit PowerPC">ppc64el</acronym>, added
+in 14.04. This has been something of a saga, and has involved rather more
+late-night hacking than is probably good for me.
+
+## Book the First: Recalled to a life of strange build systems
+
+You might not know it from the sheer bulk of uploads I do sometimes, but I
+actually don't speak a word of Haskell and it's not very high up my list of
+things to learn. But I am a pretty experienced build engineer, and I enjoy
+porting things to new architectures: I'm firmly of the belief that breadth
+of architecture support is a good way to shake out certain categories of
+issues in code, that it's worth doing aggressively across an entire
+distribution, and that, even if you don't think you need something now, new
+requirements have a habit of coming along when you least expect them and you
+might as well be prepared in advance. Furthermore, it annoys me when we
+have excessive noise in our [build failure](http://qa.ubuntuwire.com/ftbfs/)
+and [proposed-migration](https://wiki.ubuntu.com/ProposedMigration) output
+and I often put bits and pieces of spare time into gardening miscellaneous
+problems there, and at one point there was a lot of Haskell stuff on the
+list and it got a bit annoying to have to keep sending patches rather than
+just fixing things myself, and ... well, I ended up as probably the only
+non-Haskell-programmer on the Debian Haskell team and found myself fixing
+problems there in my free time. Life is a bit weird sometimes.
+
+Bootstrapping packages on a new architecture is a bit of a black art that
+only a fairly small number of relatively bitter and twisted people know very
+much about. Doing it in Ubuntu is specifically painful because we've always
+forbidden direct binary uploads: all binaries have to come from a build
+daemon. Compilers in particular often tend to be written in the language
+they compile, and it's not uncommon for them to build-depend on themselves:
+that is, you need a previous version of the compiler to build the compiler,
+stretching back to the dawn of time where somebody put things together with
+a big magnet or something. So how do you get started on a new architecture?
+Well, what we do in this case is we construct a binary somehow (usually
+involving cross-compilation) and insert it as a build-dependency for a
+proper build in Launchpad. The ability to do this is restricted to a small
+group of Canonical employees, partly because it's very easy to make mistakes
+and partly because things like the classic "[Reflections on Trusting
+Trust](http://cm.bell-labs.com/who/ken/trust.html)" are in the backs of our
+minds somewhere. We have an iron rule for our own sanity that the injected
+build-dependencies must themselves have been built from the unmodified
+source package in Ubuntu, although there can be source modifications further
+back in the chain. Fortunately, we don't need to do this very often, but it
+does mean that as somebody who can do it I feel an obligation to try and
+unblock other people where I can.
+
+As far as constructing those build-dependencies goes, sometimes we look for
+binaries built by other distributions (particularly Debian), and that's
+pretty straightforward. In this case, though, these two architectures are
+pretty new and the Debian ports are only just getting going, and as far as I
+can tell none of the other distributions with active arm64 or ppc64el ports
+(or trivial name variants) has got as far as porting GHC yet. Well, OK.
+This was somewhere around the Christmas holidays and I had some time.
+Muggins here cracks his knuckles and decides to have a go at bootstrapping
+it from scratch. It can't be that hard, right? Not to mention that it was
+a blocker for over 600 entries on that build failure list I mentioned, which
+is definitely enough to make me sit up and take notice; we'd even had the
+odd customer request for it.
+
+Several attempts later and I was starting to doubt my sanity, not least for
+trying in the first place. We ship GHC 7.6, and upgrading to 7.8 is not a
+project I'd like to tackle until the much more experienced Haskell folks in
+Debian have switched to it in unstable. The [porting documentation for
+7.6](https://ghc.haskell.org/trac/ghc/wiki/Building/Porting) has bitrotted
+more or less beyond usability, and the [corresponding documentation for
+7.8](https://ghc.haskell.org/trac/ghc/wiki/CrossCompilation) really isn't
+backportable to 7.6. I tried building 7.8 for ppc64el anyway, picking that
+on the basis that we had quicker hardware for it and didn't seem likely to
+be particularly more arduous than arm64 (ho ho), and I even got to the point
+of having a cross-built stage2 compiler (stage1, in the cross-building case,
+is a GHC binary that runs on your starting architecture and generates code
+for your target architecture) that I could copy over to a ppc64el box and
+try to use as the base for a fully-native build, but it segfaulted
+incomprehensibly just after spawning any child process. Compilers tend to
+do rather a lot, especially when they're built to use GCC to generate object
+code, so this was a pretty serious problem, and it resisted analysis. I
+poked at it for a while but didn't get anywhere, and I had other things to
+do so declared it a write-off and gave up.
+
+## Book the Second: The golden thread of progress
+
+In March, another mailing list conversation prodded me into finding a [blog
+entry by Karel
+Gardas](https://ghcarm.wordpress.com/2014/01/18/unregisterised-ghc-head-build-for-arm64-platform/)
+on building GHC for arm64. This was enough to be worth another look, and
+indeed it turned out that (with some help from Karel in private mail) I was
+able to cross-build a compiler that actually worked and could be used to run
+a fully-native build that also worked. Of course this was 7.8, since as I
+mentioned cross-building 7.6 is unrealistically difficult unless you're
+considerably more of an expert on GHC's labyrinthine build system than I am.
+OK, no problem, right? Getting a GHC at all is the hard bit, and 7.8 must
+be at least as capable as 7.6, so it should be able to build 7.6 easily
+enough ...
+
+Not so much. What I'd missed here was that compiler engineers generally
+only care very much about building the compiler with *older* versions of
+itself, and if the language in question has any kind of deprecation cycle
+then the compiler itself is likely to be behind on various things compared
+to more typical code since it has to be buildable with older versions. This
+means that the removal of some deprecated interfaces from 7.8 posed a
+problem, as did some changes in certain <acronym title="primitive
+operations">primops</acronym> that had gained an associated compatibility
+layer in 7.8 but nobody had gone back to put the corresponding compatibility
+layer into 7.6. GHC supports running Haskell code through the C
+preprocessor, and there's a `__GLASGOW_HASKELL__` definition with the
+compiler's version number, so this was just a slog tracking down changes in
+git and adding `#ifdef`-guarded code that coped with the newer compiler
+(remembering that stage1 will be built with 7.8 and stage2 with stage1, i.e.
+7.6, from the same source tree). More inscrutably, GHC has its own
+packaging system called Cabal which is also used by the compiler build
+process to determine which subpackages to build and how to link them against
+each other, and some crucial subpackages weren't being built: it looked like
+it was stuck on picking versions from "stage0" (i.e. the initial compiler
+used as an input to the whole process) when it should have been building its
+own. Eventually I figured out that this was because GHC's use of its
+packaging system hadn't anticipated this case, and was selecting the higher
+version of the `ghc` package itself from stage0 rather than the version it
+was about to build for itself, and thus never actually tried to build most
+of the compiler. Editing `ghc_stage1_DEPS` in `ghc/stage1/package-data.mk`
+after its initial generation sorted this out. One late night building round
+and round in circles for a while until I had something stable, and a Debian
+source upload to add basic support for the architecture name (and other
+changes which were a bit over the top in retrospect: I didn't need to touch
+the embedded copy of libffi, as we build with the system one), and I was
+able to feed this all into Launchpad and watch the builders munch away very
+satisfyingly at the Haskell library stack for a while.
+
+This was all interesting, and finally all that work was actually paying off
+in terms of getting to watch a slew of several hundred build failures vanish
+from arm64 (the final count was something like 640, I think). The fly in
+the ointment was that ppc64el was still blocked, as the problem there wasn't
+building 7.6, it was getting a working 7.8. But now I really did have other
+much more urgent things to do, so I figured I just wouldn't get to this by
+release time and stuck it on the figurative shelf.
+
+## Book the Third: The track of a bug
+
+Then, last Friday, I cleared out my urgent pile and thought I'd have another
+quick look. (I get a bit obsessive about things like this that smell of
+"interesting intellectual puzzle".) slyfox on the #ghc IRC channel gave me
+some general debugging advice and, particularly usefully, a reduced example
+program that I could use to debug just the process-spawning problem without
+having to wade through noise from running the rest of the compiler. I
+reproduced the same problem there, and then found that the program crashed
+earlier (in `stg_ap_0_fast`, part of the run-time system) if I compiled it
+with `+RTS -Da -RTS`. I nailed it down to a small enough region of assembly
+that I could see all of the assembly, the source code, and an intermediate
+representation or two from the compiler, and then started meditating on what
+makes ppc64el special.
+
+You see, the vast majority of porting bugs come down to what I might call
+gross properties of the architecture. You have things like whether it's
+32-bit or 64-bit, big-endian or little-endian, whether `char` is signed or
+unsigned, that sort of thing. There's a [big
+table](https://wiki.debian.org/ArchitectureSpecificsMemo) on the Debian wiki
+that handily summarises most of the important ones. Sometimes you have to
+deal with distribution-specific things like whether GL or GLES is used;
+often, especially for new variants of existing architectures, you have to
+cope with foolish configure scripts that think they can guess certain things
+from the architecture name and get it wrong (assuming that `powerpc*` means
+big-endian, for instance). We often have to update `config.guess` and
+`config.sub`, and on ppc64el we have the additional hassle of updating
+libtool macros too. But I've done a lot of this stuff and I'd accounted for
+everything I could think of. ppc64el is actually a lot like amd64 in terms
+of many of these porting-relevant properties, and not even that far off
+arm64 which I'd just successfully ported GHC to, so I couldn't be dealing
+with anything particularly obvious. There was some hand-written assembly
+which certainly could have been problematic, but I'd carefully checked that
+this wasn't being used by the "unregisterised" (no specialised machine
+dependencies, so relatively easy to port but not well-optimised) build I was
+using. A problem around spawning processes suggested a problem with
+`SIGCHLD` handling, but I ruled that out by slowing down the first child
+process that it spawned and using `strace` to confirm that `SIGSEGV` was the
+first signal received. What on earth was the problem?
+
+From some painstaking gdb work, one thing I eventually noticed was that
+`stg_ap_0_fast`'s local stack appeared to be being corrupted by a function
+call, specifically a call to the colourfully-named `debugBelch`. Now, when
+IBM's toolchain engineers were putting together ppc64el based on ppc64, they
+took the opportunity to fix a number of problems with their ABI: there's an
+[OpenJDK bug](https://bugs.openjdk.java.net/browse/JDK-8035647) with a handy
+list of references. One of the things I noticed there was that there were
+some [stack allocation
+optimisations](http://gcc.gnu.org/ml/gcc-patches/2013-11/msg01149.html) in
+the new ABI, which affected functions that don't call any vararg functions
+and don't call any functions that take enough parameters that some of them
+have to be passed on the stack rather than in registers. `debugBelch` takes
+varargs: hmm. Now, the calling code isn't quite in C as such, but in a
+related dialect called "Cmm", a variant of C-- (yes, minus), that GHC uses
+to help bridge the gap between the functional world and its code generation,
+and which is compiled down to C by GHC. When importing C functions into
+Cmm, GHC generates prototypes for them, but it doesn't do enough parsing to
+work out the true prototype; instead, they all just get something like
+`extern StgFunPtr f(void);`. In most architectures you can get away with
+this, because the arguments get passed in the usual calling convention
+anyway and it all works out, but on ppc64el this means that the caller
+doesn't generate enough stack space and then the callee tries to save its
+varargs onto the stack in an area that in fact belongs to the caller, and
+suddenly everything goes south. Things were starting to make sense.
+
+Now, `debugBelch` is only used in optional debugging code; but
+`runInteractiveProcess` (the function associated with the initial round of
+failures) takes no fewer than twelve arguments, plenty to force some of them
+onto the stack. I poked around the GCC patch for this ABI change a bit and
+determined that it only optimised away the stack allocation if it had a full
+prototype for all the callees, so I guessed that changing those prototypes
+to `extern StgFunPtr f();` might work: it's still technically wrong, not
+least because omitting the parameter list is an obsolescent feature in C11,
+but it's at least just omitting information about the parameter list rather
+than actively lying about it. I tweaked that and ran the cross-build from
+scratch again. Lo and behold, suddenly I had a working compiler, and I
+could go through the same build-7.6-using-7.8 procedure as with arm64, much
+more quickly this time now that I knew what I was doing. One [upstream
+bug](https://ghc.haskell.org/trac/ghc/ticket/8965), one Debian upload, and
+several bootstrapping builds later, and GHC was up and running on another
+architecture in Launchpad. Success!
+
+## Epilogue
+
+There's still more to do. I gather there may be a Google Summer of Code
+project in Linaro to write proper native code generation for GHC on arm64:
+this would make things a good deal faster, but also enable GHCi (the
+interpreter) and Template Haskell, and thus clear quite a few more build
+failures. Since there's already native code generation for ppc64 in GHC,
+getting it going for ppc64el would probably only be a couple of days' work
+at this point. But these are niceties by comparison, and I'm more than
+happy with what I got working for 14.04.
+
+The upshot of all of this is that I may be the first non-Haskell-programmer
+to ever port GHC to two entirely new architectures. I'm not sure if I gain
+much from that personally aside from a lot of lost sleep and being
+considered extremely strange. It has, however, been by far the most
+challenging set of packages I've ported, and a fascinating trip through some
+odd corners of build systems and undefined behaviour that I don't normally
+need to touch.
--- /dev/null
+Title: Python SIGPIPE handling
+Slug: python-sigpipe
+Date: 2009-07-02 08:14:26 +0000
+Tags: python, planet-debian, planet-ubuntu
+
+[Enrico](http://www.enricozini.org/2009/debian/python-pipes/) writes about
+creating pipelines with Python's `subprocess` module, and notes that you
+need to take care to close stdout in non-final subprocesses so that
+subprocesses get `SIGPIPE` correctly. This is correct as far as it goes
+(and true in any language, although there's a [Python bug report requesting
+that `subprocess` be able to do this
+itself](http://bugs.python.org/issue1615376), but there's an additional
+gotcha with Python that you missed.</p>
+
+Python ignores `SIGPIPE` on startup, because it prefers to check every write
+and raise an `IOError` exception rather than taking the signal. This is all
+well and good for Python itself, but most Unix subprocesses don't expect to
+work this way. Thus, when you are creating subprocesses from Python, it is
+**very important** to set `SIGPIPE` back to the default action. Before I
+realised this was necessary, I wrote code that caused serious data loss due
+to a child process carrying on out of control after its parent process died!
+
+ import signal
+ import subprocess
+
+ def subprocess_setup():
+ # Python installs a SIGPIPE handler by default. This is usually not what
+ # non-Python subprocesses expect.
+ signal.signal(signal.SIGPIPE, signal.SIG_DFL)
+
+ subprocess.Popen(command, preexec_fn=subprocess_setup)
+
+I filed a [patch](http://bugs.python.org/issue1652) a while back to add a
+`restore_sigpipe` option to `subprocess.Popen`, which would take care of
+this. As I say in that bug report, in a future release I think this ought
+to be made the default, as it's very easy to get things dangerously wrong
+right now.
--- /dev/null
+Title: Quality in Ubuntu 12.04 LTS
+Slug: quality-in-12-04
+Date: 2011-10-24 14:57:41 +0100
+Category: ubuntu
+Tags: ubuntu, planet-debian, planet-ubuntu
+
+As is natural for an LTS cycle, lots of people are thinking and talking
+about work focused on quality rather than features. With Canonical
+[extending LTS
+support](http://www.canonical.com/content/ubuntu-1204-feature-extended-support-period-desktop-users)
+to five years on the desktop for 12.04, much of this is quite rightly
+focused on the desktop. I'm really not a desktop hacker in any way, shape,
+or form, though. I spent my first few years in Ubuntu working mainly on the
+installer - I still do, although I do some other things now too - and I used
+to say only half-jokingly that my job was done once X started. Of course
+there are plenty of bugs I can fix, but I wanted to see if I could do
+something with a bit more structure, so I got to thinking about projects we
+could work on at the foundations level that would make a big difference.
+
+## Image build pipeline
+
+One difficulty we have is that quite a few of our bugs - especially
+installer bugs, although this goes for some other things too - are only
+really caught when people are doing coordinated image testing just before a
+milestone release. Now, it takes a while to do all the builds and then it
+takes a while to test them. The excellent work of the QA team has meant
+that testing is much quicker now than it used to be, and a certain amount of
+smoke-testing is automated (particularly for server images). On the other
+hand, the build phase has only got longer as we've added more flavours and
+architectures, particularly as some parts of the process are still
+serialised per architecture or subarchitecture so ARM builds in particular
+take a very long time indeed. Exact timings are a bit difficult to get for
+various reasons, but I think the minimum time between a developer uploading
+a fix and us having a full set of candidate images on all architectures
+including that fix is currently somewhere north of eight hours, and that's
+with people cutting corners and pulling strings which is a suboptimal thing
+to have to do around release time. This obviously makes us reluctant to
+respin for anything short of showstopper bugs. If we could get things down
+to something closer to two hours, respins would be a much less horrible
+proposition and so we might be able to fix a few bugs that are serious but
+not showstoppers, not to mention that the release team would feel less
+burned out.
+
+We discussed this problem at the release sprint, and came up with a [laundry
+list of
+improvements](https://blueprints.launchpad.net/ubuntu/+spec/foundations-p-image-build-pipeline);
+I've scheduled this for discussion at UDS in case we can think of any more.
+Please come along if you're interested!
+
+One thing in particular that I'm working on is refactoring
+[Germinate](https://launchpad.net/germinate), a tool which dates right back
+to our first meeting before Ubuntu was even called Ubuntu and whose job is
+to expand dependencies starting from our lists of "seed" packages; we use
+this, among other things, to generate `Task` fields in the archive and to
+decide which packages to copy into our images. This was acceptably quick in
+2004, but now that we run it forty times (eight flavours multiplied by five
+architectures) at the end of every publisher run it's actually become rather
+a serious performance problem: `cron.germinate` takes about ten minutes,
+which is over a third of the typical publisher runtime. It parses Packages
+files eight times as often as it needs to, Sources files forty times as
+often as it needs to, and recalculates the dependency tree of the base
+system five times as often as it needs to. I am confident that we can
+significantly reduce the runtime here, and I think there's some hope that we
+might be able to move the publisher back to a 30-minute cycle, which would
+increase the velocity of Ubuntu development in general.
+
+## Maintaining the development release
+
+Our release cycle always starts with syncing and merging packages from
+Debian unstable (or testing in the case of LTS cycles). The vast majority
+of packages in Ubuntu arrive this way, and generally speaking if we didn't
+do this we would fall behind in ways that would be difficult to recover from
+later. However, this does mean that we get a "big bang" of changes at the
+start of the cycle, and it takes a while for the archive to be usable again.
+Furthermore, even once we've taken care of this, we have a long-established
+rhythm where the first part of the cycle is mainly about feature development
+and the second part of the cycle is mainly about stabilisation. As a
+result, we've got used to the archive being fairly broken for the first few
+months, and we even tell people that they shouldn't expect things to work
+reliably until somewhere approaching beta.
+
+This makes some kind of sense from the inside. But how are you supposed to
+do feature development that relies on other things in the development
+release?
+
+In the first few years of Ubuntu, this question didn't matter very much.
+Nearly all the people doing serious feature development were themselves
+serious Ubuntu developers; they were capable of fixing problems in the
+development release as they went along, and while it got in their way a
+little bit it wasn't all that big a deal. Now, though, we have people
+focusing on things like Unity development, and we shouldn't assume that just
+because somebody is (say) an OpenGL expert or a window management expert
+that they should be able to recover from arbitrary failures in development
+release upgrades. One of the best things we could do to help the 12.04
+desktop be more stable is to have the entire system be less unstable as we
+go along, so that developers further up the stack don't have to be
+distracted by things wobbling underneath them. Plus, it's just good
+software engineering to keep the basics working as you go along: it should
+always build, it should always install, it should always upgrade. Ubuntu is
+too big to do something like having everyone stop any time the build breaks,
+the way you might do in a smaller project, but we shouldn't let things slide
+for months either.
+
+I've been talking to [Rick Spencer](http://theravingrick.blogspot.com/) and
+the other Ubuntu engineering leads at Canonical about this. Canonical has a
+system of "rotations", where you can go off to another team for a while if
+you're in need of a change or want to branch out a bit; so I proposed that
+we allow our engineers to spend a month or two at a time on what I'm calling
+the **+1 Maintenance Team**, whose job is simply to keep the development
+release buildable, installable, and upgradeable at all times. Rick has been
+very receptive to this, and we're going to be running this as a trial
+throughout the 12.04 cycle, with probably about three people at a time. As
+well as being professional archive gardeners, these people will also work on
+developing infrastructure to help us keep better track of what we need to
+do. For instance, we could deploy better tools from Debian QA to help us
+track uninstallable packages, or we could enhance
+[some](http://people.canonical.com/~ubuntu-archive/nbs.html) of our
+[many](http://conflictchecker.ubuntu.com/possible-conflicts/oneiric/main.txt)
+[existing](http://people.canonical.com/~ubuntu-archive/component-mismatches.txt)
+[reports](http://people.canonical.com/~ubuntu-archive/testing/precise_probs.html)
+to have bug links and/or comment facilities, or we could spruce up the
+[weather
+report](http://reports.qa.ubuntu.com/reports/ogasawara/weatherreport.html);
+there are lots of things we could do to make our own lives easier.
+
+By 12.04, I would like, in no particular order:
+
+ * Precise to have been more or less continuously usable from Alpha 1 onward
+ for people with reasonable general technical ability
+ * Canonical engineering teams outside Ubuntu (DX, Ubuntu One, Launchpad,
+ etc.) to be comfortable with running the development release on at least
+ one system from Alpha 2 onward
+ * Installability problems in daily image builds to be dealt with within one
+ working day, or preferably before they even make it to daily builds
+ * The archive to be close to consistent as we start milestone preparation,
+ rather than the release team having to scramble to make it so
+ * A very significant reduction in our long-term backlog of
+ automatically-detected problems
+
+Of course, this overlaps to a certain extent with the kinds of things that
+the MOTU team have been doing for years, not to mention with what all
+developers should be doing to keep their own houses in reasonable order, and
+I'd like us to work together on this; we're trying to provide some extra
+hands here to make Ubuntu better for everyone, not take over! I would love
+this to be an opportunity to re-energise MOTU and bring some new people on
+board.
+
+I've registered a couple of blueprints
+([priorities](https://blueprints.launchpad.net/ubuntu/+spec/other-p-plusonemaint-priorities),
+[infrastructure](https://blueprints.launchpad.net/ubuntu/+spec/other-p-plusonemaint-infrastructure))
+for discussion at UDS. These are deliberately open-ended skeleton sessions,
+and I'll try to make sure they're scheduled fairly early in the week, so
+that we have time for break-out sessions later on. If you're interested,
+please come along and give your feedback!
--- /dev/null
+Title: Re: Perl is strange
+Slug: reply-perl-is-strange
+Date: 2008-06-23 15:58:08 +0000
+Modified: 2008-06-23 15:59:12 +0000
+Tags: planet-debian
+
+[Christoph](http://www.df7cb.de/blog/2008/Perl_is_strange.html): That's
+because `=~` binds more tightly than `+`. This does what you meant:
+
+ $ perl -le 'print "yoo" if (1 + 1) =~ /3/'
+
+`perlop(1)` has a useful table of precedence.
--- /dev/null
+Title: aptitude safe-upgrade
+Slug: safe-upgrade
+Date: 2007-11-29 20:51:23 +0000
+Tags: planet-debian, planet-ubuntu
+
+[Erich](http://blog.drinsama.de/erich/en/linux/debian/2007112801-dist-upgrade-hints.html):
+I do sometimes wonder why we don't relax the definition of "safe" upgrades
+to include installing new packages but still not removing old ones. I know
+that many of my uses of dist-upgrade are just for when something grows a new
+dependency that I didn't previously have installed.
+
+(Of course this wouldn't always help as it wouldn't account for a new
+dependency that conflicted with an old dependency, but never mind. It would
+certainly do wonders for the metapackage case.)
--- /dev/null
+Title: Single-stage installer
+Slug: single-stage-installer
+Date: 2006-01-03 15:32:27 +0000
+Category: ubuntu
+Tags: d-i, planet-debian, planet-ubuntu
+
+Hot on the heels of [Joey's tale of getting rid of
+base-config](http://kitenet.net/~joey/blog/entry/all_this_for_a_progress_bar-2005-12-27-20-32.html)
+(the second stage of the installer) in Debian, we've now pretty much got rid
+of it in Ubuntu Dapper too. The upshot of this is that rather than asking a
+bunch of questions, installing the base system, and rebooting to install
+everything else, we now just install everything in one go and reboot into a
+completed system.
+
+This does mean that, if your system doesn't boot, you don't get to find out
+about it for a bit longer. However, it has lots of advantages in terms of
+speed (the much-maligned archive-copier mostly goes away), reducing code
+duplication (base-config had a bunch of infrastructure of its own which was
+done better in the core installer anyway), comprehensibility, and killing
+off some annoying bugs like [#13561 (duplicate mirror questions in netboot
+installs)](https://bugzilla.ubuntu.com/show_bug.cgi?id=13561), [#15213
+(second stage hangs if you skip archive-copier in the first
+stage)](https://bugzilla.ubuntu.com/show_bug.cgi?id=15213), and [#19571
+(kernel messages scribble over base-config's
+UI)](https://bugzilla.ubuntu.com/show_bug.cgi?id=19571).</p>
+
+To go with Joey's Debian timeline, the Ubuntu history looks a bit like this:
+
+ * 2004 (Jul): First base-config modifications for Ubuntu; we need to
+ install the default desktop rather than dropping into tasksel.
+ * 2004 (Aug): Mark phones me up and asks if I can make the installer not
+ need the CD in the second stage by copying all the packages across
+ beforehand. Although it's a bit awkward, I can see the UI advantages in
+ that, so I write archive-copier at the Canonical conference in Oxford.
+ * 2004 (Sep): Mark asks me if we can ask the timezone, user/password, and
+ apt configuration questions before the first reboot. With less than a
+ month to go until our first release, I have a
+ [heart-attack](http://lists.ubuntu.com/archives/ubuntu-devel/2004-September/000103.html)
+ at how much needs to be done, and it eventually gets deferred to
+ [Hoary](https://wiki.ubuntu.com/HoaryGoals).
+ * 2005 (Jan): Matt fixes up debconf's passthrough frontend for use on the
+ live CD, and we realise that this is an obvious way to run bits of
+ base-config before the first reboot. It's rather messy and takes until
+ March or so before it really works right, but we get there in the end.
+ * 2005 (Apr): I get "put a progress bar in front of the dpkg output in the
+ second stage" as a
+ [goal](https://wiki.ubuntu.com/UbuntuDownUnder/BOFs/InstallerStage2Progress)
+ for Breezy. Naïvely, I think it's a simple matter of programming, since
+ I'd already done something similar for debootstrap and base-installer the
+ previous year.
+ * 2005 (May): I hack progress bar support into debconf. Nothing actually
+ uses it for anything yet, except as a convenient passthrough stub.
+ * 2005 (Jul/Aug): I actually try to implement the second-stage progress bar
+ and realise that it's about an order of magnitude harder than I thought,
+ requiring a whole load of extra infrastructure in apt. Fortunately
+ Michael Vogt saves the day here by writing lots of working code, and the
+ progress bar works by early August.
+ * 2005 (Sep-Dec): Upstream d-i development ramps back up again, with
+ tzsetup, clock-setup, apt-setup, and user-setup all being cranked out in
+ short order and the corresponding pieces removed from base-config. I
+ merge these as they mature, and manage to get
+ [agreement](http://lists.debian.org/debian-boot/2005/10/msg01407.html) on
+ including the Ubuntu debconf template changes in upstream apt-setup,
+ which helps the diff size a lot.
+ * 2005 (Nov/Dec): Joey and I
+ [chat](http://lists.debian.org/debian-boot/2005/11/msg01381.html) one
+ evening about the Ubuntu second-stage progress bar work, and we end up
+ designing and writing debconf-apt-progress based on its ideas, after
+ which Joey knocks up pkgsel in no time flat.
+ * 2006 (Jan): The rest of the pieces land in Ubuntu, and we drop
+ base-config out of the installer. To my surprise, nearly everything
+ still just works.
+
+Although it caused some friction, I'm glad that we did the first cuts of
+many of these things outside Debian and got to try things out before landing
+version-2-quality code in Debian. The end result is much nicer than the
+intermediate ones ever were.
--- /dev/null
+Title: Unix tools: sponge
+Slug: sponge
+Date: 2006-02-06 20:45:38
+Tags: planet-debian, planet-ubuntu
+
+Joey
+[writes](http://kitenet.net/~joey/blog/entry/unix_tools_vidir-2006-02-05-21-33.html)
+about the lack of new tools that fit into the Unix philosophy. My favourite
+of such things I've written is
+[sponge](http://riva.ucam.org/svn/cjwatson/bin/sponge). It addresses the
+problem of editing files in-place with Unix tools, namely that if you just
+redirect output to the file you're trying to edit then the redirection takes
+effect (clobbering the contents of the file) before the first command in the
+pipeline gets round to reading from the file. Switches like `sed -i` and
+`perl -i` work around this, but not every command you might want to use in a
+pipeline has such an option, and you can't use that approach with
+multiple-command pipelines anyway.
+
+I normally use sponge a bit like this:
+
+ sed '...' file | grep '...' | sponge file
+
+Since it's so trivial I imagine lots of other people have written something
+similar (another common name for it seems to be inplace; my name indicates
+soaking up all the input and then squeezing it all out again); but I do keep
+meaning to try to get a rewritten version into coreutils at some point.
--- /dev/null
+Title: Don't use sshkeygen.com to generate keys!
+Slug: ssh-keygen
+Date: 2008-06-23 09:31:57 +0000
+Modified: 2008-06-23 09:54:53 +0000
+Tags: openssh, planet-debian, planet-ubuntu
+
+To my horror, I recently saw [this online SSH key
+generator](http://www.sshkeygen.com/).
+
+I hope nobody reading this needs to be told why this is a bad idea.
+However, in case you do, here are a few reasons:
+
+ * Every SSH implementation I know of - certainly all the major ones - that
+ support public key authentication also provide a key generation utility.
+ Even aside from all the good reasons not to, there is simply no reason
+ why you should need to use a web-based tool in the first place.
+ * How can you trust the person running this site? Without implying that I
+ know he or she is untrustworthy (I don't), and with the best will in the
+ world, it's a big Internet with a lot of nasty people on it. Do you
+ really want somebody you don't know in a position to keep a copy of all
+ your private keys?
+ * Even if the person is trustworthy, the server running sshkeygen.com is
+ now a giant blinking target. If lots of people use it, there is every
+ incentive in the world for the bad guys to try to take control of it so
+ that they can keep a copy of all your private keys. (Or, as we know from
+ recent bitter experience, they can just give out keys from a limited set
+ and it will probably take a couple of years before anyone notices ...)
+ * The front page of sshkeygen.com says that the keys are escrowed. The
+ plain English meaning of this would be that the operator of that site
+ keeps a copy of the private key, to be held in trust in case (presumably)
+ you lose it and need to retrieve it. Normally this sort of thing depends
+ on a legal trust relationship, perhaps linked to a contract. What does
+ it mean here? Is it just a buzzword? If it isn't, then this just makes
+ sshkeygen.com even more of a target.
+ * sshkeygen.com delivers keys to you over unencrypted HTTP. Yes, this is
+ on its [to-do list](http://www.sshkeygen.com/about.php). That isn't
+ really an excuse.
+ * Even if keys were delivered over HTTPS, that still relies on people
+ diligently checking the authenticity of the certificate. A
+ self-signature (as suggested as an alternative in the to-do list) would
+ be impossible to check with any reliability; and will people who have
+ trouble with non-web-based key generation software really be able or
+ inclined to confirm the signature chain? Browsers typically don't
+ enforce this very strictly, or if they do they provide fairly simple ways
+ to bypass the enforcement, simply because so many sites have broken or
+ poorly-signed SSL certificates, and keeping up with all the CAs is pretty
+ hard work too.
+ * Furthermore, delivering private keys over HTTPS makes that SSL
+ certificate a single giant blinking target. Might it be compromised?
+ How would you tell? What servers would need to be compromised in order
+ to get a copy of the private SSL key?
+ * Sure, Debian is in an awkward position here given the recent OpenSSL
+ random number generation vulnerability. However, how do you know that
+ sshkeygen.com is running on a system that doesn't suffer from this? (As
+ it happens, I have checked, and it doesn't appear to suffer from this
+ vulnerability - but most people won't check and won't know how to check.)
+
+I *think* this is probably being done in innocent seriousness (although I
+kind of hope it's a joke in poor taste), and have e-mailed the contact
+address offering to explain why it's a bad idea.
--- /dev/null
+Title: Testing wanted: GRUB 2.02~beta2 Debian/Ubuntu packages
+Slug: testing-wanted-grub-2.02-beta2
+Date: 2014-01-18 01:46:55 +0000
+Category: grub
+Tags: grub, ubuntu, planet-debian, planet-ubuntu
+
+This is mostly a repost of my [ubuntu-devel
+mail](https://lists.ubuntu.com/archives/ubuntu-devel/2014-January/037978.html)
+for a wider audience, but see below for some additions.
+
+I'd like to upgrade to GRUB 2.02 for Ubuntu 14.04; it's currently in beta.
+This represents a year and a half of upstream development, and contains many
+new features, which you can see in the
+[NEWS](http://git.savannah.gnu.org/gitweb/?p=grub.git;a=blob;f=NEWS) file.
+
+Obviously I want to be very careful with substantial upgrades to the default
+boot loader. So, I've put this in trusty-proposed, and filed a [blocking
+bug](https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1269992) to ensure
+that it doesn't reach trusty proper until it's had a reasonable amount of
+manual testing. If you are already using trusty and have some time to try
+this out, it would be very helpful to me. I suggest that you only attempt
+this if you're comfortable driving `apt-get` directly and recovering from
+errors at that level, and if you're willing to spend time working with me on
+narrowing down any problems that arise.
+
+Please ensure that you have rescue media to hand before starting testing.
+The simplest way to upgrade is to enable trusty-proposed, upgrade ONLY
+packages whose names start with "grub" (e.g. use `apt-get dist-upgrade` to
+show the full list, say no to the upgrade, and then pass all the relevant
+package names to `apt-get install`), and then (very important!) disable
+trusty-proposed again. Provided that there were no errors in this process,
+you should be safe to reboot. If there were errors, you should be able to
+downgrade back to 2.00-22 (or 1.27+2.00-22 in the case of
+grub-efi-amd64-signed).
+
+Please report your experiences (positive and negative) with this upgrade in
+the [tracking
+bug](https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1269992). I'm
+particularly interested in systems that are complex in any way: UEFI Secure
+Boot, non-trivial disk setups, manual configuration, that kind of thing. If
+any of the problems you see are also ones you saw with earlier versions of
+GRUB, please identify those clearly, as I want to prioritise handling
+regressions over anything else. I've assigned myself to that bug to ensure
+that messages to it are filtered directly into my inbox.
+
+I'll add a couple of things that weren't in my ubuntu-devel mail. Firstly,
+this is all in Debian experimental as well (I do all the work in Debian and
+sync it across, so the grub2 source package in Ubuntu is a verbatim copy of
+the one in Debian these days). There are some configuration differences
+applied at build time, but a large fraction of test cases will apply equally
+well to both. I don't have a definite schedule for pushing this into jessie
+yet - I only just finished getting 2.00 in place there, and the release
+schedule gives me a bit more time - but I certainly want to ship jessie with
+2.02 or newer, and any test feedback would be welcome. It's probably best
+to just e-mail feedback to me directly for now, or to the pkg-grub-devel
+list.
+
+Secondly, a couple of news sites have picked this up and run it as
+"Canonical intends to ship Ubuntu 14.04 LTS with a beta version of GRUB".
+This isn't in fact my intent at all. I'm doing this now because I think
+GRUB 2.02 will be ready in non-beta form in time for Ubuntu 14.04, and
+indeed that putting it in our development release will help to stabilise it;
+I'm an upstream GRUB developer too and I find the exposure of widely-used
+packages very helpful in that context. It will certainly be much easier to
+upgrade to a beta now and a final release later than it would be to try to
+jump from 2.00 to 2.02 in a month or two's time.
+
+Even if there's some unforeseen delay and 2.02 isn't released in time,
+though, I think nearly three months of stabilisation is still plenty to
+yield a boot loader that I'm comfortable with shipping in an LTS. I've been
+backporting a lot of changes to 2.00 and even 1.99, and, as ever for an
+actively-developed codebase, it gets harder and harder over time (in
+particular, I've spent longer than I'd like hunting down and backporting
+fixes for non-512-byte sector disks). While I can still manage it, I don't
+want to be supporting 2.00 for five more years after upstream has moved on;
+I don't think that would be in anyone's best interests. And I definitely
+want some of the new features which aren't sensibly backportable, such as
+several of the new platforms (ARM, ARM64, Xen) and various networking
+improvements; I can imagine a number of our users being interested in things
+like optional signature verification of files GRUB reads from disk, improved
+Mac support, and the TrueCrypt ISO loader, just to name a few. This should
+be a much stronger base for five-year support.
--- /dev/null
+Title: Thoughts on 3.0 (quilt) format
+Slug: thoughts-on-3.0-quilt-format
+Date: 2010-03-25 23:45:28 +0000
+Modified: 2010-03-26 01:00:06 +0000
+Tags: dpkg, planet-debian, planet-ubuntu
+
+Note: I wrote most of this before [Neil Williams' recent comments on the 3.0
+family of
+formats](http://www.linux.codehelp.co.uk/serendipity/index.php?/archives/201-lintian,-source-format-3.0-and-blog-comments.html),
+so despite the timing this isn't really a reaction to that although I do
+have a couple of responses. On the whole I think I agree that the Lintian
+message is a bit heavy-handed and I'm not sure I'm thrilled about the idea
+of the default source format being changed (though I can see why the dpkg
+maintainers are interested in that). That said, as far as I personally am
+concerned, there is a vast cognitive benefit to me in having as much as
+possible be common to all my packages. Once I have more than a couple of
+packages that require patching and benefit from the `3.0 (quilt)` format as
+a result, I find it in my interest to use it for all my non-native packages
+even if they're patchless right now, so that for instance if they need
+patches in the future I can handle them the same way. It's not unheard of
+for me to apply temporary patches even to packages I actively maintain
+upstream, so I don't discount those either. I haven't decided what to do
+with my native packages yet; unless they're big enough for bzip2 compression
+to be worthwhile, there doesn't seem to be much immediate advantage to `3.0
+(native)`.
+
+Anyway, on to the main body of this post:
+
+I've been one of the holdouts resisting use of patch systems for a long
+time, on the basis that I felt strongly that `dpkg-source -x` ought to give
+you the source that's actually built, rather than having to mess around with
+`debian/rules` targets in order to see it. Now that the `3.0 (quilt)`
+format is available to fix this bug, I felt that I ought to revisit my
+resistance and start trying to use it. Migrating to it from monolithic
+diffs is of course a bit more work than migrating to it from other patch
+systems, so it's taken me a little while to get round to it. I'd been
+thinking about holding off until there was better integration with revision
+control (e.g. bzr looms), as I feel that patch files really ought to be an
+export format, but I eventually decided that I shouldn't let the perfect be
+the enemy of the good. I have enough experience with co-maintaining
+packages that use build-time patch systems to be able to compare my
+reactions.
+
+After experimenting with a couple of small packages, I moved over to the
+deep end and [converted
+openssh](http://packages.qa.debian.org/o/openssh/news/20100228T035004Z.html)
+a few weekends ago, since quite a few people have requested over the years
+that the Debian changes to openssh be easier to audit. This was a
+substantial job - over 6000 lines of upstream patches - but not actually as
+much work as I expected. I took a fairly simplistic approach: first, I
+unapplied all the upstream patches from my tree; then I ran `bzr di |
+interdiff -q /dev/stdin /dev/null >x`, reduced it to a single
+logically-discrete patch, applied it to a new quilt patch using `quilt
+fold`, and repeated until `x` was empty. This was maybe an hour or two of
+work, and then I went through and tagged all the patches according to
+[DEP-3](http://dep.debian.net/deps/dep3/), which took another few hours.
+After the first pass, I ended up with 38 patches and a much clearer idea of
+what has been forwarded upstream and what hasn't; I currently have 5 patches
+to forward or eliminate, down from 18.
+
+Good things:
+
+ * I don't lose any of my history. Since all the patches remain applied to
+ the tree in revision control (this is what `dpkg-source -x` gives you, so
+ it's the natural representation in revision control too), `bzr blame`
+ works just as you'd expect and displays both upstream and Debian changes
+ at once. I rely on tools like blame a lot, and I really hate the way
+ build-time patch systems make it hard to use revision control when the
+ tree is in a built state, so this was a hard requirement for me.
+ * I've used patch tagging before, so I was expecting some benefits, but
+ viscerally I feel much more in *control*. It's so much less laborious
+ now to see what I need to do by way of forwarding. I don't regret
+ waiting for 3.0 (quilt) to become available, but I hadn't realised quite
+ how much I was being held back beforehand.
+ * Adding new patches is pretty natural, much more so than with build-time
+ patch systems. You can create and apply the patch, test-build, and
+ commit when it works. I much prefer this over having to clean the tree
+ before committing (or commit just part of the tree, which is
+ error-prone). The more that committing to a Debian package feels like
+ committing to an upstream project, the better.
+ * There's definitely something to be said for
+ [patch-tracker](http://patch-tracker.debian.org/package/openssh) being
+ more useful. It deals with DEP-3 to the extent of linkifying URLs,
+ although it might be nice if patch descriptions were displayed on the
+ overview page for each version.
+
+Bad things:
+
+ * It's a bit awkward to set things up when checking out from revision
+ control; I didn't really want to check in the `.pc` directory, and the
+ tree checks out in the patched state (as it should), so I needed some way
+ for developers to get quilt working easily after a checkout. This is
+ sort of the reverse of the previous problem, where users had to do
+ something special after `dpkg-source -x`, and I consider it less serious
+ so I'm willing to put up with it. I ended up with [a rune in
+ debian/rules that ought to live somewhere more
+ common](http://bugs.debian.org/572204).
+ * Everything ends up represented twice in revision control: the patch
+ files, plus the changes to the patched files themselves. I'm OK with
+ this although it is a little inelegant.
+ * Although I haven't had to do it yet, I expect that merging new upstream
+ releases will be a bit harder. bzr will deal with resolving conflicts in
+ the patched files themselves, and that's why I use a revision control
+ system after all, but then I'll have to go and refresh all the patches
+ and will probably end up doing some of the same conflict resolution a
+ second time. I think the best answer right now is to `quilt pop -a`,
+ force a merge despite the modified working tree, and then `quilt push &&
+ quilt refresh -pab` until I get back to the top of the stack, modulo
+ slight fiddliness when a patch disappears entirely; thus effectively
+ using quilt's conflict resolution rather than bzr's. I suppose this will
+ serve as additional incentive to reduce my patch count. I know that
+ people have been working on making this work nicely with topgit, although
+ I'm certainly not going to put up with the rest of git due to that; I'm
+ happy to wait for looms to become usable and integrated. :-)
+ * It would be nice if there were some standard DEP-3 way to note that a
+ patch has been accepted or rejected upstream, beyond just putting it in
+ the description. In particular, it seems to me that listing patches
+ accepted upstream could be used to speed up the process of merging new
+ upstream releases.
+
+On the whole I'm satisfied with this, and the benefits definitely outweigh
+the costs. Thanks to the dpkg team for all their work on this!
--- /dev/null
+Title: Tissue of lies
+Slug: tissue-of-lies
+Date: 2009-11-13 17:37:36 +0000
+Modified: 2009-11-13 19:56:01 +0000
+Category: ubuntu
+Tags: ubuntu, planet-debian, planet-ubuntu
+
+In case it isn't obvious, in ["Ubuntu 9.10 SP1 coming in spring
+2010"](http://ubuman.wordpress.com/2009/11/13/ubuntu-9-10-sp1-coming-in-spring-2010/),
+"Ubuman" is blatantly lying in attributing a number of statements to me.
+None of the text there was written by me, and if you thought any of it was
+true then you should probably make sure your troll radar is working
+properly. Nice joke, but try harder next time - it doesn't even look like
+my writing style.
+
+(I wouldn't normally bother to respond, since I'm probably just giving it
+more publicity, but apparently one or two people may already have been taken
+in by it. One person was sensible enough to write to me and check the
+facts.)
--- /dev/null
+Title: Totem BBC plugin
+Slug: totem-bbc-plugin
+Date: 2008-10-27 12:40:48 +0000
+Category: ubuntu
+Tags: planet-ubuntu
+
+A while back, the [BBC](http://www.bbc.co.uk/) approached
+[Canonical](http://www.canonical.com/) about providing seamless access to
+unencumbered BBC content for all [Ubuntu](http://www.ubuntu.com/) users (in
+the UK and elsewhere). We agreed to approach this by way of a plugin for
+our primary media player, [Totem](http://www.gnome.org/projects/totem/), and
+asked [Collabora Multimedia](http://www.collabora.co.uk/) to do the plugin
+development work.
+
+The results are in what will shortly be released as Ubuntu 8.10, and are
+looking really rather good. At the moment the content available from the
+BBC at present is mostly audio, but support for video is in place and the
+feed is expected to be fleshed out here over time. We have a genre
+classification scheme in place, and will see how that scales as the amount
+of available content grows. The code has been [submitted
+upstream](http://bugzilla.gnome.org/show_bug.cgi?id=555823), although there
+are still a few issues to work out there.
+
+This is not the same thing as iPlayer; all the content available here is
+DRM-free. Some of it is geographically restricted to the UK, and these
+restrictions are handled on the server side to make sure that the client is
+free of encumbrances.
+
+[Christian Schaller](http://blogs.gnome.org/uraeus/) from Collabora
+[posted](http://blogs.gnome.org/uraeus/2008/10/08/868/) about this a little
+while ago. Since then, the UI has been improved somewhat and some I/O
+issues have been fixed to the point where we felt comfortable enabling the
+BBC plugin (as well as the YouTube plugin) by default in Ubuntu 8.10.
+Here's a [screenshot](http://people.ubuntu.com/~cjwatson/totem-bbc-2.png) of
+the current interface.
+
+This is exciting stuff with a lot of potential. To try it out, run
+Applications -> Sound & Video -> Movie Player and select the "BBC" entry
+from the drop-down box labelled "Playlist". If you find bugs, please
+[report](https://bugs.launchpad.net/ubuntu/+source/totem/+filebug) them!
--- /dev/null
+Title: UTF-8 manual pages
+Slug: utf-8-manual-pages
+Date: 2008-01-29 01:57:51 +0000
+Category: man-db
+Tags: man-db, planet-debian, planet-ubuntu
+
+See [Encodings in man-db]({filename}/man-db-encodings.md) for context.
+
+Yesterday, I uploaded [man-db
+2.5.1-1](http://lists.debian.org/debian-devel-changes/2008/01/msg02665.html)
+to unstable. With this version, not only is it possible to install manual
+pages in UTF-8 (as with 2.5.0, although with fewer bugs), but it's also
+possible to ask man to produce a version of an arbitrary page in the
+encoding of your choice, and have it guess the source encoding for you
+fairly reliably. This finally provides enough support to have debhelper
+[automatically recode manual pages to
+UTF-8](http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=462937).
+
+It'll probably take a little while to shake out the corner-case bugs, but
+I'm generally pretty happy with this. Once the new man-db and debhelper
+land in testing, I'll send a note to debian-devel-announce and push harder
+on my [policy
+amendment](http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=440420).
+
+Considering the historical state of man-db when it comes to localisation,
+and all of the dependencies and general yak-shaving that had to be tackled
+to get here, this represents the end of probably several hundred hours of
+work, so I'm pretty happy that this is out the door. The only remaining
+step is to add UTF-8 input support to groff, which fortunately Brian M.
+Carlson is
+[working](http://lists.gnu.org/archive/html/groff/2007-11/msg00018.html)
+[on](http://lists.gnu.org/archive/html/groff/2008-01/msg00004.html). After
+that, we can reasonably claim to have dragged manual pages kicking and
+screaming into the 21st century.
--- /dev/null
+Title: Vim omni completion for Launchpad bugs
+Slug: vim-lpbug-omnicomplete
+Date: 2008-01-31 11:17:58 +0000
+Modified: 2008-01-31 11:19:27 +0000
+Category: ubuntu
+Tags: planet-debian, planet-ubuntu
+
+I hacked together a little timesaver for developers this morning: omni
+completion for Launchpad bugs in Vim's debchangelog mode. To use it,
+install vim 7.1-138+1ubuntu3 once it hits the mirrors, open up a
+`debian/changelog` file, type "LP: #", and hit Ctrl-X Ctrl-O. It'll think
+for a while and then give you a list of all the bugs open in Launchpad
+against the package in question, from which you can select to insert the bug
+number into your changelog.
+
+Here's a screenshot to make it clearer:
+
+
+
+Thanks to Stefano Zacchiroli for doing the same for Debian bugs back in
+July.
--- /dev/null
+Title: Windows applications making GRUB 2 unbootable
+Slug: windows-applications-making-grub2-unbootable
+Date: 2010-08-28 00:47:21 +0100
+Category: grub
+Tags: grub, planet-debian, planet-ubuntu
+
+If you find that running Windows makes a GRUB 2-based system unbootable
+([Debian bug](http://bugs.debian.org/550702), [Ubuntu
+bug](https://bugs.launchpad.net/bugs/441941)), then I'd like to hear from
+you. This is a bug in which some proprietary Windows-based software
+overwrites particular sectors in the gap between the master boot record and
+the first partition, sometimes called the "embedding area". GRUB Legacy and
+GRUB 2 both normally use this part of the disk to store one of their key
+components: GRUB Legacy calls this component Stage 1.5, while GRUB 2 calls
+it the core image
+([comparison](http://www.gnu.org/software/grub/manual/grub.html#Images)).
+However, Stage 1.5 is less useful than the core image (for example, the
+latter provides a rescue shell which can be used to recover from some
+problems), and is therefore rather smaller: somewhere around 10KB vs. 24KB
+for the common case of ext[234] on plain block devices. It seems that the
+Windows-based software writes to a sector which is after the end of Stage
+1.5, but before the end of the core image. This is why the problem appears
+to be new with GRUB 2.
+
+At least some occurrences of this are with software which writes a signature
+to the embedding area which hangs around even after uninstallation (even
+with one of those tools that tracks everything the installation process did
+and reverses it, I gather), so that you cannot uninstall and reinstall the
+application to defeat a trial period. This seems like a fine example of an
+[antifeature](http://wiki.mako.cc/Antifeatures), especially given its
+destructive consequences for free software, and is in general a poor piece
+of engineering; what happens if multiple such programs want to use the same
+sector, I wonder? They clearly aren't doing much checking that the sector
+is unused, not that that's really possible anyway. While I do not normally
+think that GRUB should go to any great lengths to accommodate proprietary
+software, this is a case where we need to defend ourselves against the
+predatory practices of some companies making us look bad: a relatively small
+number of people do enough detective work to realise that it's the fault of
+a particular Windows application, but many more simply blame our operating
+system because it won't start any more.
+
+I believe that it may be possible to assemble a collection of signatures of
+such software, and arrange to avoid the disk sectors they have stolen.
+Indeed, I have a first draft of the necessary code. This is not a
+particularly pleasant solution, but it seems to be the most practical way
+around the problem; I'm hoping that several of the programs at fault are
+using common "licence manager" code or something like that, so that we can
+address most of the problems with a relatively small number of signatures.
+In order to do this, I need to hear from as many people as possible who are
+affected by this problem.
+
+If you suffer from this problem, then please do the following:
+
+ * Save the output of `fdisk -lu` to a file. In this output, take note of
+ the start sector of the first partition (usually 63, but might also be
+ 2048 on recent installations, or occasionally something else). If this
+ is something other than 63, then replace 63 in the following items with
+ your number.
+ * Save the contents of the embedding area to a file (replace `/dev/sda`
+ with your disk device if it's something else): `dd if=/dev/sda of=sda.1
+ count=63`
+ * Do whatever you do to make GRUB unbootable (presumably starting Windows),
+ then boot into a recovery environment. Before you reinstall GRUB, save
+ the new contents of the embedding area to a different file: `dd
+ if=/dev/sda of=sda.2 count=63`
+ * Follow up to either the Debian or the Ubuntu bug with these three files
+ (the output of `fdisk -lu`, and the embedding area before and after
+ making GRUB unbootable.
+
+I hope that this will help me to assemble enough information to fix this bug
+at least for most people, and of course if you provide this information then
+I can make sure to fix your particular version of this problem. Thanks in
+advance!
--- /dev/null
+Title: Wubi bug 693671
+Slug: wubi-bug-693671
+Date: 2011-03-14 12:56:57 +0000
+Modified: 2011-03-15 10:12:09 +0000
+Category: grub
+Tags: grub, ubuntu, planet-debian, planet-ubuntu
+
+I spent most of last week working on [Ubuntu bug
+693671](https://bugs.launchpad.net/bugs/693671) ("wubi install will not boot
+- phase 2 stops with: Try (hd0,0): NTFS5"), which was quite a challenge to
+debug since it involved digging into parts of the Wubi boot process I'd
+never really touched before. Since I don't think much of this is very
+well-documented, I'd like to spend a bit of time explaining what was
+involved, in the hope that it will help other developers in the future.
+
+[Wubi](http://en.wikipedia.org/wiki/Wubi_%28Ubuntu_installer%29) is a system
+for installing Ubuntu into a file in a Windows filesystem, so that it
+doesn't require separate partitions and can be uninstalled like any other
+Windows application. The purpose of this is to make it easy for Windows
+users to try out Ubuntu without the need to worry about repartitioning,
+before they commit to a full installation. Wubi started out as an external
+project, and initially patched the installer on the fly to do all the rather
+unconventional things it needed to do; we integrated it into Ubuntu 8.04
+LTS, which involved turning these patches into proper installer facilities
+that could be accessed using preseeding, so that Wubi only needs to handle
+the Windows user interface and other Windows-specific tasks.
+
+Anyone familiar with a GNU/Linux system's boot process will immediately see
+that this isn't as simple as it sounds. Of course,
+[ntfs-3g](http://www.tuxera.com/community/ntfs-3g-download/) is a pretty
+solid piece of software so we can handle the Windows filesystem without too
+much trouble, and loopback mounts are well-understood so we can just have
+the initramfs loop-mount the root filesystem. Where are you going to get
+the kernel and initramfs from, though? Well, we used to copy them out to
+the NTFS filesystem so that GRUB could read them, but this was overly
+complicated and error-prone. When we switched to GRUB 2, we could instead
+use its built-in loopback facilities, and we were able to simplify this. So
+all was more or less well, except for the elephant in the room. How are you
+going to load GRUB?
+
+In a Wubi installation, NTLDR (or BOOTMGR in Windows Vista and newer) still
+owns the boot process. Ubuntu is added as a boot menu option using BCDEdit.
+You might then think that you can just have the Windows boot loader
+chain-load GRUB. Unfortunately, NTLDR only loads 16 sectors - 8192 bytes -
+from disk. GRUB won't fit in that: the smallest core.img you can generate
+at the moment is over 18 kilobytes. Thus, you need something that is small
+enough to be loaded by NTLDR, but that is intelligent enough to understand
+NTFS to the point where it can find a particular file in the root directory
+of a filesystem, load boot loader code from it, and jump to that. The
+answer for this was [GRUB4DOS](http://gna.org/projects/grub4dos/). Most of
+GRUB4DOS is based on GRUB Legacy, which is not of much interest to us any
+more, but it includes an assembly-language program called GRLDR that
+supports doing this very thing for FAT, NTFS, and ext2. In Wubi, we build
+GRLDR as `wubildr.mbr`, and build a specially-configured GRUB core image as
+`wubildr`.
+
+Now, the messages shown in the bug report suggested a failure either within
+GRLDR or very early in GRUB. The first thing I did was to remember that
+GRLDR has been integrated into the grub-extras `ntldr-img` module suitable
+for use with GRUB 2, so I tried building `wubildr.mbr` from that; no change,
+but this gave me a modern baseline to work on. OK; now to try QEMU (you can
+use tricks like `qemu -hda /dev/sda` if you're very careful not to do
+anything that might involve writing to the host filesystem from within the
+guest, such as recursively booting your host OS ... [**update:** Tollef Fog
+Heen and Zygmunt Krynicki both point out that you can use the `-snapshot`
+option to make this safer]). No go; it hung somewhere in the middle of
+NTLDR. Still, I could at least insert debug statements, copy the built
+`wubildr.mbr` over to my test machine, and reboot for each test, although it
+would be slow and tedious. Couldn't I?
+
+Well, yes, I mostly could, but that 8192-byte limit came back to bite me,
+along with an internal 2048-byte limit that GRLDR allocates for its NTFS
+bootstrap code. There were only a few spare bytes. Something like this
+would more or less fit, to print a single mark character at various points
+so that I could see how far it was getting:
+
+ :::gas
+ pushal
+ xorw %bx, %bx /* video page 0 */
+ movw $0x0e4d, %ax /* print 'M' */
+ int $0x10
+ popal
+
+In a few places, if I removed some code I didn't need on my test machine
+(say, CHS compatibility), I could even fit in cheap and nasty code to print
+a single register in hex (as long as you didn't mind 'A' to 'F' actually
+being ':' to '?' in ASCII; and note that this is real-mode code, so the loop
+counter is `%cx` not `%ecx`):
+
+ :::gas
+ /* print %edx in dumbed-down hex */
+ pushal
+ xorw %bx, %bx
+ movb $0xe, %ah
+ movw $8, %cx
+ 1:
+ roll $4, %edx
+ movb %dl, %al
+ andb $0xf, %al
+ int $0x10
+ loop 1b
+ popal
+
+After a considerable amount of work tracking down problems by bisection like
+this, I also observed that GRLDR's NTFS code bears quite a bit of
+resemblance in its logical flow to GRUB 2's NTFS module, and indeed the same
+person wrote much of both. Since I knew that the latter worked, I could use
+it to relieve my brain of trying to understand assembly code logic directly,
+and could compare the two to look for discrepancies. I did find a few of
+these, and corrected a simple one. Testing at this point suggested that the
+boot process was getting as far as GRUB but still wasn't printing anything.
+I removed some Ubuntu patches which quieten down GRUB's startup: still
+nothing - so I switched my attentions to
+[grub-core/kern/i386/pc/startup.S](http://bazaar.launchpad.net/~vcs-imports/grub/grub2-bzr/view/head:/grub-core/kern/i386/pc/startup.S),
+which contains the first code executed from GRUB's core image. Code before
+the first call to `real_to_prot` (which switches the processor into
+protected mode) succeeded, while code after that point failed. Even more
+mysteriously, code added to `real_to_prot` *before* the actual switch to
+protected mode failed too. Now I was clearly getting somewhere interesting,
+but what was going on? What I really wanted was to be able to single-step,
+or at least see what was at the memory location it was supposed to be
+jumping to.
+
+Around this point I was venting on IRC, and somebody asked if it was
+reproducible in QEMU. Although I'd tried that already, I went back and
+tried again. Ubuntu's `qemu` is actually built from qemu-kvm, and if I used
+`qemu -no-kvm` then it worked much better. Excellent! Now I could use GDB:
+
+ (gdb) target remote | qemu -gdb stdio -no-kvm -hda /dev/sda
+
+This let me run until the point when NTLDR was about to hand over control,
+then interrupt and set a breakpoint at `0x8200` (the entry point of
+`startup.S`). This revealed that the address that should have been
+`real_to_prot` was in fact garbage. I set a breakpoint at `0x7c00` (GRLDR's
+entry point) and stepped all the way through to ensure it was doing the
+right thing. In the process it was helpful to know that [GDB and QEMU don't
+handle real mode very well between
+them](http://sourceware.org/ml/gdb/2009-01/msg00008.html). Useful tricks
+here were:
+
+ * Use `set architecture i8086` before disassembling real-mode code (and
+ `set architecture i386` to switch back).
+ * GDB prints addresses relative to the current segment base, but if you
+ want to enter an address then you need to calculate a linear address
+ yourself. For example, breakpoints must be set at `(CS << 4) +
+ IP`, rather than just at `IP`.
+
+Single-stepping showed that GRLDR was loading the entirety of `wubildr`
+correctly and jumping to it. The first instruction it jumped to wasn't in
+`startup.S`, though, and then I remembered that we prefix the core image
+with
+[grub-core/boot/i386/pc/lnxboot.S](http://bazaar.launchpad.net/~vcs-imports/grub/grub2-bzr/view/3095/grub-core/boot/i386/pc/lnxboot.S).
+Stepping through this required a clear head since it copies itself around
+and changes segment registers a few times. The interesting part was at
+`real_code_2`, where it copies a sector of the kernel to the target load
+address, and then checks a known offset to find out whether the "kernel" is
+in fact GRUB rather than a Linux kernel. I checked that offset by hand, and
+there was the smoking gun. GRUB recently acquired Reed-Solomon error
+correction on its core image, to allow it to recover from other software
+writing over sectors in the boot track. This moved the magic number
+`lnxboot.S` was checking somewhat further into the core image, after the
+first sector. `lnxboot.S` couldn't find it because it hadn't copied it yet!
+A bit of
+[adjustment](http://bazaar.launchpad.net/~vcs-imports/grub/grub2-bzr/revision/3096)
+and all was well again.
+
+The lesson for me from all of this has been to try hard to get an
+interactive debugger working. Really hard. It's worth quite a bit of
+up-front effort if it saves you from killing neurons stepping through pages
+of code by hand. I think the real-mode debugging tricks I picked up should
+be useful for working on GRUB in the future.
+++ /dev/null
-Forwarding bugs to the IETF
-
-<p>Sometimes following up on a bug takes you a lot further than you
-expected. <a href="http://bugs.debian.org/337041">Debian bug #337041</a>
-looked like it was going to be fairly straightforward once I upgraded
-coreutils to figure out what the
-<a href="http://www.cl.cam.ac.uk/~mgk25/unicode.html#mod">new IUTF8 flag</a>
-actually did, since the SSH2 protocol already supports transferring termios
-flags around.</p>
-
-<p>Unfortunately, since IUTF8 is relatively new, it doesn't have a number
-assigned in the
-<a href="http://www.ietf.org/internet-drafts/draft-ietf-secsh-connect-25.txt">
-draft connection protocol</a>. Moreover, that Internet-Draft is in the last
-stages before becoming an RFC and can't be modified any more, and it doesn't
-include any facility for private-use extensions. D'oh. To add further
-complication, since IUTF8 is Linux-specific, it's not hard to imagine that
-some other OS might introduce something with the same name but subtly
-different semantics, and so the SSH protocols can't just defer to POSIX for
-the definition but instead have to spell out exactly what they mean.</p>
-
-<p>As a result of all of this, it looks like the best way to make progress
-might be for me to write an I-D myself that creates a channel extension to
-set or clear IUTF8, and attempt to enlist support from some upstream
-implementors. I didn't expect bug triage to lead me into the Internet
-standardisation process quite so quickly!</p>
+++ /dev/null
-debconf/cdebconf coinstallability
-
-<p><a href="http://kitenet.net/~joey/blog/">Joey</a> has been
-<a href="http://lists.debian.org/debian-devel/2005/08/msg00136.html">campaigning</a>
-for a while to get everything in the archive changed to depend on
-<code>debconf | debconf-2.0</code> or similar rather than just
-<code>debconf</code>, in order that we can start rolling out
-<code>cdebconf</code> as its replacement. Like most jobs that involve
-touching the bulk of the archive, this looks set to take quite a while, as
-the <a href="http://bugs.debian.org/328498">list of bugs</a> should
-indicate.</p>
-
-<p>Recently it occurred to me that we didn't necessarily have to do it that
-way round. In a bout of late-night hacking while staying awake to look after
-a sick child (he seems mostly OK now, although the rushed trip to the
-hospital earlier was a bit on the nerve-wracking side), I've shuffled things
-around in the cdebconf package so that it no longer has any file conflicts
-with debconf or debconf-doc, and changed the debconf confmodule to fire up
-the cdebconf frontend rather than its own if the
-<kbd>DEBCONF_USE_CDEBCONF</kbd> environment variable is non-empty. (The
-details of this may change before it actually gets uploaded, as I'd like to
-get Joey to look it over and approve it first.) This allows you to install
-cdebconf, set that environment variable, and play around with cdebconf with
-relative ease; when we come to switch to cdebconf for real, instead of a
-huge conflicting mess that apt will probably have trouble resolving, it'll
-just be a matter of changing a couple of lines in
-<code>/usr/share/debconf/confmodule</code>.</p>
-
-<p>Of course, don't expect cdebconf to be a complete working replacement for
-debconf just yet; if you try using it for a dist-upgrade run it'll fall
-over. Due to its d-i heritage, it doesn't yet load templates automatically;
-that has to be done by hand. Frontend names differ from debconf's, which
-will need some migration code. At the moment it can only handle UTF-8
-templates, which are mandated in the installer but only optional in the rest
-of the system. It doesn't have all of debconf's rich array of database
-modules. I haven't adapted the Perl or Python confmodules yet. The list goes
-on. However, I think we at least stand a chance of getting a handle on the
-problem now.</p>
-
-<p>(I'll post this article to debian-devel once the changes have been
-reviewed and uploaded.)</p>
+++ /dev/null
-aptitude safe-upgrade
-
-<p><a href="http://blog.drinsama.de/erich/en/linux/debian/2007112801-dist-upgrade-hints.html">Erich</a>:
-I do sometimes wonder why we don't relax the definition of "safe" upgrades
-to include installing new packages but still not removing old ones. I know
-that many of my uses of dist-upgrade are just for when something grows a new
-dependency that I didn't previously have installed.</p>
-
-<p>(Of course this wouldn't always help as it wouldn't account for a new
-dependency that conflicted with an old dependency, but never mind. It would
-certainly do wonders for the metapackage case.)</p>
+++ /dev/null
-Re: Perl is strange
-
-<p><a href="http://www.df7cb.de/blog/2008/Perl_is_strange.html">Christoph</a>:
-That's because <code>=~</code> binds more tightly than <code>+</code>. This
-does what you meant:</p>
-
-<pre>
-$ perl -le 'print "yoo" if (1 + 1) =~ /3/'
-</pre>
-
-<p>perlop(1) has a useful table of precedence.</p>
+++ /dev/null
-Catching up
-
-<p>I did a bit of catching up on my Debian backlog over the last week or so. Among the things I got round to:</p>
-
-<ul>
- <li>I released man-db 2.5.7. This was mostly an "I've been meaning to do this for ages" kind of thing to reduce the bug list a bit, closing ten Debian bugs, but there were a few interesting things in there as well, such as always saving cat pages in UTF-8 and recoding to the user's locale at display time (long overdue), adjusting the search order for localised manual pages by request of quite a few non-native English speakers to prefer a page in the right section over a page in the right language, and a cute gimmick to make things like <code>man /usr/bin/time</code> display the appropriate manual page rather than the text of the executable. See the <a href="http://bazaar.launchpad.net/~cjwatson/man-db/trunk/annotate/head%3A/NEWS">NEWS file</a> for more details.</li>
- <li>binfmt-support now <a href="http://bugs.debian.org/565109">installs cleanly on non-Linux systems</a>, even if it doesn't do anything useful yet.</li>
- <li>I fixed a couple of <a href="http://bugs.debian.org/256226">shell</a> <a href="http://bugs.debian.org/547750">bugs</a> in groff.</li>
- <li>halibut now <a href="http://bugs.debian.org/464821">complies with the Debian Vim policy</a>, even though I can't say I entirely agree with it in this case.</li>
- <li>I fixed a <a href="http://lists.debian.org/debian-devel-changes/2010/02/msg02219.html">really odd build failure in troffcvt</a>. Yay imake, or something.</li>
- <li>All Debian patches to putty are now upstream, or will be once I upload a new snapshot. Thanks to Simon Tatham and Jacob Nevins.</li>
- <li>I did a few bits and pieces of packaging cleanup with an eye on my <a href="http://qa.debian.org/developer.php">DDPO</a> list, and added some watch files where they were missing.</li>
- <li>Responded to an offer to take over icoutils maintenance.</li>
-</ul>
-
-<p>So nothing really earth-shaking, and as ever <a href="http://lists.debian.org/debian-ssh/2010/01/msg00017.html">openssh could use some attention</a>, but I feel a bit better about my backlog now. I do still have a <a href="http://bugs.debian.org/564559">critical bug in makepasswd</a> to fix, and a sponsored upload of parrot; those are the next two things on my to-do list.</p>
+++ /dev/null
-debhelper statistics
-
-<p>I don't know if anyone else has been tracking this recently, but a while back I got curious about the relative proportions of dh(1) and CDBS in the archive, and started running some daily analysis on the Lintian lab. Apologies for my poor graphing abilities, but the graph is here (occasionally updated):</p>
-
-<p><img src="http://people.debian.org/~cjwatson/dhstats.png" /></p>
-
-<p>Although dh is still a bit behind CDBS, the steady upward trend is quite striking - it looks set to break 20% soon, up from under 13% in September - compared with CDBS which has been sitting within half a percentage point of 25% the whole time.</p>
-
-<p>Incidentally, was that an ftpmaster trying to sign his name in the graph over Christmas or something? :-)</p>
+++ /dev/null
-parted 2.2 transition
-
-<p>I've started the <a href="http://lists.debian.org/debian-release/2010/03/msg00121.html">transition of parted 2.2 to unstable</a>. This is a major update needed for sensible support of newer hard disks with alignment requirements different from the archaic cylinder alignment tradition. I posted to debian-boot with a <a href="http://lists.debian.org/debian-boot/2010/03/msg00420.html">summary of the partman changes involved</a>.</p>
+++ /dev/null
-Thoughts on 3.0 (quilt) format
-
-<p>Note: I wrote most of this before <a href="http://www.linux.codehelp.co.uk/serendipity/index.php?/archives/201-lintian,-source-format-3.0-and-blog-comments.html">Neil Williams' recent comments on the 3.0 family of formats</a>, so despite the timing this isn't really a reaction to that although I do have a couple of responses. On the whole I think I agree that the Lintian message is a bit heavy-handed and I'm not sure I'm thrilled about the idea of the default source format being changed (though I can see why the dpkg maintainers are interested in that). That said, as far as I personally am concerned, there is a vast cognitive benefit to me in having as much as possible be common to all my packages. Once I have more than a couple of packages that require patching and benefit from the <code>3.0 (quilt)</code> format as a result, I find it in my interest to use it for all my non-native packages even if they're patchless right now, so that for instance if they need patches in the future I can handle them the same way. It's not unheard of for me to apply temporary patches even to packages I actively maintain upstream, so I don't discount those either. I haven't decided what to do with my native packages yet; unless they're big enough for bzip2 compression to be worthwhile, there doesn't seem to be much immediate advantage to <code>3.0 (native)</code>.</p>
-
-<p>Anyway, on to the main body of this post:</p>
-
-<p>I've been one of the holdouts resisting use of patch systems for a long time, on the basis that I felt strongly that <code>dpkg-source -x</code> ought to give you the source that's actually built, rather than having to mess around with <code>debian/rules</code> targets in order to see it. Now that the <code>3.0 (quilt)</code> format is available to fix this bug, I felt that I ought to revisit my resistance and start trying to use it. Migrating to it from monolithic diffs is of course a bit more work than migrating to it from other patch systems, so it's taken me a little while to get round to it. I'd been thinking about holding off until there was better integration with revision control (e.g. bzr looms), as I feel that patch files really ought to be an export format, but I eventually decided that I shouldn't let the perfect be the enemy of the good. I have enough experience with co-maintaining packages that use build-time patch systems to be able to compare my reactions.</p>
-
-<p>After experimenting with a couple of small packages, I moved over to the deep end and <a href="http://packages.qa.debian.org/o/openssh/news/20100228T035004Z.html">converted openssh</a> a few weekends ago, since quite a few people have requested over the years that the Debian changes to openssh be easier to audit. This was a substantial job - over 6000 lines of upstream patches - but not actually as much work as I expected. I took a fairly simplistic approach: first, I unapplied all the upstream patches from my tree; then I ran <code>bzr di | interdiff -q /dev/stdin /dev/null >x</code>, reduced it to a single logically-discrete patch, applied it to a new quilt patch using <code>quilt fold</code>, and repeated until <code>x</code> was empty. This was maybe an hour or two of work, and then I went through and tagged all the patches according to <a href="http://dep.debian.net/deps/dep3/">DEP-3</a>, which took another few hours. After the first pass, I ended up with 38 patches and a much clearer idea of what has been forwarded upstream and what hasn't; I currently have 5 patches to forward or eliminate, down from 18.</p>
-
-<p>Good things:</p>
-
-<ul>
- <li>I don't lose any of my history. Since all the patches remain applied to the tree in revision control (this is what <code>dpkg-source -x</code> gives you, so it's the natural representation in revision control too), <code>bzr blame</code> works just as you'd expect and displays both upstream and Debian changes at once. I rely on tools like blame a lot, and I really hate the way build-time patch systems make it hard to use revision control when the tree is in a built state, so this was a hard requirement for me.</li>
- <li>I've used patch tagging before, so I was expecting some benefits, but viscerally I feel much more in <em>control</em>. It's so much less laborious now to see what I need to do by way of forwarding. I don't regret waiting for 3.0 (quilt) to become available, but I hadn't realised quite how much I was being held back beforehand.</li>
- <li>Adding new patches is pretty natural, much more so than with build-time patch systems. You can create and apply the patch, test-build, and commit when it works. I much prefer this over having to clean the tree before committing (or commit just part of the tree, which is error-prone). The more that committing to a Debian package feels like committing to an upstream project, the better.</li>
- <li>There's definitely something to be said for <a href="http://patch-tracker.debian.org/package/openssh">patch-tracker</a> being more useful. It deals with DEP-3 to the extent of linkifying URLs, although it might be nice if patch descriptions were displayed on the overview page for each version.</li>
-</ul>
-
-<p>Bad things:</p>
-
-<ul>
- <li>It's a bit awkward to set things up when checking out from revision control; I didn't really want to check in the <code>.pc</code> directory, and the tree checks out in the patched state (as it should), so I needed some way for developers to get quilt working easily after a checkout. This is sort of the reverse of the previous problem, where users had to do something special after <code>dpkg-source -x</code>, and I consider it less serious so I'm willing to put up with it. I ended up with <a href="http://bugs.debian.org/572204">a rune in debian/rules that ought to live somewhere more common</a>.</li>
- <li>Everything ends up represented twice in revision control: the patch files, plus the changes to the patched files themselves. I'm OK with this although it is a little inelegant.</li>
- <li>Although I haven't had to do it yet, I expect that merging new upstream releases will be a bit harder. bzr will deal with resolving conflicts in the patched files themselves, and that's why I use a revision control system after all, but then I'll have to go and refresh all the patches and will probably end up doing some of the same conflict resolution a second time. I think the best answer right now is to <code>quilt pop -a</code>, force a merge despite the modified working tree, and then <code>quilt push && quilt refresh -pab</code> until I get back to the top of the stack, modulo slight fiddliness when a patch disappears entirely; thus effectively using quilt's conflict resolution rather than bzr's. I suppose this will serve as additional incentive to reduce my patch count. I know that people have been working on making this work nicely with topgit, although I'm certainly not going to put up with the rest of git due to that; I'm happy to wait for looms to become usable and integrated. :-)</li>
- <li>It would be nice if there were some standard DEP-3 way to note that a patch has been accepted or rejected upstream, beyond just putting it in the description. In particular, it seems to me that listing patches accepted upstream could be used to speed up the process of merging new upstream releases.</li>
-</ul>
-
-<p>On the whole I'm satisfied with this, and the benefits definitely outweigh the costs. Thanks to the dpkg team for all their work on this!</p>
+++ /dev/null
-Hacking on grub2
-
-<p>Various people observed in a <a href="http://lists.debian.org/debian-devel/2010/05/msg00769.html">long thread on debian-devel</a> that the grub2 package was in a bit of a mess in terms of its release-critical bug count, and <a href="http://oskuro.net/blog">Jordi</a> and <a href="http://upsilon.cc/~zack/blog/planet-debian/">Stefano</a> both got in touch with me directly to gently point out that I probably ought to be doing something about it as one of the co-maintainers.</p>
-
-<p>Actually, I don't think grub2 was in quite as bad a state as its 18 RC bugs suggested. Of course every boot loader failure is critical to the person affected by it, not to mention that GRUB 2 offers more complex functionality than any other boot loader (e.g. LVM and RAID), and so it tends to accumulate RC bugs at rather a high rate. That said, we'd been neglecting its bug list for some time; <a href="http://robertmh.wordpress.com/">Robert</a> and Felix have both been taking some time off, Jordi mostly only cared about PowerPC and can't do that any more due to hardware failure, and I hadn't been able to pick up the slack.</p>
-
-<p>Most of my projects at <a href="http://www.ubuntu.com/">work</a> for the next while involve GRUB in one way or another, so I decided it was a perfectly reasonable use of work time to do something about this; I was going to need fully up-to-date snapshots anyway, and practically all the Debian grub2 bugs affect Ubuntu too. Thus, with the exception of some other little things like releasing the first Maverick alpha, I've spent pretty much the last week and a half solidly trying to get the grub2 package back into shape, with four uploads so far.</p>
-
-<p>The RC issues that remain are:</p>
-
-<ul>
- <li>
- <code>upgrade-from-grub-legacy</code> problems (<a href="http://bugs.debian.org/547944">#547944</a>, <a href="http://bugs.debian.org/550477">#550477</a>):<br />
- I think this has just been traditionally undertested. I'm setting up a KVM image now with GRUB Legacy which I can snapshot just before and after running <code>upgrade-from-grub-legacy</code>, and I should be able to unpick the bugs this way.
- </li>
- <li>
- LVM snapshots break GRUB's LVM module (<a href="http://bugs.debian.org/574863">#574863</a>):<br />
- <a href="http://www.seanius.net/feeds/planet-debian/">Sean</a> has been working on this and seems to be nearly there. Yay.
- </li>
- <li>
- RAID metadata version 1.x not supported (<a href="http://bugs.debian.org/492897">#492897</a>):<br />
- This became rather more of an issue recently since <code>mdadm</code> switched its default from the old 0.90 format which GRUB understood. Felix put together a branch implementing the hard parts of this a while back, and I've been trying to finish it off. The hard bit is dealing with device naming, especially as the new-format and rather more useful names under <code>/dev/md/</code> don't show up during <a href="http://www.debian.org/devel/debian-installer">d-i</a> after creating RAID volumes; I think this is because we always create them as <code>/dev/md0</code> etc. It's looking tractable, though.
- </li>
- <li>
- Another odd problem probing RAID (<a href="http://bugs.debian.org/548648">#548648</a>):<br />
- Not sure about this one, and I'll need to work with Josip on it as soon as I get a chance.
- </li>
- <li>
- Stable device naming <a href="http://bugs.debian.org/554790">#554790</a>) and consequential problems due to <code>grub-install</code> not being properly run (<a href="http://bugs.debian.org/557425">#557425</a> and many other sub-RC bugs):<br />
- Ubuntu's been carrying a patch to rearrange device presentation in the postinst, which Robert OKed in principle ages ago and so I've been intending to merge it for a while, but there are a few known problems with it that I need to fix first. One known unfixable problem is that it will have to ask some people which devices they want GRUB to be installed on, even if they'd answered that question before: this will be one-time, and it's because it recorded the answer using unstable device names and so has in some sense forgotten. Simple cases (e.g. single-disk) can be handled without needing to ask again, though.
- </li>
- <li>
- Alignment errors on SPARC (<a href="http://bugs.debian.org/560823">#560823</a>):<br />
- I have no idea what's going on here, I'm afraid. I'll try to trace it, but may have to downgrade it at some point since after all we don't install GRUB by default on SPARC yet.
- </li>
- <li>
- Fonts not shown in gfxmenu (<a href="http://bugs.debian.org/564844">#564844</a>):<br />
- Apparently fixed upstream, but I couldn't find the responsible commit so I want to make sure I can get gfxmenu working before closing this.
- </li>
- <li>
- Sensitivity to out-of-date <code>device.map</code> files (<a href="http://bugs.debian.org/575076">#575076</a> and other sub-RC bugs):<br />
- We're trying to get rid of <code>device.map</code> in general. It was fine in the 1990s but it's hopeless now. Unfortunately there are still a small number of problems with running entirely without one, and one of my patches to help is controversial upstream, so we probably won't get to that for squeeze. In the meantime we'll probably just need some extra sanity-checking and robustness in the event that there's an incorrect or out-of-date <code>device.map</code> lying around, which we may just be able to do in the maintainer scripts or something if necessary.
- </li>
- <li>
- Seriously weird failures to load initramfs (<a href="http://bugs.debian.org/582342">#582342</a>):<br />
- If anyone can produce a reproduction recipe for this, that would really help me out. There are too many reports to discount as user error, but I haven't seen this myself yet.
- </li>
- <li>
- Build failure on sparc (unfiled):<br />
- We've been discussing this upstream, but for the time being I'm just going to stop building <code>grub-emu</code> on sparc as a workaround.
- </li>
-</ul>
-
-<p>If we can fix that lot, or even just the ones that are reasonably well-understood, I think we'll be in reasonable shape. I'd also like to make <code>grub-mkconfig</code> a bit more robust in the event that the root filesystem isn't one that GRUB understands (<a href="http://bugs.debian.org/561855">#561855</a>, <a href="http://bugs.debian.org/562672">#562672</a>), and I'd quite like to write some more documentation.</p>
-
-<p>On the upside, progress has been good. We have multiple terminal support thanks to a new upstream snapshot (<a href="http://bugs.debian.org/506707">#506707</a>), <code>update-grub</code> runs much faster (<a href="http://bugs.debian.org/508834">#508834</a>, <a href="http://bugs.debian.org/574088">#574088</a>), we have DM-RAID support with a following wind (<a href="http://bugs.debian.org/579919">#579919</a>), the new scheme with symlinks under <code>/dev/mapper/</code> works (<a href="http://bugs.debian.org/550704">#550704</a>), we have basic support for btrfs <code>/</code> as long as you have something GRUB understands properly on <code>/boot</code> (<a href="http://bugs.debian.org/540786">#540786</a>), we have full info documentation covering all the user-adjustable settings in <code>/etc/default/grub</code>, and a host of other smaller fixes. I'm hoping we can keep this up.</p>
-
-<p>If you'd like to help, contact me, especially if there's something particular that isn't being handled that you think you could work on. GRUB 2 is actually quite a pleasant codebase to work on once you get used to its layout; it's certainly much easier to fix bugs in than GRUB Legacy ever was, as far as I'm concerned. Thanks to tools like <code>grub-probe</code> and <code>grub-fstest</code>, it's very often possible to fix problems without needing to reboot for anything other than a final sanity check (although KVM certainly helps), and you can often debug very substantial bits of the boot loader - the bits that actually go wrong - using standard tools such as <code>strace</code> and <code>gdb</code>. Upstream is helpful and I've been able to get many of the problems above fixed directly there. If you have a sound knowledge of C and a decent level of understanding of the environment a boot loader needs to operate in - or for that matter specialist knowledge of interesting device types - then you should be able to find something to do.</p>
+++ /dev/null
-GRUB 2 boot problems
-
-<p>(This is partly a repost of material I've posted to bug reports and to debian-release, put together with some more detail for a wider audience.)</p>
-
-<p>You could be forgiven for looking at the RC bug activity on <a href="http://bugs.debian.org/src:grub2">grub2</a> over the last couple of days and thinking that it's all gone to hell in a handbasket with recent uploads. In fact, aside from an interesting case which turned out to be due to botched handling of the GRUB Legacy to GRUB 2 chainloading setup (which prompted me to fix three other RC bugs along the way), all the recent problems people have been having have been duplicates of one of these bugs which have existed essentially forever:</p>
-
-<ul>
- <li><a href="http://bugs.debian.org/554790">#554790 - grub-pc/install_devices uses unstable device names</a></li>
- <li><a href="http://bugs.debian.org/583271">#583271 - device.map uses unstable device names</a></li>
-</ul>
-
-<p>When GRUB boots, its boot sector first loads its "core image", which is usually embedded in the gap between the boot sector and the first partition on the same disk as the boot sector. This core image then figures out where to find /boot/grub, and loads grub.cfg from it as well as more GRUB modules.</p>
-
-<p>The thing that tends to go wrong here is that the core image must be from the same version of GRUB as any modules it loads. /boot/grub/*.mod are updated only by grub-install, so this normally works OK. However, for various reasons (deliberate or accidental) some people install GRUB to multiple disks. In this case, grub-install might update /boot/grub/*.mod along with the core image on one disk, but your BIOS might actually be booting from a different disk. The effect of this will be that you'll have an old core image and new modules, which will probably blow up in any number of possible ways. Quite often, this problem lies dormant for a while because GRUB happens not to change in a way that causes incompatibility between the core image and modules, but then we get massive spikes of bug reports any time the interface does change. Since these bugs sometimes bite people upgrading from testing to unstable, they get interpreted as regressions from the version in testing even though that isn't strictly true (but it tends not to be very productive to argue this line; after all, people's computers suddenly don't boot!). Any problem that causes the core image to be installed to a disk other than the one actually being booted from, or not to be installed at all, will show up this way sooner or later.</p>
-
-<p>On 2010-06-10, there was a substantial upstream change to the handling of list iterators (to reduce core image size and make code clearer and faster) which introduced an incompatibility between old core images and newer modules. This caused a bunch of dormant problems to flare up again, and so there was a flood of reports of booting problems with 1.98+20100614-1 and newer, often described as "the unaligned pointer bug" due to how it happened to manifest this time round. In previous cases, GRUB reported undefined symbols on boot, but it's all essentially the same problem even though there are different symptoms.</p>
-
-<p>The confusing bit when handling bug reports is that not only are there different symptoms with the same cause, but there are also multiple causes for the same symptom! This takes a certain amount of untangling, especially when lots of people have thought "ooh, that bug looks a bit like mine" and jumped in with their own comments. Working through this was a worthwhile exercise, as it came up with an entirely new cause for a problem I thought was fairly well-understood (thanks to debugging assistance from Sedat Dilek). If you had set up GRUB 2 to be automatically chainloaded from GRUB Legacy (which happens automatically on upgrade from the latter to the former), never got round to running <code>upgrade-from-grub-legacy</code> once you confirmed it worked, and then later ran <code>grub-install</code> by hand for one reason or another, then the core image you installed by hand would never be updated and would eventually <a href="http://bugs.debian.org/586143">fall over</a> the next time the core/modules interface changed. Fixing future cases of this was easy enough, but fixing existing cases involved figuring out how to detect whether an installed GRUB boot sector came from GRUB Legacy or GRUB 2, which isn't as easy as you might think. Fortunately, it turns out that there are a limited number of jump offsets that have ever been used in the second byte of the boot sector, and none of the GRUB 2 values clash with the only value ever used in GRUB Legacy; so, if you still have <code>/boot/grub/stage2</code> et al on upgrade, we scan all disks for a GRUB 2 boot sector, and if we find one then we offer to complete the upgrade to GRUB 2.</p>
-
-<p>Unless anything new shows up, that just leaves the problems that were already understood. Today, I posted a <a href="http://lists.gnu.org/archive/html/grub-devel/2010-06/msg00118.html">patch to generate stable device names in device.map by default</a>. If this is accepted, then we can do something or other to fix up device.map on upgrade, switch over to <code>/dev/disk/by-id</code> names in <code>grub-pc/install_devices</code> at the same time, and that should take care of the vast majority of this kind of upgrade bug. I think at that point it should be feasible to get a new version into testing, and we should be down from 18 RC bugs towards the end of last month to around 6. We can then start attacking things like the lack of support for mdadm 1.x metadata.</p>
-
-<p>Since my <a href="http://www.chiark.greenend.org.uk/ucgi/~cjwatson/blosxom/debian/2010-06-04-hacking-on-grub2">last blog entry on GRUB 2</a>, improvements have included:</p>
-
-<ul>
- <li>Substantial work on <code>info grub</code>, with, among other things, new sections on <code>/etc/default/grub</code> and on configuring authentication.</li>
- <li>A workaround for GRUB's inability to probe dm-crypt devices, thanks to Marc Haber.</li>
- <li>Several build fixes for architectures I wasn't testing, and a fix for broken nested partition handling on Debian GNU/kFreeBSD. I'm now testing GNU/kFreeBSD locally.</li>
- <li>Rather less cruft in <code>fs.lst</code>, <code>partmap.lst</code>, and <code>video.lst</code>, which should speed up booting a bit by e.g. avoiding unnecessary filesystem probing.</li>
- <li><code>upgrade-from-grub-legacy</code> actually now installs GRUB 2 to the boot sector (!).</li>
- <li>Ask for confirmation if <code>grub-pc/install_devices</code> is left empty.</li>
-</ul>
-
-<p>The next upstream snapshot will bring several improvements to EFI video support, mainly thanks to Vladimir Serbinenko. I've been working on making <code>grub-install</code> actually work on UEFI systems as one of my goals for the next Ubuntu release, and I hope to get this landed in the not-too-distant future.</p>
+++ /dev/null
-GRUB 2: With luck ...
-
-<p>... this version, or something not too far away from it, might actually stand a chance of getting into testing.</p>
-
-<p>I've just uploaded grub2 1.98+20100702-1. The most significant set of changes in this release is that it switches <code>/boot/grub/device.map</code> and the <code>grub-pc/install_devices</code> debconf question over to stable device names under <code>/dev/disk/by-id</code> (on Linux kernels). The code implementing this is reasonably careful, and it should make it quite difficult for people to accidentally fail to upgrade their installed GRUB core image; I explained the problems that tends to cause in the <a href="http://www.chiark.greenend.org.uk/ucgi/~cjwatson/blosxom/debian/2010-06-21-grub2-boot-problems.txt">previous post in this series</a>. There will probably be a few small glitches I need to clear up, but I've given this much more extensive testing than usual so I hope I won't break too many people's computers (again).</p>
-
-<p>I did this work first in Ubuntu as one of my major goals for 10.04 LTS, which exposed a few problems that I wanted to fix before inflicting it on Debian as well (fixes for those are now under testing for 10.04.1). Most significantly, I felt it was necessary to start offering partitions in the select list for <code>grub-pc/install_devices</code>, but I went a bit overboard and offered all partitions in a giant list. This seemed like a good idea at the time, but it tended to confuse people into just selecting everything in the list, which in particular tended to make Windows unbootable! So I dialled that back a bit, and in the version I just merged it will only offer the partitions mounted on <code>/</code>, <code>/boot</code>, and <code>/boot/grub</code> (de-duplicating if necessary). This seems like a reasonable compromise between confusing people too much and forcing them to install only to MBRs.</p>
-
-<p>My next priority will be making whatever fixes are necessary to get this version into testing, since the problems with <code>/dev/mapper</code> symlinks in testing aren't getting any less urgent, and this is finally a version that shouldn't break for most people due to the kernel's switch to libata. I expect that I'll try to get mdadm 1.x metadata sorted out immediately after that.</p>
-
-<p>Other improvements since my last entry have included:</p>
-
-<ul>
- <li>Further documentation work. Thanks to Vladimir Serbinenko (and to Jordan Uggla for hosting it temporarily), there's now an <a href="http://www.gnu.org/software/grub/manual/">HTML version of the GRUB manual from trunk</a> online, which includes new sections on embedded configuration files, the various GRUB image files, <code>device.map</code>, and (shortly) a summary of changes from GRUB Legacy.</li>
- <li>Video improvements: among other things, UEFI systems whose firmware uses the Graphics Output Protocol should now work rather better, and GRUB now includes specific support for some cards often used with minimal firmware support under emulation.</li>
- <li>A fix to handle large memory maps exposed by some UEFI firmware.</li>
- <li>Automatic configuration support for Fedora 13. You may need <a href="http://packages.qa.debian.org/o/os-prober/news/20100628T171748Z.html">os-prober 1.39</a> from unstable as well.</li>
- <li>Automatic configuration support for Linux on Xen.</li>
- <li>Skip LVM snapshots rather than failing when they're present.</li>
-</ul>
+++ /dev/null
-debhelper statistics, redux
-
-<p>Apropos of <a href="http://www.chiark.greenend.org.uk/ucgi/~cjwatson/blosxom/debian/2010-03-03-debhelper-statistics.html">my previous post</a>, I see that dh has now overtaken CDBS as the most popular rules helper system of its kind in Debian unstable, and shows no particular sign of slowing its rate of uptake any time soon. The resolution of the graph is such that you can't see it yet, but dh drew dead level with CDBS on Thursday, and today 3836 packages are using dh as opposed to 3823 using CDBS.</p>
-
-<p><img src="http://people.debian.org/~cjwatson/dhstats.png" /></p>
+++ /dev/null
-Windows applications making GRUB 2 unbootable
-
-<p>If you find that running Windows makes a GRUB 2-based system unbootable (<a href="http://bugs.debian.org/550702">Debian bug</a>, <a href="https://bugs.launchpad.net/bugs/441941">Ubuntu bug</a>), then I'd like to hear from you. This is a bug in which some proprietary Windows-based software overwrites particular sectors in the gap between the master boot record and the first partition, sometimes called the "embedding area". GRUB Legacy and GRUB 2 both normally use this part of the disk to store one of their key components: GRUB Legacy calls this component Stage 1.5, while GRUB 2 calls it the core image (<a href="http://www.gnu.org/software/grub/manual/grub.html#Images">comparison</a>). However, Stage 1.5 is less useful than the core image (for example, the latter provides a rescue shell which can be used to recover from some problems), and is therefore rather smaller: somewhere around 10KB vs. 24KB for the common case of ext[234] on plain block devices. It seems that the Windows-based software writes to a sector which is after the end of Stage 1.5, but before the end of the core image. This is why the problem appears to be new with GRUB 2.</p>
-
-<p>At least some occurrences of this are with software which writes a signature to the embedding area which hangs around even after uninstallation (even with one of those tools that tracks everything the installation process did and reverses it, I gather), so that you cannot uninstall and reinstall the application to defeat a trial period. This seems like a fine example of an <a href="http://wiki.mako.cc/Antifeatures">antifeature</a>, especially given its destructive consequences for free software, and is in general a poor piece of engineering; what happens if multiple such programs want to use the same sector, I wonder? They clearly aren't doing much checking that the sector is unused, not that that's really possible anyway. While I do not normally think that GRUB should go to any great lengths to accommodate proprietary software, this is a case where we need to defend ourselves against the predatory practices of some companies making us look bad: a relatively small number of people do enough detective work to realise that it's the fault of a particular Windows application, but many more simply blame our operating system because it won't start any more.</p>
-
-<p>I believe that it may be possible to assemble a collection of signatures of such software, and arrange to avoid the disk sectors they have stolen. Indeed, I have a first draft of the necessary code. This is not a particularly pleasant solution, but it seems to be the most practical way around the problem; I'm hoping that several of the programs at fault are using common "licence manager" code or something like that, so that we can address most of the problems with a relatively small number of signatures. In order to do this, I need to hear from as many people as possible who are affected by this problem.</p>
-
-<p>If you suffer from this problem, then please do the following:</p>
-
-<ul>
- <li>Save the output of <code>fdisk -lu</code> to a file. In this output, take note of the start sector of the first partition (usually 63, but might also be 2048 on recent installations, or occasionally something else). If this is something other than 63, then replace 63 in the following items with your number.</li>
- <li>Save the contents of the embedding area to a file (replace <code>/dev/sda</code> with your disk device if it's something else): <code>dd if=/dev/sda of=sda.1 count=63</code></li>
- <li>Do whatever you do to make GRUB unbootable (presumably starting Windows), then boot into a recovery environment. Before you reinstall GRUB, save the new contents of the embedding area to a different file: <code>dd if=/dev/sda of=sda.2 count=63</code></li>
- <li>Follow up to either the Debian or the Ubuntu bug with these three files (the output of <code>fdisk -lu</code>, and the embedding area before and after making GRUB unbootable.</li>
-</ul>
-
-<p>I hope that this will help me to assemble enough information to fix this bug at least for most people, and of course if you provide this information then I can make sure to fix your particular version of this problem. Thanks in advance!</p>
--- /dev/null
+#!/usr/bin/env bash
+##
+# This section should match your Makefile
+##
+PY=${PY:-python}
+PELICAN=${PELICAN:-pelican}
+PELICANOPTS=
+
+BASEDIR=$(pwd)
+INPUTDIR=$BASEDIR/content
+OUTPUTDIR=$BASEDIR/output
+CONFFILE=$BASEDIR/pelicanconf.py
+
+###
+# Don't change stuff below here unless you are sure
+###
+
+SRV_PID=$BASEDIR/srv.pid
+PELICAN_PID=$BASEDIR/pelican.pid
+
+function usage(){
+ echo "usage: $0 (stop) (start) (restart) [port]"
+ echo "This starts Pelican in debug and reload mode and then launches"
+ echo "an HTTP server to help site development. It doesn't read"
+ echo "your Pelican settings, so if you edit any paths in your Makefile"
+ echo "you will need to edit your settings as well."
+ exit 3
+}
+
+function alive() {
+ kill -0 $1 >/dev/null 2>&1
+}
+
+function shut_down(){
+ PID=$(cat $SRV_PID)
+ if [[ $? -eq 0 ]]; then
+ if alive $PID; then
+ echo "Stopping HTTP server"
+ kill $PID
+ else
+ echo "Stale PID, deleting"
+ fi
+ rm $SRV_PID
+ else
+ echo "HTTP server PIDFile not found"
+ fi
+
+ PID=$(cat $PELICAN_PID)
+ if [[ $? -eq 0 ]]; then
+ if alive $PID; then
+ echo "Killing Pelican"
+ kill $PID
+ else
+ echo "Stale PID, deleting"
+ fi
+ rm $PELICAN_PID
+ else
+ echo "Pelican PIDFile not found"
+ fi
+}
+
+function start_up(){
+ local port=$1
+ echo "Starting up Pelican and HTTP server"
+ shift
+ $PELICAN --debug --autoreload -r $INPUTDIR -o $OUTPUTDIR -s $CONFFILE $PELICANOPTS &
+ pelican_pid=$!
+ echo $pelican_pid > $PELICAN_PID
+ cd $OUTPUTDIR
+ $PY -m pelican.server $port &
+ srv_pid=$!
+ echo $srv_pid > $SRV_PID
+ cd $BASEDIR
+ sleep 1
+ if ! alive $pelican_pid ; then
+ echo "Pelican didn't start. Is the Pelican package installed?"
+ return 1
+ elif ! alive $srv_pid ; then
+ echo "The HTTP server didn't start. Is there another service using port" $port "?"
+ return 1
+ fi
+ echo 'Pelican and HTTP server processes now running in background.'
+}
+
+###
+# MAIN
+###
+[[ ($# -eq 0) || ($# -gt 2) ]] && usage
+port=''
+[[ $# -eq 2 ]] && port=$2
+
+if [[ $1 == "stop" ]]; then
+ shut_down
+elif [[ $1 == "restart" ]]; then
+ shut_down
+ start_up $port
+elif [[ $1 == "start" ]]; then
+ if ! start_up $port; then
+ shut_down
+ fi
+else
+ usage
+fi
+++ /dev/null
-text/html;
+++ /dev/null
-<span class="dateheader">$dw, $da $mo $yr</span>
+++ /dev/null
-<br />
-
- </td>
-
- <td bgcolor="#dddddd"><nobr> </nobr>
-
- </tr>
-
- <tr>
-
- <td colspan="6" bgcolor="#000000" height="1"></td>
-
- </tr>
-
-</table>
-
-</body>
-
-</html>
-
+++ /dev/null
-<br />
-
- </td>
- </tr>
-
- <tr>
- <td colspan="9" bgcolor="#000000" height="1"></td>
- </tr>
-
- <tr>
- <td colspan="3" align="right">
- <a href="http://www.blosxom.com/">Powered by Blosxom</a>
- </td>
- </tr>
-
-</table>
-
-</body>
-
-</html>
-
+++ /dev/null
-<html>
-
-<head>
-<title>$blog_title</title>
-
-<link rel="alternate" type="application/rss+xml" title="RSS" href="$url/index.rss" />
-
-<style>
-body,td { font-family: Verdana,Geneva,Arial,Sans-serif;
- font-size: medium; color: #000000; }
-body { background-color: #ffffff; }
-a { color: #000000 }
-.pagetitle { font-size: large; font-weight: bold; color: #336699; }
-.dateheader { font-size: x-large; font-weight: bold; color: #336699; }
-h1 { font-size: large; font-weight: bold; color: #336699; }
-h2 { font-size: large; font-weight: bold; }
-.meta { font-size: small; }
-</style>
-
-</head>
-
-<body>
-
-<table border="0" width="100%" cellpadding="0" cellspacing="0">
- <tr>
- <!-- main title -->
- <td colspan="6"><span class="pagetitle">$blog_title</span></td>
- </tr>
-
- <tr>
- <td colspan="6" bgcolor="#000000" height="1" ></td>
- </tr>
-
- <tr valign="top">
- <td bgcolor="#6699cc"><nobr> </nobr>
-
- <!-- left column -->
- <td bgcolor="#6699cc" class="meta">
-
- <p>
- <br />
- <b>About</b> <br />
- $blog_description<br />
- <a href="mailto:cjwatson@debian.org">cjwatson@debian.org</a>
- </p>
-
- <p>
- <b>Subscribe</b> <br />
- <a href="$url/index.rss">Subscribe</a> to a syndicated feed of my blog.
- </p>
-
- <p>
- <b>Flavours</b> <br />
- <ul>
- <li><a href="$url/index.index">index</a></li>
- <li><a href="$url/index.rss">RSS</a></li>
- </ul>
- </p>
-
- <p>
- <br />
- <a href="http://www.blosxom.com/">Powered by Blosxom</a>
- </p>
-
- </td>
-
- <td bgcolor="#6699cc"><nobr> </nobr></td>
-
- <td bgcolor="#dddddd"><nobr> </nobr></td>
-
- <!-- main blog entry column -->
- <td bgcolor="#dddddd" width="100%">
-
- <br />
-
+++ /dev/null
-<html>
-
-<head>
-<title>$blog_title</title>
-<!--<link rel=stylesheet type="text/css" href="/blogbook.css">-->
-
-<style>
-body,td { font-family: Verdana,Geneva,Arial,Sans-serif;
- font-size: 11px; color: #666666; }
-a { text-decoration: none; }
-.title { font-size: 12pt; font-weight: bold; color: #336699; }
-.blosxomDate {font-weight: bold; color: #336699; }
-.blosxomTitle { font-weight: bold; color: #000000; }
-.blosxomTime { text-decoration: italicize; color: #336699; }
-</style>
-
-</head>
-
-<body bgcolor="#ffffff">
-
-<table border="0" width="100%" cellpadding="0" cellspacing="0">
- <tr>
- <td colspan="3">
- <span class="title">$blog_title</span>
- </td>
- </tr>
-
- <tr>
- <td colspan="9" bgcolor="#000000" height="1" ></td>
- </tr>
-
- <tr valign="top">
- <td bgcolor="#cccccc" width="100%">
-
- <br />
-
+++ /dev/null
-<p>
-<a name="$fn"><h1>$title</h1></a>
-</p>
-$body
-
-<p align="right">
-<i>[<a href="$url$path">$path</a>]
-<a href="$url$path/$fn.$flavour">permanent link</a></i>
-</p>
-
+++ /dev/null
-<a href="$url/$yr/$mo_num/$da#$fn">$title</a><br />
--- /dev/null
+#!/usr/bin/env python
+# -*- coding: utf-8 -*- #
+from __future__ import unicode_literals
+
+AUTHOR = u'Colin Watson'
+SITENAME = u"Colin Watson's blog"
+SITEURL = ''
+
+PATH = 'content'
+SLUGIFY_SOURCE = 'basename'
+
+TIMEZONE = 'Europe/London'
+DEFAULT_DATE_FORMAT = '%Y-%m-%d'
+
+DEFAULT_LANG = u'en'
+
+FEED_ALL_ATOM = 'feeds/all.atom.xml'
+CATEGORY_FEED_ATOM = 'feeds/%s.atom.xml'
+TAG_FEED_ATOM = 'feeds/tag/%s.atom.xml'
+TRANSLATION_FEED_ATOM = None
+AUTHOR_FEED_ATOM = None
+AUTHOR_FEED_RSS = None
+
+SOCIAL = (
+ ('Twitter', 'https://twitter.com/colmmacuait'),
+ ('Google+', 'https://plus.google.com/u/0/115774225464003520154/posts'),
+ )
+TWITTER_USERNAME = 'colmmacuait'
+
+DEFAULT_PAGINATION = 10
+
+# Uncomment following line if you want document-relative URLs when developing
+#RELATIVE_URLS = True
+
+TYPOGRIFY = True
+THEME = 'cjwatson-theme'
--- /dev/null
+#!/usr/bin/env python
+# -*- coding: utf-8 -*- #
+from __future__ import unicode_literals
+
+# This file is only used if you use `make publish` or
+# explicitly specify it as your config file.
+
+import os
+import sys
+sys.path.append(os.curdir)
+from pelicanconf import *
+
+SITEURL = 'http://www.chiark.greenend.org.uk/~cjwatson/blog'
+RELATIVE_URLS = False
+
+FEED_ALL_ATOM = 'feeds/all.atom.xml'
+CATEGORY_FEED_ATOM = 'feeds/%s.atom.xml'
+
+DELETE_OUTPUT_DIRECTORY = True
+
+# Following items are often useful when publishing
+
+#DISQUS_SITENAME = ""
+++ /dev/null
-Google Summer of Code project started (Debian)
-
-<p>I'm mentoring <a href="http://xsunblog.blogspot.com/">Matheus Morais</a>
-in the <a href="http://code.google.com/soc/">Google Summer of Code</a>,
-porting d-i to the Hurd. We've exchanged a few mails and he has in hand all
-the preliminary (but not yet functional; wouldn't want to make it too easy
-:-)) patches I've put together in the past. I think I should be reasonably
-well-placed to judge his progress.</p>
-
-<p>Best of luck, Matheus!</p>
+++ /dev/null
-Single-stage installer
-
-<p>Hot on the heels of
-<a href="http://kitenet.net/~joey/blog/entry/all_this_for_a_progress_bar-2005-12-27-20-32.html">
-Joey's tale of getting rid of base-config</a> (the second stage of the
-installer) in Debian, we've now pretty much got rid of it in Ubuntu Dapper
-too. The upshot of this is that rather than asking a bunch of questions,
-installing the base system, and rebooting to install everything else, we now
-just install everything in one go and reboot into a completed system.</p>
-
-<p>This does mean that, if your system doesn't boot, you don't get to find
-out about it for a bit longer. However, it has lots of advantages in terms
-of speed (the much-maligned archive-copier mostly goes away), reducing code
-duplication (base-config had a bunch of infrastructure of its own which was
-done better in the core installer anyway), comprehensibility, and killing
-off some annoying bugs like
-<a href="https://bugzilla.ubuntu.com/show_bug.cgi?id=13561">#13561
-(duplicate mirror questions in netboot installs)</a>,
-<a href="https://bugzilla.ubuntu.com/show_bug.cgi?id=15213">#15213 (second
-stage hangs if you skip archive-copier in the first stage)</a>, and
-<a href="https://bugzilla.ubuntu.com/show_bug.cgi?id=19571">#19571 (kernel
-messages scribble over base-config's UI)</a>.</p>
-
-<p>To go with Joey's Debian timeline, the Ubuntu history looks a bit like
-this:</p>
-
-<ul>
- <li>2004 (Jul): First base-config modifications for Ubuntu; we need to
- install the default desktop rather than dropping into tasksel.</li>
-
- <li>2004 (Aug): Mark phones me up and asks if I can make the installer not
- need the CD in the second stage by copying all the packages across
- beforehand. Although it's a bit awkward, I can see the UI advantages in
- that, so I write archive-copier at the Canonical conference in
- Oxford.</li>
-
- <li>2004 (Sep): Mark asks me if we can ask the timezone, user/password,
- and apt configuration questions before the first reboot. With less than a
- month to go until our first release, I have a
- <a href="http://lists.ubuntu.com/archives/ubuntu-devel/2004-September/000103.html">
- heart-attack</a> at how much needs to be done, and it eventually gets
- deferred to <a href="https://wiki.ubuntu.com/HoaryGoals">Hoary</a>.</li>
-
- <li>2005 (Jan): Matt fixes up debconf's passthrough frontend for use on
- the live CD, and we realise that this is an obvious way to run bits of
- base-config before the first reboot. It's rather messy and takes until
- March or so before it really works right, but we get there in the
- end.</li>
-
- <li>2005 (Apr): I get "put a progress bar in front of the dpkg output in
- the second stage" as a
- <a href="https://wiki.ubuntu.com/UbuntuDownUnder/BOFs/InstallerStage2Progress">goal</a>
- for Breezy. Naïvely, I think it's a simple matter of programming,
- since I'd already done something similar for debootstrap and
- base-installer the previous year.</li>
-
- <li>2005 (May): I hack progress bar support into debconf. Nothing actually
- uses it for anything yet, except as a convenient passthrough stub.</li>
-
- <li>2005 (Jul/Aug): I actually try to implement the second-stage progress
- bar and realise that it's about an order of magnitude harder than I
- thought, requiring a whole load of extra infrastructure in apt.
- Fortunately Michael Vogt saves the day here by writing lots of working
- code, and the progress bar works by early August.</li>
-
- <li>2005 (Sep-Dec): Upstream d-i development ramps back up again, with
- tzsetup, clock-setup, apt-setup, and user-setup all being cranked out in
- short order and the corresponding pieces removed from base-config. I merge
- these as they mature, and manage to get
- <a href="http://lists.debian.org/debian-boot/2005/10/msg01407.html">
- agreement</a> on including the Ubuntu debconf template changes in upstream
- apt-setup, which helps the diff size a lot.</li>
-
- <li>2005 (Nov/Dec): Joey and I
- <a href="http://lists.debian.org/debian-boot/2005/11/msg01381.html">chat</a>
- one evening about the Ubuntu second-stage progress bar work, and we end up
- designing and writing debconf-apt-progress based on its ideas, after which
- Joey knocks up pkgsel in no time flat.</li>
-
- <li>2006 (Jan): The rest of the pieces land in Ubuntu, and we drop
- base-config out of the installer. To my surprise, nearly everything still
- just works.</li>
-</ul>
-
-<p>Although it caused some friction, I'm glad that we did the first cuts of
-many of these things outside Debian and got to try things out before landing
-version-2-quality code in Debian. The end result is much nicer than the
-intermediate ones ever were.</p>
+++ /dev/null
-Vim omni completion for Launchpad bugs
-
-<p>I hacked together a little timesaver for developers this morning: omni
-completion for Launchpad bugs in Vim's debchangelog mode. To use it, install
-vim 7.1-138+1ubuntu3 once it hits the mirrors, open up a
-<tt>debian/changelog</tt> file, type "LP: #", and hit Ctrl-X Ctrl-O. It'll
-think for a while and then give you a list of all the bugs open in Launchpad
-against the package in question, from which you can select to insert the bug
-number into your changelog.</p>
-
-<p>Here's a screenshot to make it clearer:</p>
-
-<p><img src="http://people.ubuntu.com/~cjwatson/lp-omnicomplete.png"></p>
-
-<p>Thanks to Stefano Zacchiroli for doing the same for Debian bugs back in
-July.</p>
+++ /dev/null
-Desktop automounting pain
-
-<p>Ubuntu's live CD installer, Ubiquity, needs to suppress desktop
-automounting while it's doing partitioning and generally messing about with
-mount points, otherwise its temporary mount points end up busy on unmount
-due to some smart-arse desktop component that decides to open a window for
-it.</p>
-
-<p>To date, it employs the following methods, each of which was sufficient
-at the time:</p>
-
-<ul>
- <li>
- Set the <code>/desktop/gnome/volume_manager/automount_drives</code> and
- <code>/desktop/gnome/volume_manager/automount_media</code> gconf keys to
- <code>false</code>.
- </li>
- <li>
- Tell <code>kded</code> to unload its <code>medianotifier</code> module,
- and load it again just before the installer exits.
- </li>
- <li>
- Set the <code>/apps/nautilus/desktop/volumes_visible</code> gconf key to
- <code>false</code>.
- </li>
- <li>
- Set the <code>AutomountDrives</code> and <code>AutomountMedia</code> keys
- in <code>$HOME/.config/Thunar/volmanrc</code> to <code>FALSE</code>.
- </li>
- <li>
- Set the <code>/apps/nautilus/preferences/media_automount</code> and
- <code>/apps/nautilus/preferences/media_automount_open</code> gconf keys to
- <code>false</code>.
- </li>
- <li>
- The entire installer is run under <code>hal-lock --interface
- org.freedesktop.Hal.Device.Storage --exclusive</code>.
- </li>
- <li>
- Set the <code>/apps/nautilus/preferences/media_autorun_never</code> gconf
- key to <code>true</code>
- (<a href="https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/210620">experimental,
- but apparently now required since nautilus uses the gio volume
- monitor</a>).
- </li>
-</ul>
-
-<p>This is getting <strong>ridiculous</strong>. Dear desktop implementors:
-please pick a configuration mechanism and stick to it, and provide backward
-compatibility if you can't. This is not a rocket-science concept.</p>
-
-<p>I rather liked the <code>hal-lock</code> mechanism; it was simple and
-involved minimal fuss. I had hoped that it might end up as a standard, but I
-guess that would be too easy.</p>
+++ /dev/null
-Totem BBC plugin
-
-<p>A while back, the <a href="http://www.bbc.co.uk/">BBC</a> approached
-<a href="http://www.canonical.com/">Canonical</a> about providing seamless
-access to unencumbered BBC content for all
-<a href="http://www.ubuntu.com/">Ubuntu</a> users (in the UK and elsewhere). We
-agreed to approach this by way of a plugin for our primary media player,
-<a href="http://www.gnome.org/projects/totem/">Totem</a>, and asked
-<a href="http://www.collabora.co.uk/">Collabora Multimedia</a> to do the plugin
-development work.</p>
-
-<p>The results are in what will shortly be released as Ubuntu 8.10, and are
-looking really rather good. At the moment the content available from the BBC at
-present is mostly audio, but support for video is in place and the feed is
-expected to be fleshed out here over time. We have a genre classification
-scheme in place, and will see how that scales as the amount of available
-content grows. The code has been
-<a href="http://bugzilla.gnome.org/show_bug.cgi?id=555823">submitted
-upstream</a>, although there are still a few issues to work out there.</p>
-
-<p>This is not the same thing as iPlayer; all the content available here is
-DRM-free. Some of it is geographically restricted to the UK, and these
-restrictions are handled on the server side to make sure that the client is
-free of encumbrances.</p>
-
-<p><a href="http://blogs.gnome.org/uraeus/">Christian Schaller</a> from
-Collabora <a href="http://blogs.gnome.org/uraeus/2008/10/08/868/">posted</a>
-about this a little while ago. Since then, the UI has been improved somewhat
-and some I/O issues have been fixed to the point where we felt comfortable
-enabling the BBC plugin (as well as the YouTube plugin) by default in Ubuntu
-8.10. Here's a
-<a href="http://people.ubuntu.com/~cjwatson/totem-bbc-2.png">screenshot</a> of
-the current interface.</p>
-
-<p>This is exciting stuff with a lot of potential. To try it out, run
-Applications -> Sound & Video -> Movie Player and select the "BBC"
-entry from the drop-down box labelled "Playlist". If you find bugs, please
-<a href="https://bugs.launchpad.net/ubuntu/+source/totem/+filebug">report</a>
-them!</p>
+++ /dev/null
-Bug triage rants
-
-<p>I hate to say this, but often when somebody does lots of bug triage on a package I work on, I find it to be a net loss for me. I end up having to go through all the things that were changed, correct a bunch of them, occasionally pacify angry bug submitters, and all the rest of it, and often the benefits are minimal at best.</p>
-
-<p>I would very much like this not to be the case. Bug triage is supposed to help developers be more efficient, and I think most people who do bug triage are generally well-intentioned and eager to help. Accordingly, here is a series of mini-rants intended to have educational value.</p>
-
-<ul>
- <li>
- <p><strong>Bugs are not like fruit.</strong></p>
-
- <p>Fruit goes bad if you leave it too long. By and large, bugs don't, especially if they're on software that doesn't change very much. There is no reason why a bug filed against a package in Ubuntu 4.10 where the relevant code hasn't changed much since shouldn't still be perfectly valid. Even if it isn't, it deserves proper consideration.</p>
-
- <p>My biggest single annoyance with bug triage is people coming around and asking if bugs are still valid when they haven't put any effort into reproducing them themselves. This annoys bug submitters too; every so often somebody replies and says "didn't you even bother to check?". This gives a very bad impression of us as a project - wouldn't it be better if we looked as if we knew what we were talking about? There is a good reason to do this kind of check, of course: random undiagnosed crash reports and the like may well go away due to related changes, and it is occasionally worth checking. But if the bug is already well-understood and/or well-described, you should just go and check whether it's still there rather than asking.</p>
-
- <p>As I understand it, the intended workflow is that people file bugs, then if they aren't clear enough bug triagers work with the submitter to gather information until they are, then they're passed to developers for further work. We seem to have added an extra step wherein submitters must periodically give their bug a health-check, and if they don't then it gets closed as being out of date. In a small minority of cases this is useful; in most cases, frankly, it makes us look a bit clueless. Can we please stop doing this? The more we waste people's time doing this, the less likely it is that they'll bother to respond to us, and this might help our statistics but doesn't help the project as a whole.</p>
-
- <p>I know that there's a problem with bug count. I think every project of non-trivial size has that problem. But, honestly, the right answer is to <em>fix more bugs</em> - and, personally, I would be able to spend more time doing that if I weren't often running around trying to make sure that bugs I care about aren't getting overenthusiastically closed just because somebody thinks they've been lying around too long.</p>
-
- <p>There is a good way to expire bugs like this, of course. It goes something like this: "I've read through your bug and tried to reproduce it with a current release, but I'm afraid I can't do so. Are you still experiencing it? If not, then I think it might have been fixed by [this change I found in the package's history that seems to be related]." You can't do this <em>en masse</em>, but you'll get a much better response from submitters, you'll learn more doing it, and in the process of doing the necessary investigation of each bug you'll find that there are many cases you don't have to ask about at all.</p>
- </li>
-
- <li>
- <p><strong>Wishlist bugs are not intrinsically bad.</strong></p>
-
- <p>There are certainly cases where something is far too broad or vague for a bug report; but there are also plenty of cases, probably far more, where the wish in question is a relatively small change to the program, or doesn't need any more sophisticated tracking, and a wishlist bug is just right. If you don't know the program very well, it may be difficult to tell whether a wishlist bug is appropriate or not; in that case, just leave the bug alone.</p>
-
- <p>Please, for the love of all that's holy, don't close wishlist bugs saying that people should use Brainstorm or write a specification instead! If you don't want to see wishlist bugs in your statistics, just filter them out; it's quite easy to do. Even worse, don't tell people that something probably isn't a good idea when you aren't familiar with the software; people who have gone to the effort of writing up their idea for us deserve a response from somebody who knows the software well. I've encountered cases where friends of mine submitted a bug report (sometimes even at my request) and then a triager told them it was a bad idea and closed their bug. This sort of thing puts people off Ubuntu.</p>
-
- <p>Specifications are software design documents. As such, they are best written by software designers. People who tell other people to go and write a specification may not realise that as a result of doing this for three years it's now essentially impossible to find anything in the specification system! The intent was never that every user of Ubuntu would need to write a specification to get anything changed; specifications are used by developers to document the results of discussions and write up plans. They are not a straightforward alternative to wishlist bugs, nor do they turn out to work very well as what many formal processes call "requirements documents"; the process of refining the latter in the context of Ubuntu might involve wishlist bugs, mailing list threads, wiki pages, private discussions with developers, or things of that nature, and probably shouldn't involve creating a specification until the requirements-gathering process is well underway.</p>
- </li>
-
- <li>
- <p><strong>Closing a bug is taking an item off somebody's to-do list.</strong></p>
-
- <p>You wouldn't go up to a colleague's whiteboard and take an eraser to it unless you were sure that was OK, would you? Yet people seem to do that all the time with bugs. It's OK when the bug is really just like a support request - "help, it crashed, what do I do?" - and either you're pretty sure it's user error or there's just no way to get enough information to fix it. But once the initial triage process is done, now it's on somebody's to-do list.</p>
-
- <p>This is closely related to ...</p>
- </li>
-
- <li>
- <p><strong>If a developer has accepted it, leave it alone.</strong></p>
-
- <p>Every so often I find that there's a bug that I have accepted by way of a bug comment or setting to Triaged or whatever, or even a bug that I filed on a package I work on as a reminder to myself, and somebody comes along and asks for more information or asks if we can still reproduce it or something. The hit rate on this kind of thing is extraordinarily low. There's a good chance that the developer went and verified the bug against the code, and in that case it certainly doesn't need more information (or they would have asked for it) and it probably isn't going to go away without anyone noticing.</p>
-
- <p>In most other free software projects, developers file bug reports themselves as a reminder about things that need to be done, and people leave them alone unless they're intending to help with the fix. In Ubuntu, developers also have to spend time making sure that those to-do items don't get expired. Nobody is helped by this.</p>
-
- <p><a href="https://launchpad.net/launchpad-gm-scripts">launchpad-gm-scripts</a> includes a Greasemonkey script called <code>lp_karma_suffix</code>, which can help you to identify developers without having to spend lots of time clicking around.</p>
- </li>
-
- <li>
- <p><strong>Check whether the package is being actively worked on.</strong></p>
-
- <p>Some packages are actively worked on in Ubuntu; some aren't (e.g. we just sync packages from Debian, or they're basically orphaned, or whatever). It's worth checking which is which before doing any kind of extensive triage work. If it's being actively worked on, why not go and talk to the developer(s) in question first? It's only polite, and it will probably help you to do a better job.</p>
- </li>
-</ul>
+++ /dev/null
-Bug triage, redux
-
-<p>I've been a bit surprised by the strong positive response to my <a href="http://www.chiark.greenend.org.uk/ucgi/~cjwatson/blosxom/ubuntu/2009-02-27-bug-triage-rants.html">previous post</a>. People generally seemed to think it was quite non-ranty; maybe I should clean the rust off my flamethrower. :-) My hope was that I'd be able to persuade people to change some practices, so I guess that's a good thing.</p>
-
-<p>Of course, there are many very smart people doing bug triage very well, and I don't want to impugn their fine work. Like its medical namesake, bug triage is a skilled discipline. While it's often repetitive, and there are lots of people showing up with similar symptoms, a triage nurse can really make a difference by spotting urgent cases, cleaning up some of the initial blood, and referring the patient quickly to a doctor for attention. Or, if a pattern of cases suddenly appears, a triage nurse might be able to warn of an incipient epidemic. [Note: I have no medical experience, so please excuse me if I'm talking crap here. :-)] The bug triagers who do this well are an absolute godsend; especially when they respond to repetitive tasks with tremendously useful pieces of automation like <a href="https://launchpad.net/bughelper">bughelper</a>. The cases I have trouble with are more like somebody showing up untrained, going through everyone in the waiting room, and telling each of them that they just need to go home, get some rest, and stop complaining so much. Sometimes of course they'll be right, but without taking the time to understand the problem they're probably going to do more harm than good.</p>
-
-<p>Ian Jackson reminded me that it's worth mentioning the purpose of bug reports on free software: namely, <strong>to improve the software</strong>. The GNU Project has some <a href="http://www.gnu.org/prep/maintain/maintain.html#Mail">advice to maintainers</a> on this. I think sometimes we stray into regarding bug reports more like support tickets. In that case it would be appropriate to focus on resolving each case as quickly as possible, if necessary by means of a workaround rather than by a software change, and only bother the developers when necessary. This is the wrong way to look at bug reports, though. The reason that we needed to set up a bug triage community in Ubuntu was that we had a relatively low developer-to-package ratio and a very high user-to-developer ratio, and we were getting a lot of bug reports that weren't fleshed out enough for a developer to investigate them without spending a lot of time in back-and-forth with the reporter, so a number of people volunteered to take care of the initial back-and-forth so that good clear bug reports could be handed over to developers. This is all well and good, and indeed I encouraged it because I was personally finding myself unable to keep up with incoming bugs and actually fix anything at the same time. Somewhere along the way, though, some people got the impression that what we wanted was a first-line support firewall to try to defend developers from users, which of course naturally leads to ideas such as closing wishlist bugs containing ideas because obviously those important developers wouldn't want to be bothered by them, and closing old bugs because clearly they must just be getting in developers' way. Let me be clear about this now: I absolutely appreciate help getting bug reports into a state where I can deal with them efficiently, but <strong>I do not want to be defended from my users</strong>! I don't have a basis from which to state that all developers feel the same way, but my guess is that most do.</p>
-
-<p><a href="http://antti-juhani.kaijanaho.fi/newblog/archives/471">Antti-Juhani Kaijanaho</a> said he'd experienced most of these problems in Debian. I hadn't actually intended my post to go to Planet Debian - I'd forgotten that the "ubuntu" category on my blog goes there too, which generally I see as a feature, but if I'd remembered that I would have been a little clearer that I was talking about Ubuntu bug triage. If I had been talking about Debian bug triage I'd probably have emphasised different things. Nevertheless, it's interesting that at least one Debian (and non-Ubuntu) developer had experienced similar problems.</p>
-
-<p><a href="http://jldugger.livejournal.com/25994.html">Justin Dugger</a> mentions a practice of marking duplicate bugs invalid that he has problems with. I agree that this is suboptimal and try not to do it myself. That said, this is not something I object to to the same extent. Given that the purpose of bugs is to improve the software, the real goal is to be able to spend more time fixing bugs, not to get bugs into the ideal state when the underlying problem has already been solved. If it's a choice between somebody having to spend time tracking down the exact duplicate bug number versus fixing another bug, I know which I'd take. Obviously, when doing this, it's worth apologising that you weren't able to find the original bug number, and explaining what the user can do if they believe that you're mistaken (particularly if it's a bug that's believed to be fixed); the stock text people often use for this doesn't seem informative enough to me.</p>
-
-<p>Sebastien Bacher commented that preferred bug triage practices differ among teams: for instance, the Ubuntu desktop team deals with packages that are very much to the forefront of users' attention and so get a lot of duplicate bugs. Indeed - and bug triagers who are working closely with the desktop team on this are almost certainly doing things the way the developers on the desktop team prefer, so I have no problem with that. The best advice I can give bug triagers is that their ultimate aim is to help developers, and so they should figure out which developers they need to work with and <strong>go and talk to them</strong>! That way, rather than duplicating work or being counterproductive, they can tailor their work to be most effective. Everybody wins.</p>
+++ /dev/null
-code_swarm video of Ubuntu uploads
-
-<p>Joey Hess posted a
-<a href="http://lists.debian.org/debian-boot/2009/05/msg00265.html">draft</a>
-of a <a href="http://code.google.com/p/codeswarm/">code_swarm</a> video for
-d-i a couple of weeks ago, which reminded me that I've been meaning to do
-something similar for Ubuntu for a while now as it's just about our
-archive's fifth birthday. I have a more or less complete archive of all our
--changes mailing lists locally (I think I'm missing some of the very early
-ones, before the end of July 2004; let me know if you were one of the very
-early Canonical employees and have a record of these), and with the aid of
-<a href="https://help.launchpad.net/API/launchpadlib">launchpadlib</a> it's
-fairly easy to map all the e-mail addresses into Launchpad user names,
-massage out some of the more obvious duplicates, and then treat the stream
-of uploads as if it were a stream of commits.</p>
-
-<p>If you haven't seen code_swarm before, each dot represents an upload, and
-the dots "swarm" around their corresponding committers' names; more active
-committers have larger swarms of dots and brighter names. I assigned a
-colour to each of our archive components (uploads aren't really at the C
-code vs. Python code vs. translations vs. whatever kind of granularity that
-you see in other code_swarm videos), which mostly means that people who
-predominantly upload to main are in roughly an Ubuntu tan colour, people who
-predominantly upload to universe are coloured bluish, and people with a good
-mixture tend to come out coloured green. If I get a bit more time I may try
-to figure out enough about video editing software to add some captions.</p>
-
-<p>Here's the
-<a href="http://people.ubuntu.com/~cjwatson/ubuntu-uploads.ogv">video</a>
-(194 MB).</p>
+++ /dev/null
-Tissue of lies
-
-<p>In case it isn't obvious, in
-<a href="http://ubuman.wordpress.com/2009/11/13/ubuntu-9-10-sp1-coming-in-spring-2010/">
-"Ubuntu 9.10 SP1 coming in spring 2010"</a>, "Ubuman" is blatantly lying in
-attributing a number of statements to me. None of the text there was written
-by me, and if you thought any of it was true then you should probably make
-sure your troll radar is working properly. Nice joke, but try harder next
-time - it doesn't even look like my writing style.</p>
-
-<p>(I wouldn't normally bother to respond, since I'm probably just giving it
-more publicity, but apparently one or two people may already have been taken
-in by it. One person was sensible enough to write to me and check the
-facts.)</p>
+++ /dev/null
-OpenSSH 5.5p1 for Lucid
-
-<p>For various reasons, I chose to leave Ubuntu 10.04 LTS using OpenSSH 5.3p1. The <a href="http://www.openssh.org/txt/release-5.4">new features in 5.4p1</a> such as certificate authentication, the new smartcard handling, netcat mode, and tab-completion in sftp are great, but unfortunately it was available just a little bit too late for me to be able to land it for 10.04 LTS. I realise that many Lucid users want to make use of these features for one reason or another, though, so as a compromise here's a PPA containing <a href="https://launchpad.net/~cjwatson/+archive/openssh">OpenSSH 5.5p1 for Lucid</a>.</p>
-
-<p>I intend to keep this up to date for as long as I reasonably can, and I'm happy to accept bug reports on it in the <a href="https://bugs.launchpad.net/ubuntu/+source/openssh">usual place</a>.</p>
+++ /dev/null
-NTP synchronisation problems
-
-<p>The Ubuntu Technical Board is currently conducting a review of the top ten Brainstorm issues users have raised about Ubuntu, and Matt asked me to investigate and respond to <a href="http://brainstorm.ubuntu.com/idea/25301/">Idea #25301: Keeping the time accurate over the Internet by default</a>.</p>
-
-<p>My first reaction was "hey, that's odd - I thought we already did that?". We install the <code>ntpdate</code> package by default (although it's <a href="http://www.eecis.udel.edu/~mills/ntp/html/ntpdate.html">deprecated upstream</a> in favour of other tools, but that shouldn't be important here). <code>ntpdate</code> is run from <code>/etc/network/if-up.d/ntpdate</code>, in other words every time you connect to a network, which should be acceptably frequent for most people, so it really ought to Just Work by default. But this is one of the top ten problems where users have gone to the trouble of proposing solutions on Brainstorm, so it couldn't be that simple. What was going on?</p>
-
-<p>I brought up a clean virtual machine with a development version of Natty (the current Ubuntu development version, which will eventually become 11.04), and had a look in its logs: it was indeed synchronising its time from <code>ntp.ubuntu.com</code>, and I didn't think anything in that area had changed recently. On the other hand, I had occasionally noticed that my own laptop wasn't always synchronising its time quite right, but I'd put it down to local weirdness as my network isn't always very stable. Maybe this wasn't so local after all?</p>
-
-<p>So, I started tracing through the scripts to figure out what was going on. It turned out that I had an empty <code>/etc/ntp.conf</code> file on my laptop. The <code>/usr/sbin/ntpdate-debian</code> script assumed that that meant I had a full NTP server installed (I don't), and fetched the list of servers from it; since the file was empty, it ended up synchronising time from no servers, that is, not synchronising at all. I removed the file and all was well.</p>
-
-<p>That left the question of where that file came from. It didn't seem to be owned by any package; I was pretty sure I hadn't created it by hand either. I had a look through some bug reports, and soon found <a href="https://bugs.launchpad.net/bugs/83604">ntpdate 1:4.2.2.p4+dfsg-1ubuntu2 has a flawed configuration file</a>. It turns out that <code>time-admin</code> (System -> Administration -> Time and Date) creates an empty <code>/etc/ntp.conf</code> file if you press the reload button (tooltip: "Synchronise now"), as part of an attempt to update NTP configuration. Aha!</p>
-
-<p>Once I knew where the problems were, it was easy to fix them. I've uploaded the following changes, which will be in the 11.04 release:</p>
-
-<ul>
- <li>Disregard empty <code>ntp.conf</code> files in <code>ntpdate-debian</code>.</li>
- <li>Remove an empty <code>/etc/ntp.conf</code> file on fresh installation of the <code>ntp</code> package, so that it doesn't interfere with creating the normal configuration file.</li>
- <li>Don't create the NTP configuration file in the <code>time-admin</code> backend if it doesn't exist already.</li>
-</ul>
-
-<p>I've also sent these changes to <a href="http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=606107">Debian</a> and <a href="https://bugzilla.gnome.org/show_bug.cgi?id=449267">GNOME</a> as appropriate.</p>
-
-<p>There are still a few problems. The "Synchronise now" button doesn't work quite right in general (<a href="https://bugs.launchpad.net/bugs/90524">bug #90524</a>), and if your network doesn't allow time synchronisation from <code>ntp.ubuntu.com</code> then you'll have to change the value of <code>NTPSERVERS</code> in <code>/etc/default/ntpdate</code>. Furthermore, the <code>time-admin</code> interface is confusing and makes it seem as though the default is not to synchronise the time automatically; this interface is being <a href="https://wiki.ubuntu.com/TimeAndDate">redesigned</a> at the moment, which should be a good opportunity to make it less confusing, and I will contact the designers to mention this problem. On the whole, though, I think that many fewer people should have this kind of problem in Ubuntu 11.04.</p>
-
-<p>It's always possible that I missed some other problem that breaks automatic time synchronisation for people. Please do file a bug report if it still doesn't work for you in 11.04, or contact me directly (cjwatson at ubuntu.com).</p>
+++ /dev/null
-Wubi bug 693671
-
-<p>I spent most of last week working on <a href="https://bugs.launchpad.net/bugs/693671">Ubuntu bug 693671</a> ("wubi install will not boot - phase 2 stops with: Try (hd0,0): NTFS5"), which was quite a challenge to debug since it involved digging into parts of the Wubi boot process I'd never really touched before. Since I don't think much of this is very well-documented, I'd like to spend a bit of time explaining what was involved, in the hope that it will help other developers in the future.</p>
-
-<p><a href="http://en.wikipedia.org/wiki/Wubi_%28Ubuntu_installer%29">Wubi</a> is a system for installing Ubuntu into a file in a Windows filesystem, so that it doesn't require separate partitions and can be uninstalled like any other Windows application. The purpose of this is to make it easy for Windows users to try out Ubuntu without the need to worry about repartitioning, before they commit to a full installation. Wubi started out as an external project, and initially patched the installer on the fly to do all the rather unconventional things it needed to do; we integrated it into Ubuntu 8.04 LTS, which involved turning these patches into proper installer facilities that could be accessed using preseeding, so that Wubi only needs to handle the Windows user interface and other Windows-specific tasks.</p>
-
-<p>Anyone familiar with a GNU/Linux system's boot process will immediately see that this isn't as simple as it sounds. Of course, <a href="http://www.tuxera.com/community/ntfs-3g-download/">ntfs-3g</a> is a pretty solid piece of software so we can handle the Windows filesystem without too much trouble, and loopback mounts are well-understood so we can just have the initramfs loop-mount the root filesystem. Where are you going to get the kernel and initramfs from, though? Well, we used to copy them out to the NTFS filesystem so that GRUB could read them, but this was overly complicated and error-prone. When we switched to GRUB 2, we could instead use its built-in loopback facilities, and we were able to simplify this. So all was more or less well, except for the elephant in the room. How are you going to load GRUB?</p>
-
-<p>In a Wubi installation, NTLDR (or BOOTMGR in Windows Vista and newer) still owns the boot process. Ubuntu is added as a boot menu option using BCDEdit. You might then think that you can just have the Windows boot loader chain-load GRUB. Unfortunately, NTLDR only loads 16 sectors - 8192 bytes - from disk. GRUB won't fit in that: the smallest core.img you can generate at the moment is over 18 kilobytes. Thus, you need something that is small enough to be loaded by NTLDR, but that is intelligent enough to understand NTFS to the point where it can find a particular file in the root directory of a filesystem, load boot loader code from it, and jump to that. The answer for this was <a href="http://gna.org/projects/grub4dos/">GRUB4DOS</a>. Most of GRUB4DOS is based on GRUB Legacy, which is not of much interest to us any more, but it includes an assembly-language program called GRLDR that supports doing this very thing for FAT, NTFS, and ext2. In Wubi, we build GRLDR as <code>wubildr.mbr</code>, and build a specially-configured GRUB core image as <code>wubildr</code>.</p>
-
-<p>Now, the messages shown in the bug report suggested a failure either within GRLDR or very early in GRUB. The first thing I did was to remember that GRLDR has been integrated into the grub-extras <code>ntldr-img</code> module suitable for use with GRUB 2, so I tried building <code>wubildr.mbr</code> from that; no change, but this gave me a modern baseline to work on. OK; now to try QEMU (you can use tricks like <code>qemu -hda /dev/sda</code> if you're very careful not to do anything that might involve writing to the host filesystem from within the guest, such as recursively booting your host OS ... [<strong>update:</strong> Tollef Fog Heen and Zygmunt Krynicki both point out that you can use the <code>-snapshot</code> option to make this safer]). No go; it hung somewhere in the middle of NTLDR. Still, I could at least insert debug statements, copy the built <code>wubildr.mbr</code> over to my test machine, and reboot for each test, although it would be slow and tedious. Couldn't I?</p>
-
-<p>Well, yes, I mostly could, but that 8192-byte limit came back to bite me, along with an internal 2048-byte limit that GRLDR allocates for its NTFS bootstrap code. There were only a few spare bytes. Something like this would more or less fit, to print a single mark character at various points so that I could see how far it was getting:</p>
-
-<pre>
- pushal
- xorw %bx, %bx /* video page 0 */
- movw $0x0e4d, %ax /* print 'M' */
- int $0x10
- popal
-</pre>
-
-<p>In a few places, if I removed some code I didn't need on my test machine (say, CHS compatibility), I could even fit in cheap and nasty code to print a single register in hex (as long as you didn't mind 'A' to 'F' actually being ':' to '?' in ASCII; and note that this is real-mode code, so the loop counter is <code>%cx</code> not <code>%ecx</code>):</p>
-
-<pre>
- /* print %edx in dumbed-down hex */
- pushal
- xorw %bx, %bx
- movb $0xe, %ah
- movw $8, %cx
-1:
- roll $4, %edx
- movb %dl, %al
- andb $0xf, %al
- int $0x10
- loop 1b
- popal
-</pre>
-
-<p>After a considerable amount of work tracking down problems by bisection like this, I also observed that GRLDR's NTFS code bears quite a bit of resemblance in its logical flow to GRUB 2's NTFS module, and indeed the same person wrote much of both. Since I knew that the latter worked, I could use it to relieve my brain of trying to understand assembly code logic directly, and could compare the two to look for discrepancies. I did find a few of these, and corrected a simple one. Testing at this point suggested that the boot process was getting as far as GRUB but still wasn't printing anything. I removed some Ubuntu patches which quieten down GRUB's startup: still nothing - so I switched my attentions to <a href="http://bazaar.launchpad.net/~vcs-imports/grub/grub2-bzr/view/head:/grub-core/kern/i386/pc/startup.S">grub-core/kern/i386/pc/startup.S</a>, which contains the first code executed from GRUB's core image. Code before the first call to <code>real_to_prot</code> (which switches the processor into protected mode) succeeded, while code after that point failed. Even more mysteriously, code added to <code>real_to_prot</code> <em>before</em> the actual switch to protected mode failed too. Now I was clearly getting somewhere interesting, but what was going on? What I really wanted was to be able to single-step, or at least see what was at the memory location it was supposed to be jumping to.</p>
-
-<p>Around this point I was venting on IRC, and somebody asked if it was reproducible in QEMU. Although I'd tried that already, I went back and tried again. Ubuntu's <code>qemu</code> is actually built from qemu-kvm, and if I used <code>qemu -no-kvm</code> then it worked much better. Excellent! Now I could use GDB:</p>
-
-<pre>
-(gdb) target remote | qemu -gdb stdio -no-kvm -hda /dev/sda
-</pre>
-
-<p>This let me run until the point when NTLDR was about to hand over control, then interrupt and set a breakpoint at <code>0x8200</code> (the entry point of <code>startup.S</code>). This revealed that the address that should have been <code>real_to_prot</code> was in fact garbage. I set a breakpoint at <code>0x7c00</code> (GRLDR's entry point) and stepped all the way through to ensure it was doing the right thing. In the process it was helpful to know that <a href="http://sourceware.org/ml/gdb/2009-01/msg00008.html">GDB and QEMU don't handle real mode very well between them</a>. Useful tricks here were:</p>
-
-<ul>
- <li>Use <code>set architecture i8086</code> before disassembling real-mode code (and <code>set architecture i386</code> to switch back).</li>
- <li>GDB prints addresses relative to the current segment base, but if you want to enter an address then you need to calculate a linear address yourself. For example, breakpoints must be set at <code>(CS << 4) + IP</code>, rather than just at <code>IP</code>.</li>
-</ul>
-
-<p>Single-stepping showed that GRLDR was loading the entirety of <code>wubildr</code> correctly and jumping to it. The first instruction it jumped to wasn't in <code>startup.S</code>, though, and then I remembered that we prefix the core image with <a href="http://bazaar.launchpad.net/~vcs-imports/grub/grub2-bzr/view/3095/grub-core/boot/i386/pc/lnxboot.S">grub-core/boot/i386/pc/lnxboot.S</a>. Stepping through this required a clear head since it copies itself around and changes segment registers a few times. The interesting part was at <code>real_code_2</code>, where it copies a sector of the kernel to the target load address, and then checks a known offset to find out whether the "kernel" is in fact GRUB rather than a Linux kernel. I checked that offset by hand, and there was the smoking gun. GRUB recently acquired Reed-Solomon error correction on its core image, to allow it to recover from other software writing over sectors in the boot track. This moved the magic number <code>lnxboot.S</code> was checking somewhat further into the core image, after the first sector. <code>lnxboot.S</code> couldn't find it because it hadn't copied it yet! A bit of <a href="http://bazaar.launchpad.net/~vcs-imports/grub/grub2-bzr/revision/3096">adjustment</a> and all was well again.</p>
-
-<p>The lesson for me from all of this has been to try hard to get an interactive debugger working. Really hard. It's worth quite a bit of up-front effort if it saves you from killing neurons stepping through pages of code by hand. I think the real-mode debugging tricks I picked up should be useful for working on GRUB in the future.</p>
+++ /dev/null
-Top ideas on Ubuntu Brainstorm (August 2011)
-
-<p>The Ubuntu Technical Board conducts a regular review of the most popular <a href="http://brainstorm.ubuntu.com/">Ubuntu Brainstorm</a> ideas (previous reviews conducted by <a href="http://mdzlog.alcor.net/2010/12/10/ubuntu-brainstorm-top-10-for-december-2010/">Matt Zimmerman</a> and <a href="http://www.piware.de/2011/04/top-ideas-on-ubuntu-brainstorm-march-2011/">Martin Pitt</a>). This time it was my turn. Apologies for the late arrival of this review.</p>
-
-<h2>Contact lens in the Unity Dash (<a href="http://brainstorm.ubuntu.com/idea/27584/">#27584</a>)</h2>
-
-<p>Unity supports <a href="https://wiki.ubuntu.com/Unity/Lenses">Lenses</a>, which provide a consistent way for users to quickly search for information via the Dash. Current lenses include Applications, Files, and Music, but a number of people have asked for contacts to be accessible using the same interface.</p>
-
-<p>While Canonical's DX team isn't currently working on this for Ubuntu 11.10 or 12.04, we'd love somebody who's interested in this to get involved. Allison Randal <a href="http://allisonrandal.com/2011/09/27/contacts-lens/">explains how to get started</a>, including some skeleton example code and several useful links.</p>
-
-<h2>Displaying Ubuntu version information (<a href="http://brainstorm.ubuntu.com/idea/27460/">#27460</a>)</h2>
-
-<p>Several people have asked for it to be more obvious what Ubuntu version they're running, as well as other general information about their system.</p>
-
-<p>John Lea, user experience architect on the Unity team, responds that in Ubuntu 11.10 the new LightDM greeter shows the Ubuntu version number, making that basic information very easily visible. For more detail, System Settings -> System Info provides a simple summary.</p>
-
-<h2>Volume adjustments for headphone use (<a href="http://brainstorm.ubuntu.com/idea/27275/">#27275</a>)</h2>
-
-<p>People often find that they need to adjust their sound volume when plugging in or removing headphones. It seems as though the computer ought to be able to remember this kind of thing and do it automatically; after all, a major goal of Ubuntu is to make the desktop Just Work.</p>
-
-<p>David Henningson, a member of Canonical's OEM Services group and an Ubuntu audio developer, <a href="http://voices.canonical.com/david.henningsson/2011/09/29/independent-volume-for-headphones-and-speakers/">responds</a> on his blog with a summary of how PulseAudio jack detection has improved matters in Ubuntu 11.10, and what's left to do:</p>
-
-<blockquote>
-<p>The good news: in the upcoming Ubuntu Oneiric (11.10), this is actually working. The bad news: it isn't working for everyone.</p>
-</blockquote>
-
-<h2>Making it easier to find software to handle a file (<a href="http://brainstorm.ubuntu.com/idea/28148/">#28148</a>)</h2>
-
-<p>Ubuntu is not always as helpful as it could be when you don't have the right software installed to handle a particular file.</p>
-
-<p>Michael Vogt, one of the developers of the Ubuntu Software Center, responded to this. It seems that most of the pieces to make this work nicely are in place, but there are a few more bits of glue required:</p>
-
-<blockquote>
-<p>Thanks a lot for this suggestion. I like the idea and it's something that software-center itself supports now. In the coming version 5.0 we will offer to "sort by top-rated" (based on the ratings&reviews data). It's also possible to search for an application based on its mime data. To search for a mime-type, you can enter "mime:text/html" or "mime:audio/ogg" into the search field. What is needed however is better integration into the file manager nautilus. I will make sure this gets attention at the next developer meeting and filed <a href="https://launchpad.net/bugs/860536">bug #860536</a> about it.</p>
-
-<p>In nautilus, there is now a button called "Find applications online" available as an option when opening an unknown file or when the user selects "open with...other application" in the context menu. But that will not use the data from software-center.</p>
-</blockquote>
-
-<h2>Show pop-up alert on low battery (<a href="http://brainstorm.ubuntu.com/idea/28037/">#28037</a>)</h2>
-
-<p>Some users have reported on Brainstorm that they are not alerted frequently enough when their laptop's battery is low, as they clearly ought to be.</p>
-
-<p>This is an odd one, because there are already several power alert levels and this has been working well for us for some time. Nevertheless, enough people have voted for this idea that there must be something behind it, perhaps a bug that only affects certain systems. Martin Pitt, technical lead of the Ubuntu desktop team, has <a href="http://brainstorm.ubuntu.com/idea/28037/">responded</a> directly to the Brainstorm idea with a description of the current system and how to file a bug when it does not work as intended.</p>
+++ /dev/null
-Quality in Ubuntu 12.04 LTS
-
-<p>As is natural for an LTS cycle, lots of people are thinking and talking about work focused on quality rather than features. With Canonical <a href="http://www.canonical.com/content/ubuntu-1204-feature-extended-support-period-desktop-users">extending LTS support</a> to five years on the desktop for 12.04, much of this is quite rightly focused on the desktop. I'm really not a desktop hacker in any way, shape, or form, though. I spent my first few years in Ubuntu working mainly on the installer - I still do, although I do some other things now too - and I used to say only half-jokingly that my job was done once X started. Of course there are plenty of bugs I can fix, but I wanted to see if I could do something with a bit more structure, so I got to thinking about projects we could work on at the foundations level that would make a big difference.</p>
-
-<h2>Image build pipeline</h2>
-
-<p>One difficulty we have is that quite a few of our bugs - especially installer bugs, although this goes for some other things too - are only really caught when people are doing coordinated image testing just before a milestone release. Now, it takes a while to do all the builds and then it takes a while to test them. The excellent work of the QA team has meant that testing is much quicker now than it used to be, and a certain amount of smoke-testing is automated (particularly for server images). On the other hand, the build phase has only got longer as we've added more flavours and architectures, particularly as some parts of the process are still serialised per architecture or subarchitecture so ARM builds in particular take a very long time indeed. Exact timings are a bit difficult to get for various reasons, but I think the minimum time between a developer uploading a fix and us having a full set of candidate images on all architectures including that fix is currently somewhere north of eight hours, and that's with people cutting corners and pulling strings which is a suboptimal thing to have to do around release time. This obviously makes us reluctant to respin for anything short of showstopper bugs. If we could get things down to something closer to two hours, respins would be a much less horrible proposition and so we might be able to fix a few bugs that are serious but not showstoppers, not to mention that the release team would feel less burned out.</p>
-
-<p>We discussed this problem at the release sprint, and came up with a <a href="https://blueprints.launchpad.net/ubuntu/+spec/foundations-p-image-build-pipeline">laundry list of improvements</a>; I've scheduled this for discussion at UDS in case we can think of any more. Please come along if you're interested!</p>
-
-<p>One thing in particular that I'm working on is refactoring <a href="https://launchpad.net/germinate">Germinate</a>, a tool which dates right back to our first meeting before Ubuntu was even called Ubuntu and whose job is to expand dependencies starting from our lists of "seed" packages; we use this, among other things, to generate <code>Task</code> fields in the archive and to decide which packages to copy into our images. This was acceptably quick in 2004, but now that we run it forty times (eight flavours multiplied by five architectures) at the end of every publisher run it's actually become rather a serious performance problem: <code>cron.germinate</code> takes about ten minutes, which is over a third of the typical publisher runtime. It parses Packages files eight times as often as it needs to, Sources files forty times as often as it needs to, and recalculates the dependency tree of the base system five times as often as it needs to. I am confident that we can significantly reduce the runtime here, and I think there's some hope that we might be able to move the publisher back to a 30-minute cycle, which would increase the velocity of Ubuntu development in general.</p>
-
-<h2>Maintaining the development release</h2>
-
-<p>Our release cycle always starts with syncing and merging packages from Debian unstable (or testing in the case of LTS cycles). The vast majority of packages in Ubuntu arrive this way, and generally speaking if we didn't do this we would fall behind in ways that would be difficult to recover from later. However, this does mean that we get a "big bang" of changes at the start of the cycle, and it takes a while for the archive to be usable again. Furthermore, even once we've taken care of this, we have a long-established rhythm where the first part of the cycle is mainly about feature development and the second part of the cycle is mainly about stabilisation. As a result, we've got used to the archive being fairly broken for the first few months, and we even tell people that they shouldn't expect things to work reliably until somewhere approaching beta.</p>
-
-<p>This makes some kind of sense from the inside. But how are you supposed to do feature development that relies on other things in the development release?</p>
-
-<p>In the first few years of Ubuntu, this question didn't matter very much. Nearly all the people doing serious feature development were themselves serious Ubuntu developers; they were capable of fixing problems in the development release as they went along, and while it got in their way a little bit it wasn't all that big a deal. Now, though, we have people focusing on things like Unity development, and we shouldn't assume that just because somebody is (say) an OpenGL expert or a window management expert that they should be able to recover from arbitrary failures in development release upgrades. One of the best things we could do to help the 12.04 desktop be more stable is to have the entire system be less unstable as we go along, so that developers further up the stack don't have to be distracted by things wobbling underneath them. Plus, it's just good software engineering to keep the basics working as you go along: it should always build, it should always install, it should always upgrade. Ubuntu is too big to do something like having everyone stop any time the build breaks, the way you might do in a smaller project, but we shouldn't let things slide for months either.</p>
-
-<p>I've been talking to <a href="http://theravingrick.blogspot.com/">Rick Spencer</a> and the other Ubuntu engineering leads at Canonical about this. Canonical has a system of "rotations", where you can go off to another team for a while if you're in need of a change or want to branch out a bit; so I proposed that we allow our engineers to spend a month or two at a time on what I'm calling the <strong>+1 Maintenance Team</strong>, whose job is simply to keep the development release buildable, installable, and upgradeable at all times. Rick has been very receptive to this, and we're going to be running this as a trial throughout the 12.04 cycle, with probably about three people at a time. As well as being professional archive gardeners, these people will also work on developing infrastructure to help us keep better track of what we need to do. For instance, we could deploy better tools from Debian QA to help us track uninstallable packages, or we could enhance <a href="http://people.canonical.com/~ubuntu-archive/nbs.html">some</a> of our <a href="http://conflictchecker.ubuntu.com/possible-conflicts/oneiric/main.txt">many</a> <a href="http://people.canonical.com/~ubuntu-archive/component-mismatches.txt">existing</a> <a href="http://people.canonical.com/~ubuntu-archive/testing/precise_probs.html">reports</a> to have bug links and/or comment facilities, or we could spruce up the <a href="http://reports.qa.ubuntu.com/reports/ogasawara/weatherreport.html">weather report</a>; there are lots of things we could do to make our own lives easier.</p>
-
-<p>By 12.04, I would like, in no particular order:</p>
-
-<ul>
- <li>Precise to have been more or less continuously usable from Alpha 1 onward for people with reasonable general technical ability</li>
- <li>Canonical engineering teams outside Ubuntu (DX, Ubuntu One, Launchpad, etc.) to be comfortable with running the development release on at least one system from Alpha 2 onward</li>
- <li>Installability problems in daily image builds to be dealt with within one working day, or preferably before they even make it to daily builds</li>
- <li>The archive to be close to consistent as we start milestone preparation, rather than the release team having to scramble to make it so</li>
- <li>A very significant reduction in our long-term backlog of automatically-detected problems</li>
-</ul>
-
-<p>Of course, this overlaps to a certain extent with the kinds of things that the MOTU team have been doing for years, not to mention with what all developers should be doing to keep their own houses in reasonable order, and I'd like us to work together on this; we're trying to provide some extra hands here to make Ubuntu better for everyone, not take over! I would love this to be an opportunity to re-energise MOTU and bring some new people on board.</p>
-
-<p>I've registered a couple of blueprints (<a href="https://blueprints.launchpad.net/ubuntu/+spec/other-p-plusonemaint-priorities">priorities</a>, <a href="https://blueprints.launchpad.net/ubuntu/+spec/other-p-plusonemaint-infrastructure">infrastructure</a>) for discussion at UDS. These are deliberately open-ended skeleton sessions, and I'll try to make sure they're scheduled fairly early in the week, so that we have time for break-out sessions later on. If you're interested, please come along and give your feedback!</p>
+++ /dev/null
-APT resolver bugs
-
-<p>I've managed to go for eleven years working on Debian and nearly eight on Ubuntu without ever needing to teach myself how APT's resolver works. I get the impression that there's a certain mystique about it in general (alternatively, I'm just the last person to figure this out). Recently, though, I had a couple of Ubuntu upgrade bugs to fix that turned out to be bugs in the resolver, and I thought it might be interesting to walk through the process of fixing them based on the <code>Debug::pkgProblemResolver=true</code> log files.</p>
-
-<h2>Breakage with Breaks</h2>
-
-<p>The first was <a href="https://bugs.launchpad.net/bugs/922485">Ubuntu bug #922485</a> (<a href="https://launchpadlibrarian.net/91187038/apt.log">apt.log</a>). To understand the log, you first need to know that APT makes up to ten passes of the resolver to attempt to fix broken dependencies by upgrading, removing, or holding back packages; if there are still broken packages after this point, it's generally because it's got itself stuck in some kind of loop, and it bails out rather than carrying on forever. The current pass number is shown in each "Investigating" log entry, so they start with "Investigating (0)" and carry on up to at most "Investigating (9)". Any packages that you see still being investigated on the tenth pass are probably something to do with whatever's going wrong.</p>
-
-<p>In this case, most packages have been resolved by the end of the fourth pass, but <code>xserver-xorg-core</code> is causing some trouble. (Not a particular surprise, as it's an important package with lots of relationships.) We can see that each breakage is:</p>
-
-<pre>
-Broken xserver-xorg-core:i386 Breaks on xserver-xorg-video-6 [ i386 ] < none > ( none )
-</pre>
-
-<p>This is a <a href="http://www.debian.org/doc/debian-policy/ch-relationships.html#s-breaks"><code>Breaks</code></a> (a relatively new package relationship type introduced a few years ago as a sort of weaker form of <code>Conflicts</code>) on a virtual package, which means that in order to unpack <code>xserver-xorg-core</code> each package that provides <code>xserver-xorg-video-6</code> must be deconfigured. Much like <code>Conflicts</code>, APT responds to this by upgrading providing packages to versions that don't provide the offending virtual package if it can, and otherwise removing them. We can see it doing just that in the log (some lines omitted):</p>
-
-<pre>
-Investigating (0) xserver-xorg-core [ i386 ] < 2:1.7.6-2ubuntu7.10 -> 2:1.11.3-0ubuntu8 > ( x11 )
- Fixing xserver-xorg-core:i386 via remove of xserver-xorg-video-tseng:i386
-Investigating (1) xserver-xorg-core [ i386 ] < 2:1.7.6-2ubuntu7.10 -> 2:1.11.3-0ubuntu8 > ( x11 )
- Fixing xserver-xorg-core:i386 via remove of xserver-xorg-video-i740:i386
-Investigating (2) xserver-xorg-core [ i386 ] < 2:1.7.6-2ubuntu7.10 -> 2:1.11.3-0ubuntu8 > ( x11 )
- Fixing xserver-xorg-core:i386 via remove of xserver-xorg-video-nv:i386
-</pre>
-
-<p>OK, so that makes sense - presumably upgrading those packages didn't help at the time. But look at the pass numbers. Rather than just fixing all the packages that provide <code>xserver-xorg-video-6</code> in a single pass, which it would be perfectly able to do, it only fixes one per pass. This means that if a package <code>Breaks</code> a virtual package which is provided by more than ten installed packages, the resolver will fail to handle that situation. On inspection of the code, this was being handled correctly for <code>Conflicts</code> by carrying on through the list of possible targets for the dependency relation in that case, but apparently when <code>Breaks</code> support was implemented in APT this case was overlooked. The fix is to carry on through the list of possible targets for any "negative" dependency relation, not just <code>Conflicts</code>, and I've filed a patch as <a href="http://bugs.debian.org/657695">Debian bug #657695</a>.</p>
-
-<h2>My cup overfloweth</h2>
-
-<p>The second bug I looked at was <a href="https://bugs.launchpad.net/bugs/917173">Ubuntu bug #917173</a> (<a href="https://launchpadlibrarian.net/90202820/apt.log">apt.log</a>). Just as in the previous case, we can see the resolver "running out of time" by reaching the end of the tenth pass with some dependencies still broken. This one is a lot less obvious, though. The last few entries clearly indicate that the resolver is stuck in a loop:</p>
-
-<pre>
-Investigating (8) dpkg [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( admin )
-Broken dpkg:i386 Breaks on dpkg-dev [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( utils ) (< 1.15.8)
- Considering dpkg-dev:i386 29 as a solution to dpkg:i386 7205
- Upgrading dpkg-dev:i386 due to Breaks field in dpkg:i386
-Investigating (8) dpkg-dev [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( utils )
-Broken dpkg-dev:i386 Depends on libdpkg-perl [ i386 ] < none -> 1.16.1.2ubuntu5 > ( perl ) (= 1.16.1.2ubuntu5)
- Considering libdpkg-perl:i386 12 as a solution to dpkg-dev:i386 29
- Holding Back dpkg-dev:i386 rather than change libdpkg-perl:i386
-Investigating (9) dpkg [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( admin )
-Broken dpkg:i386 Breaks on dpkg-dev [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( utils ) (< 1.15.8)
- Considering dpkg-dev:i386 29 as a solution to dpkg:i386 7205
- Upgrading dpkg-dev:i386 due to Breaks field in dpkg:i386
-Investigating (9) dpkg-dev [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( utils )
-Broken dpkg-dev:i386 Depends on libdpkg-perl [ i386 ] < none -> 1.16.1.2ubuntu5 > ( perl ) (= 1.16.1.2ubuntu5)
- Considering libdpkg-perl:i386 12 as a solution to dpkg-dev:i386 29
- Holding Back dpkg-dev:i386 rather than change libdpkg-perl:i386
-</pre>
-
-<p>The new version of <code>dpkg</code> requires upgrading <code>dpkg-dev</code>, but it can't because of something wrong with <code>libdpkg-perl</code>. Following the breadcrumb trail back through the log, we find:</p>
-
-<pre>
-Investigating (1) libdpkg-perl [ i386 ] < none -> 1.16.1.2ubuntu5 > ( perl )
-Broken libdpkg-perl:i386 Depends on perl [ i386 ] < 5.10.1-8ubuntu2.1 -> 5.14.2-6ubuntu1 > ( perl )
- Considering perl:i386 1472 as a solution to libdpkg-perl:i386 12
- Holding Back libdpkg-perl:i386 rather than change perl:i386
-</pre>
-
-<pre>
-Investigating (1) perl [ i386 ] < 5.10.1-8ubuntu2.1 -> 5.14.2-6ubuntu1 > ( perl )
-Broken perl:i386 Depends on perl-base [ i386 ] < 5.10.1-8ubuntu2.1 -> 5.14.2-6ubuntu1 > ( perl ) (= 5.14.2-6ubuntu1)
- Considering perl-base:i386 5806 as a solution to perl:i386 1472
- Removing perl:i386 rather than change perl-base:i386
-</pre>
-
-<pre>
-Investigating (1) perl-base [ i386 ] < 5.10.1-8ubuntu2.1 -> 5.14.2-6ubuntu1 > ( perl )
-Broken perl-base:i386 PreDepends on libc6 [ i386 ] < 2.11.1-0ubuntu7.8 -> 2.13-24ubuntu2 > ( libs ) (>= 2.11)
- Considering libc6:i386 -17473 as a solution to perl-base:i386 5806
- Added libc6:i386 to the remove list
-</pre>
-
-<pre>
-Investigating (0) libc6 [ i386 ] < 2.11.1-0ubuntu7.8 -> 2.13-24ubuntu2 > ( libs )
-Broken libc6:i386 Depends on libc-bin [ i386 ] < 2.11.1-0ubuntu7.8 -> 2.13-24ubuntu2 > ( libs ) (= 2.11.1-0ubuntu7.8)
- Considering libc-bin:i386 10358 as a solution to libc6:i386 -17473
- Removing libc6:i386 rather than change libc-bin:i386
-</pre>
-
-<p>So ultimately the problem is something to do with libc6; but what? <a href="https://bugs.launchpad.net/ubuntu/+source/apt/+bug/917173/comments/10">As Steve Langasek said in the bug</a>, libc6's dependencies have been very carefully structured, and surely we would have seen some hint of it elsewhere if they were wrong. At this point ideally I wanted to break out GDB or at the very least experiment a bit with <code>apt-get</code>, but due to some tedious local problems I hadn't been able to restore the <code>apt-clone</code> state file for this bug onto my system so that I could attack it directly. So I fell back on the last refuge of the frustrated debugger and sat and thought about it for a bit.</p>
-
-<p>Eventually I noticed something. The numbers after the package names in the third line of each of these log entries are "scores": roughly, the more important a package is, the higher its score should be. The function that calculates these is <code>pkgProblemResolver::MakeScores()</code> in <a href="http://bazaar.launchpad.net/+branch/apt/view/1951/apt-pkg/algorithms.cc">apt-pkg/algorithms.cc</a>. Reading this, I noticed that the various values added up to make each score are almost all provably positive, for example:</p>
-
-<pre>
- Scores[I->ID] += abs(OldScores[D.ParentPkg()->ID]);
-</pre>
-
-<p>The only exceptions are an initial -1 or -2 points for <code>Priority: optional</code> or <code>Priority: extra</code> packages respectively, or some values that could theoretically be configured to be negative but weren't in this case. OK. So how come <code>libc6</code> has such a huge negative score of -17473, when one would normally expect it to be an extremely powerful package with a large positive score?</p>
-
-<p>Oh. This is computer programming, not mathematics ... and each score is stored in a <code>signed short</code>, so in a sufficiently large upgrade all those bonus points add up to something larger than 32767 and everything goes haywire. Bingo. Make it an <code>int</code> instead - the number of installed packages is going to be on the order of tens of thousands at most, so it's not as though it'll make a substantial difference to the amount of memory used - and chances are everything will be fine. I've filed a patch as <a href="http://bugs.debian.org/657732">Debian bug #657732</a>.</p>
-
-<p>I'd expected this to be a pretty challenging pair of bugs. While I certainly haven't lost any respect for the APT maintainers for dealing with this stuff regularly, it wasn't as bad as I thought. I'd expected to have to figure out how to retune some slightly out-of-balance heuristics and not really know whether I'd broken anything else in the process; but in the end both patches were very straightforward.</p>
+++ /dev/null
-Automatic installability checking
-
-<p>I've just finished deploying automatic installability checking for Ubuntu's development release, which is more or less equivalent to the way that uploads are promoted from Debian unstable to testing. See <a href="https://lists.ubuntu.com/archives/ubuntu-devel/2012-October/036043.html">my ubuntu-devel post</a> and <a href="https://lists.ubuntu.com/archives/ubuntu-devel-announce/2012-October/000989.html">my ubuntu-devel-announce post</a> for details. This now means that we'll be opening the archive for general development once glibc 2.16 packages are ready.</p>
-
-<p>I'm very excited about this because it's something I've wanted to do for a long, long time. In fact, back in 2004 when I had my very first telephone conversation with a certain spaceman about this crazy Debian-based project he wanted me to work on, I remember talking about Debian's testing migration system and some ways I thought it could be improved. I don't remember the details of that conversation any more and what I just deployed may well bear very little resemblance to it, but it should transform the extent to which our development release is continuously usable.</p>
-
-<p>The next step is to hook in <a href="http://dep.debian.net/deps/dep8/">autopkgtest</a> results. This will allow us to do a degree of automatic testing of reverse-dependencies when we upgrade low-level libraries.</p>
+++ /dev/null
-Moving on, but not too far
-
-<p>The <a href="http://www.ubuntu.com/about/about-ubuntu/conduct">Ubuntu Code of Conduct</a> says:</p>
-
-<blockquote><p><strong>Step down considerately</strong>: When somebody leaves or disengages from the project, we ask that they do so in a way that minimises disruption to the project. They should tell people they are leaving and take the proper steps to ensure that others can pick up where they left off.</p></blockquote>
-
-<p>I've been working on Ubuntu for over ten years now, almost right from the very start; I'm Canonical's employee #17 due to working out a notice period in my previous job, but I was one of the founding group of developers. I occasionally tell the story that Mark originally hired me mainly to work on what later became Launchpad Bugs due to my experience maintaining the Debian bug tracking system, but then not long afterwards Jeff Waugh got in touch and said "hey Colin, would you mind just sorting out some installable CD images for us?". This is where you imagine one of those movie time-lapse clocks ... At some point it became fairly clear that I was working on Ubuntu, and the bug system work fell to other people. Then, when Matt Zimmerman could no longer manage the entire Ubuntu team in Canonical by himself, Scott James Remnant and I stepped up to help him out. I did that for a couple of years, starting the Foundations team in the process. As the team grew I found that my interests really lay in hands-on development rather than in management, so I switched over to being the technical lead for Foundations, and have made my home there ever since. Over the years this has given me the opportunity to do all sorts of things, particularly working on our installers and on the GRUB boot loader, leading the development work on many of our archive maintenance tools, instituting the +1 maintenance effort and proposed-migration, and developing the Click package manager, and I've had the great pleasure of working with many exceptionally talented people.</p>
-
-<p>However. In recent months I've been feeling a general sense of malaise and what I've come to recognise with hindsight as the symptoms of approaching burnout. I've been working long hours for a long time, and while I can draw on a lot of experience by now, it's been getting harder to summon the enthusiasm and creativity to go with that. I have a wonderful wife, amazing children, and lovely friends, and I want to be able to spend a bit more time with them. After ten years doing the same kinds of things, I've accreted history with and responsibility for a lot of projects. One of the things I always loved about Foundations was that it's a broad church, covering a wide range of software and with a correspondingly wide range of opportunities; but, over time, this has made it difficult for me to focus on things that are important because there are so many areas where I might be called upon to help. I thought about simply stepping down from the technical lead position and remaining in the same team, but I decided that that wouldn't make enough of a difference to what matters to me. I need a clean break and an opportunity to reset my habits before I burn out for real.</p>
-
-<p>One of the things that has consistently held my interest through all of this has been making sure that the infrastructure for Ubuntu keeps running reliably and that other developers can work efficiently. As part of this, I've been able to do <a href="https://dev.launchpad.net/Contributions#colin_watson">a lot of work</a> over the years on <a href="https://launchpad.net/">Launchpad</a> where it was a good fit with my remit: this has included significant performance improvements to archive publishing, moving most archive administration operations from excessively-privileged command-line operations to the webservice, making build cancellation reliable across the board, and moving live filesystem building from an unscalable ad-hoc collection of machines into the Launchpad build farm. The Launchpad development team has generally welcomed help with open arms, and in fact I joined the <a href="https://launchpad.net/~launchpad">~launchpad team</a> last year.</p>
-
-<p>So, the logical next step for me is to make this informal involvement permanent. As such, at the end of this year I will be moving from Ubuntu Foundations to the Launchpad engineering team.</p>
-
-<p>This doesn't mean me leaving Ubuntu. Within Canonical, Launchpad development is currently organised under the Continuous Integration team, which is part of Ubuntu Engineering. I'll still be around in more or less the usual places and available for people to ask me questions. But I will in general be trying to reduce my involvement in Ubuntu proper to things that are closely related to the operation of Launchpad, and a small number of low-effort things that I'm interested enough in to find free time for them. I still need to sort out a lot of details, but it'll very likely involve me handing over project leadership of Click, drastically reducing my involvement in the installer, and looking for at least some help with boot loader work, among others. I don't expect my Debian involvement to change, and I may well find myself more motivated there now that it won't be so closely linked with my day job, although it's possible that I will pare some things back that I was mostly doing on Ubuntu's behalf. If you ask me for help with something over the next few months, expect me to be more likely to direct you to other people or suggest ways you can help yourself out, so that I can start disentangling myself from my current web of projects.</p>
-
-<p>Please contact me sooner or later if you're interested in helping out with any of the things I'm visible in right now, and we can see what makes sense. I'm looking forward to this!</p>
~cjwatson / blog.git / commitdiff
