APT resolver bugs

diff --git a/ubuntu/2012-01-29-apt-resolver-bugs.txt b/ubuntu/2012-01-29-apt-resolver-bugs.txt
new file mode 100644 (file)
index 0000000..2c0aaab
--- /dev/null
+++ b/ubuntu/2012-01-29-apt-resolver-bugs.txt
@@ -0,0 +1,93 @@
+APT resolver bugs
+
+<p>I've managed to go for eleven years working on Debian and nearly eight on Ubuntu without ever needing to teach myself how APT's resolver works.  I get the impression that there's a certain mystique about it in general (alternatively, I'm just the last person to figure this out).  Recently, though, I had a couple of Ubuntu upgrade bugs to fix that turned out to be bugs in the resolver, and I thought it might be interesting to walk through the process of fixing them based on the <code>Debug::pkgProblemResolver=true</code> log files.</p>
+
+<h2>Breakage with Breaks</h2>
+
+<p>The first was <a href="https://bugs.launchpad.net/bugs/922485">Ubuntu bug #922485</a> (<a href="https://launchpadlibrarian.net/91187038/apt.log">apt.log</a>).  To understand the log, you first need to know that APT makes up to ten passes of the resolver to attempt to fix broken dependencies by upgrading, removing, or holding back packages; if there are still broken packages after this point, it's generally because it's got itself stuck in some kind of loop, and it bails out rather than carrying on forever.  The current pass number is shown in each "Investigating" log entry, so they start with "Investigating (0)" and carry on up to at most "Investigating (9)".  Any packages that you see still being investigated on the tenth pass are probably something to do with whatever's going wrong.</p>
+
+<p>In this case, most packages have been resolved by the end of the fourth pass, but <code>xserver-xorg-core</code> is causing some trouble.  (Not a particular surprise, as it's an important package with lots of relationships.)  We can see that each breakage is:</p>
+
+<pre>
+Broken xserver-xorg-core:i386 Breaks on xserver-xorg-video-6 [ i386 ] &lt; none &gt; ( none )
+</pre>
+
+<p>This is a <a href="http://www.debian.org/doc/debian-policy/ch-relationships.html#s-breaks"><code>Breaks</code></a> (a relatively new package relationship type introduced a few years ago as a sort of weaker form of <code>Conflicts</code>) on a virtual package, which means that in order to unpack <code>xserver-xorg-core</code> each package that provides <code>xserver-xorg-video-6</code> must be deconfigured.  Much like <code>Conflicts</code>, APT responds to this by upgrading providing packages to versions that don't provide the offending virtual package if it can, and otherwise removing them.  We can see it doing just that in the log (some lines omitted):</p>
+
+<pre>
+Investigating (0) xserver-xorg-core [ i386 ] &lt; 2:1.7.6-2ubuntu7.10 -&gt; 2:1.11.3-0ubuntu8 &gt; ( x11 )
+  Fixing xserver-xorg-core:i386 via remove of xserver-xorg-video-tseng:i386
+Investigating (1) xserver-xorg-core [ i386 ] &lt; 2:1.7.6-2ubuntu7.10 -&gt; 2:1.11.3-0ubuntu8 &gt; ( x11 )
+  Fixing xserver-xorg-core:i386 via remove of xserver-xorg-video-i740:i386
+Investigating (2) xserver-xorg-core [ i386 ] &lt; 2:1.7.6-2ubuntu7.10 -&gt; 2:1.11.3-0ubuntu8 &gt; ( x11 )
+  Fixing xserver-xorg-core:i386 via remove of xserver-xorg-video-nv:i386
+</pre>
+
+<p>OK, so that makes sense - presumably upgrading those packages didn't help at the time.  But look at the pass numbers.  Rather than just fixing all the packages that provide <code>xserver-xorg-video-6</code> in a single pass, which it would be perfectly able to do, it only fixes one per pass.  This means that if a package <code>Breaks</code> a virtual package which is provided by more than ten installed packages, the resolver will fail to handle that situation.  On inspection of the code, this was being handled correctly for <code>Conflicts</code> by carrying on through the list of possible targets for the dependency relation in that case, but apparently when <code>Breaks</code> support was implemented in APT this case was overlooked.  The fix is to carry on through the list of possible targets for any "negative" dependency relation, not just <code>Conflicts</code>, and I've filed a patch as <a href="http://bugs.debian.org/657695">Debian bug #657695</a>.</p>
+
+<h2>My cup overfloweth</h2>
+
+<p>The second bug I looked at was <a href="https://bugs.launchpad.net/bugs/917173">Ubuntu bug #917173</a> (<a href="https://launchpadlibrarian.net/90202820/apt.log">apt.log</a>).  Just as in the previous case, we can see the resolver "running out of time" by reaching the end of the tenth pass with some dependencies still broken.  This one is a lot less obvious, though.  The last few entries clearly indicate that the resolver is stuck in a loop:</p>
+
+<pre>
+Investigating (8) dpkg [ i386 ] &lt; 1.15.5.6ubuntu4.5 -&gt; 1.16.1.2ubuntu5 &gt; ( admin )
+Broken dpkg:i386 Breaks on dpkg-dev [ i386 ] &lt; 1.15.5.6ubuntu4.5 -&gt; 1.16.1.2ubuntu5 &gt; ( utils ) (&lt; 1.15.8)
+  Considering dpkg-dev:i386 29 as a solution to dpkg:i386 7205
+  Upgrading dpkg-dev:i386 due to Breaks field in dpkg:i386
+Investigating (8) dpkg-dev [ i386 ] &lt; 1.15.5.6ubuntu4.5 -&gt; 1.16.1.2ubuntu5 &gt; ( utils )
+Broken dpkg-dev:i386 Depends on libdpkg-perl [ i386 ] &lt; none -&gt; 1.16.1.2ubuntu5 &gt; ( perl ) (= 1.16.1.2ubuntu5)
+  Considering libdpkg-perl:i386 12 as a solution to dpkg-dev:i386 29
+  Holding Back dpkg-dev:i386 rather than change libdpkg-perl:i386
+Investigating (9) dpkg [ i386 ] &lt; 1.15.5.6ubuntu4.5 -&gt; 1.16.1.2ubuntu5 &gt; ( admin )
+Broken dpkg:i386 Breaks on dpkg-dev [ i386 ] &lt; 1.15.5.6ubuntu4.5 -&gt; 1.16.1.2ubuntu5 &gt; ( utils ) (&lt; 1.15.8)
+  Considering dpkg-dev:i386 29 as a solution to dpkg:i386 7205
+  Upgrading dpkg-dev:i386 due to Breaks field in dpkg:i386
+Investigating (9) dpkg-dev [ i386 ] &lt; 1.15.5.6ubuntu4.5 -&gt; 1.16.1.2ubuntu5 &gt; ( utils )
+Broken dpkg-dev:i386 Depends on libdpkg-perl [ i386 ] &lt; none -&gt; 1.16.1.2ubuntu5 &gt; ( perl ) (= 1.16.1.2ubuntu5)
+  Considering libdpkg-perl:i386 12 as a solution to dpkg-dev:i386 29
+  Holding Back dpkg-dev:i386 rather than change libdpkg-perl:i386
+</pre>
+
+<p>The new version of <code>dpkg</code> requires upgrading <code>dpkg-dev</code>, but it can't because of something wrong with <code>libdpkg-perl</code>.  Following the breadcrumb trail back through the log, we find:</p>
+
+<pre>
+Investigating (1) libdpkg-perl [ i386 ] &lt; none -&gt; 1.16.1.2ubuntu5 &gt; ( perl )
+Broken libdpkg-perl:i386 Depends on perl [ i386 ] &lt; 5.10.1-8ubuntu2.1 -&gt; 5.14.2-6ubuntu1 &gt; ( perl )
+  Considering perl:i386 1472 as a solution to libdpkg-perl:i386 12
+  Holding Back libdpkg-perl:i386 rather than change perl:i386
+</pre>
+
+<pre>
+Investigating (1) perl [ i386 ] &lt; 5.10.1-8ubuntu2.1 -&gt; 5.14.2-6ubuntu1 &gt; ( perl )
+Broken perl:i386 Depends on perl-base [ i386 ] &lt; 5.10.1-8ubuntu2.1 -&gt; 5.14.2-6ubuntu1 &gt; ( perl ) (= 5.14.2-6ubuntu1)
+  Considering perl-base:i386 5806 as a solution to perl:i386 1472
+  Removing perl:i386 rather than change perl-base:i386
+</pre>
+
+<pre>
+Investigating (1) perl-base [ i386 ] &lt; 5.10.1-8ubuntu2.1 -&gt; 5.14.2-6ubuntu1 &gt; ( perl )
+Broken perl-base:i386 PreDepends on libc6 [ i386 ] &lt; 2.11.1-0ubuntu7.8 -&gt; 2.13-24ubuntu2 &gt; ( libs ) (&gt;= 2.11)
+  Considering libc6:i386 -17473 as a solution to perl-base:i386 5806
+  Added libc6:i386 to the remove list
+</pre>
+
+<pre>
+Investigating (0) libc6 [ i386 ] &lt; 2.11.1-0ubuntu7.8 -&gt; 2.13-24ubuntu2 &gt; ( libs )
+Broken libc6:i386 Depends on libc-bin [ i386 ] &lt; 2.11.1-0ubuntu7.8 -&gt; 2.13-24ubuntu2 &gt; ( libs ) (= 2.11.1-0ubuntu7.8)
+  Considering libc-bin:i386 10358 as a solution to libc6:i386 -17473
+  Removing libc6:i386 rather than change libc-bin:i386
+</pre>
+
+<p>So ultimately the problem is something to do with libc6; but what?  <a href="https://bugs.launchpad.net/ubuntu/+source/apt/+bug/917173/comments/10">As Steve Langasek said in the bug</a>, libc6's dependencies have been very carefully structured, and surely we would have seen some hint of it elsewhere if they were wrong.  At this point ideally I wanted to break out GDB or at the very least experiment a bit with <code>apt-get</code>, but due to some tedious local problems I hadn't been able to restore the <code>apt-clone</code> state file for this bug onto my system so that I could attack it directly.  So I fell back on the last refuge of the frustrated debugger and sat and thought about it for a bit.</p>
+
+<p>Eventually I noticed something.  The numbers after the package names in the third line of each of these log entries are "scores": roughly, the more important a package is, the higher its score should be.  The function that calculates these is <code>pkgProblemResolver::MakeScores()</code> in <a href="http://bazaar.launchpad.net/+branch/apt/view/1951/apt-pkg/algorithms.cc">apt-pkg/algorithms.cc</a>.  Reading this, I noticed that the various values added up to make each score are almost all provably positive, for example:</p>
+
+<pre>
+         Scores[I-&gt;ID] += abs(OldScores[D.ParentPkg()-&gt;ID]);
+</pre>
+
+<p>The only exceptions are an initial -1 or -2 points for <code>Priority: optional</code> or <code>Priority: extra</code> packages respectively, or some values that could theoretically be configured to be negative but weren't in this case.  OK.  So how come <code>libc6</code> has such a huge negative score of -17473, when one would normally expect it to be an extremely powerful package with a large positive score?</p>
+
+<p>Oh.  This is computer programming, not mathematics ... and each score is stored in a <code>signed short</code>, so in a sufficiently large upgrade all those bonus points add up to something larger than 32767 and everything goes haywire.  Bingo.  Make it an <code>int</code> instead - the number of installed packages is going to be on the order of tens of thousands at most, so it's not as though it'll make a substantial difference to the amount of memory used - and chances are everything will be fine.  I've filed a patch as <a href="http://bugs.debian.org/657732">Debian bug #657732</a>.</p>
+
+<p>I'd expected this to be a pretty challenging pair of bugs.  While I certainly haven't lost any respect for the APT maintainers for dealing with this stuff regularly, it wasn't as bad as I thought.  I'd expected to have to figure out how to retune some slightly out-of-balance heuristics and not really know whether I'd broken anything else in the process; but in the end both patches were very straightforward.</p>