Hi.

We're all familiar with the pain caused by version incompatibility.
Indeed many of the things we work on in Debian day to day are ways to
cope with it or avoid it.

We often curse upstreams, but from an upstream perspective
incompatibility is often difficult to avoid.

In my day job I'm one of the upstream team responsible for the Xen
hypervisor and associated tools.  In this talk I'm going to give an
overview of some of the compatibility difficulties we're facing and
which, sadly, we are passing on to downstreams such as Debian.

I'll go over some of the consequences and some of the approaches we
and others have used to mitigate.  I'm hoping that some of you here
might have useful suggestions either for the Debian packaging of Xen
or for us as upstreams.


Let me show you an overview of a typical Xen system, with the key
components broken out:

[slide]

(A note about terminology: Xen uses the term "dom0" to refer to the
first, privileged guest operating system, which in most systems does
all the backend work of managing the system and its real hardware.)

Many of you, even if you're Xen users, won't be familiar with many of
these pieces - particularly the internal pieces which are part of the
Xen project.  You'll see that I've drawn a line around the code for
which Xen.org is upstream.

This comes in two major pieces: the first is the Xen Hypervisor
itself.

And the second is a collection of libraries, scripts and daemons which
run (typically) in the host to do the work of managing and supporting
the system and its guests.  We call this part of the Xen system the
"tools".  (This is not to be confused with the "xen-tools" package in
Debian, which is a provisioning utility which uses the Xen toolstack
facilities.  What I'm talking about when I say "the Xen tools" is
called, in Debian, "xen-utils".)  Together with any higher layers such
as the Debian xen-tools package, libvirt, Xapi, and perhaps something
like Openstack or Ganeti, we call this the "toolstack".

However there are three very important components of the system for
which Xen.org are not the upstream.


[ Guest ]

In some ways the most important but also the easiest from a
compatibility point of view is the guest operating system.

This might be another install of the same Debian version as is running
in dom0.  Or it might be some other Free operating system which has
been specially adapted (`paravirtualised' as we call it) for running
in Xen (aka a `PV' guest).  Or, it might be an unmodified operating
system for which Xen is providing a full emulation of a PC (an `HVM'
guest) - even a proprietary system such as Windows.

The point of Xen is to provide a reliable and stable platform, and to
decouple guest operating systems from hardware and support software.
Often the guest will be run by a different person or organisation to
the host.

So we have traditionally given a very strong compatibility guarantee:
guests which are written against old versions of Xen, and work
correctly there, are supposed to work on future versions of Xen.
Currently we support guests written to run on previous Xen versions
going back to Xen 3.0, which was released in 2005.  That means we
retain the ability to service all old guest-facing hypercalls, IO and
memory management interfaces, even if they've been replaced by new
mechanisms.

Even more: for Xen versions and guests which support live migration,
we aim to support live migration of guests from old versions of Xen to
newer ones.  So if everything is working well you should be able to
upgrade your Xen hosts without needing even to reboot your guests.

This migration facility means that if you have multiple hosts and
enough spare capacity you can do what Xapi calls a "rolling upgrade".
You upgrade your entire hosting platform, one host at a time, by
migrating the guests off each host in turn.

Actually meeting this backward compatibility promise, in every detail,
is tough.

One of the most difficult aspects of it is that we don't always know
exactly how old guests behave, and sadly the interface documentation
has not always been perfect.  But almost all of these guest interfaces
have reasonable extension mechanisms: spare fields in structs;
key/value namespaces whose range of keys can be extended; and
capability negotiation mechanisms where appropriate.

That's not always sufficient because guests may be buggy, and even
very subtle changes can expose new bugs in guests.  Even when that
happens we try to cope.

For example, one very common kind of guest bug is that it gets partial
or broken support for a particular feature of newer versions of Xen.
It will then work fine on the old versions of Xen without that
feature, but can break when the feature is present and available for
use.  So for some features we have knobs the Xen administrator can
twiddle to suppress them, thus sidestepping those kind of guest bugs.

Overall what our promise means in practice is that we give forward
compatibility for guests a lot of attention in design and
implementation; and of course it means we treat compatibility bugs as
regressions deserving a fix.

As we'll see this guarantee also causes us some difficulties with our
arrangements for providing a fully virtualised PC for HVM guests.


[ Kernel ]

Another big part of a system for which we're not upstream is the dom0
kernel.

Most Xen installations use Linux as the dom0.  For many years we
maintained a fork of Linux with intrusive changes to the memory
management system, interrupt handling and hardware access.  As you
might have predicted this became unsustainable.  Nowadays (since
around Linux 3.0) all - or almost all - of the necessary code is in
Linux upstream.

The interface between the dom0 kernel and the hypervisor is much
broader than that provided to guest kernels.  The dom0 needs to be
able to support all the facilities for direct access to guests needed
by the management tools, and it needs to cooperate with Xen to manage
all the system's hardware.

Because of the way Linux is developed, and the fact that users would
constantly need new versions of dom0 for many reasons (running on new
hardware, fixing security bugs and so forth) trying to maintain our
own stable branch of it is totally impractical.

So we have a reasonably loose coupling where we expect a version of
Xen and its tools to work with a range of dom0 kernels.

On the other hand life is at least a little bit easier because we
don't try to be bug compatible: we expect that a user who is running a
Xen system is willing to run a kernel which contains Xen-related
bugfixes, even for bugs exposed by a new version of Xen well after
that kernel was released upstream.  Of course in practice we don't
expect users to roll these kernels themselves: we aim to push the
necessary changes into upstream Linux stable branches, and out to
distros.

This tolerably loose coupling of the dom0 kernel, and upstreaming it
all into mainstream Linux, means that nowadays you can use the same
kernel for your Xen dom0 as will boot on baremetal.  Indeed in Debian
nowadays there are no longer any special Xen kernels - the standard
686-bigmem and amd64 kernels are good to use as a Xen dom0 (or indeed
as a guest).


[ qemu ]

Another important component of a traditional Xen system for which
we're not upstream is qemu.

When we run unmodified guest operating systems, we need to provide
them with something that looks like a PC.  In particular we need to
provide something that looks like a screen, keyboard and IDE and
network controller, and an emulation of the underlying bus hardware
(PCI and so on).

For a long time the qemu project has been trying to provide this same
thing in a different context, so it is an obvious choice for reuse.

However, when Xen first wanted something to do this job, the qemu
project was difficult to contribute to and the codebase itself made a
large number of inappropriate assumptions and impositions about the
system architecture.  So the Xen developers took a copy of the qemu
source code into the Xen tree and hacked it until it did what they
wanted.

Over the subsequent years that version of qemu, now known in Xen
circles as "qemu-xen-traditional", gained important performance
improvements and bugfixes.  Security fixes were made directly to the
Xen version since qemu upstream didn't have a security response
process.  Xen merged from upstream qemu a couple of times and attempts
were made to try to upstream some or all of the Xen changes but a
combination of inertia (and unwillingness to expand the project's
goals) on qemu's part, poor communications on both sides, and lack of
effort on the Xen side, meant that came to very little.

However, as with Linux, things have changed.  The qemu project has
been utterly revolutionised over the past few years and is now very
fast-moving and dynamic.  We have also been aggressvely upstreaming
the changes necessary for Xen.

So "qemu-xen-traditional" is now in the deep freeze
maintainence-wise.  But why does it remain at all ?  Well one reason
is that the new qemu-xen based on upstream is not quite ready; the Xen
4.2 release will still use qemu-xen-traditional by default.  But we
expect the new upstream-based qemu-xen to be the default in Xen 4.3.

But a bigger reason is tht we promised that old guests would continue
to run.

The emulated PC provided by modern qemu-xen is fairly different to
that provided by qemu-xen-traditional.  The device IDs of many devices
are different.  The emulated "chipset" has been upgraded.  The new
upstream qemu cannot read the savefiles generated by
qemu-xen-traditional so guests cannot be saved/restored or migrated to
the new one.

Worse, many guest operating systems react poorly to a complete
"motherboard and chassis swap", which is how an upgrade to new qemu
would look.  Even if they don't mind that in theory, in practice the
new setup might expose rafts of new guest bugs which would be very
difficult to work around, especially given that we are now much more
closely coupled to the upstream qemu codebase.

And while in principle it might be theoretically possible for
upstream-based qemu to emulate the old one, with sufficient #ifdefs
and run-time flags, the resulting code would be very ugly and
intrusive and certainly not suitable for upstreaming.  It wouldn't be
suitable for long-term maintenance.

So we (Xen.org upstream) intend to maintain qemu-xen-traditional
indefinitely.  That means there are two versions of qemu in the Xen
system.

And there is worse: because of the speed of development of the qemu
project and the lack of Xen-focused committers, getting important
features and changes into upstream can be rather slow.  Support for
PCI passthrough, an important Xen feature available for many years in
qemu-xen-traditional, took 6-8 months to get into qemu upstream.

Therefore we as the Xen upstream provide a version of "upstream" qemu
which is actually an upstream qemu with Xen-related changes.  Now we
have a policy of not committing anything to qemu-xen which we think
wouldn't be suitable for upstream.  So in theory "qemu-xen" as we call
it would be entirely suitable for use as a replacement for qemu.  But
in practice we don't expect people to want to use our slightly funny
branch.  For one thing we don't do any non-Xen-related testing on it,
and anyway it might be based on a different version of qemu than the
one you would use for other purposes.

So the net result is that to get full Xen functionality a distro like
Debian needs to include /two/ additional copies of qemu, besides the
normal qemu (and perhaps yet a different version for kvm as well) -
and one of the versions used by Xen is ancient.

This is of course not the kind of thing that makes the Debian Security
Team happy.  Of course it's by far from the only occurrence of this
kind of thing in the Debian archive.  At least in this case Debian can
work with an upstream who still support all the versions that Debian
needs to ship: we as upstream do security and stability patches, and
compatibility fixes, even for the solidly frozen qemu-xen-traditional.

So it's good to see that there are some efforts being made to help
Debian manage code duplication and forking more effectively, for
example by searching the archive for similar code to ferret out copies
we didn't know about.

But ultimately at least in this case the real work of doing support,
particularly security support, for multiple versions is necessary:
there is no other way to avoid breaking those old, perhaps hard to
modify, guest operating systems.


So here we can see one of the key ways that we deal with the problem
of interface incompatibility: provide both versions of the software
and use whichever one is appropriate somehow.

Of course this is in principle the same approach as is done in a more
formal, systematic, way with any C library with a reasonable stable
API.  We're all familiar with Debian's well-established and mature
approach to this, shipping perhaps two or three versions in any one
release.

I think Debian could benefit from better tools not just for spotting
embedded copies and packages which are forks, but also for
cross-porting changes, tracking which embedded copies came from which
version of the upstream, which have had which essential security
fixes, and so on.


There is one other difficulty which shows up in this diagram:

The Xen project upstream doesn't promise that you can run one version
of the hypervisor with a different version of the tools - that is, the
userspace support and management software will work only with the
corresponding hypervisor.  We don't break compatibility within a
stable branch of a Xen release, but in general there's no
compatibility from one release to the next.

The reason for this is that the management interface to the hypervisor
is complicated and closely dependent on the features available in a
particular Xen release.  This means both that we want to be able to add
new interfaces (such as new management hypercalls), and we want to be
able to discard old ones.

This leads to a similar problem to that with Linux and udev; when
upgrading, you want to upgrade the tools on disk in the filesystem and
the hypervisor in /boot.  If you do this in the naive way then there
will be a time when the two versions on disk don't match and the
system won't be able to work if it is rebooted.  (Luckily this doesn't
make the system completely unbootable and thus hard to fix, but it
might well be unable to run any guests.)  And, it makes it difficult
to try out the new version: if it doesn't work, you'll have to fish
out the old packages to reinstall them.

So the Debian maintainers have arranged to allow coinstallation of
different versions of the Xen tools.  The paths from `make install'
are all edited to contain the version number, and wrappers for
entrypoints are provided which spot which version of the hypervisor
was booted and run the right set of binaries.

This seems quite restrictive a rule, and some would say we should be
more relaxed.  But even if we spent more effort on this area, it's not
clear how much it would help.  The most obvious improved compatibility
guarantee would be to permit using the tools from Xen version N with
the hypervisor from version N+1.  But that wouldn't do much good:

Firstly, a distro like Debian which releases less often than Xen will
still have the problem because the versions of Xen in two subsequent
Debian releases would often differ by more than one Xen release.

And even for a distro which releases more often than Xen, or if we
offered a longer compatibility period, there would still be
complicated upgrade path: you would have to make sure to install and
boot the new hypervisor first, before upgrading the tools.

So in practice Debian will probably anyway have to have the capability
to coinstall different tools versions and use the appropriate one.


There is one other big transition which doesn't show up so clearly in
this diagram.  The old "xend" daemon which formed the core of many
toolstacks is being phased out.

Mostly this is because it is a terrible mess internally, to the point
where not only does no-one understand it, but it's probably not
possible to understand while remaining sane.  Also it implements in a
non-reusable way a good deal of functionality also wanted by other
toolstacks.  One particular such toolstack of is Xapi, which is a
sophisticated API which forms the core of what is known as XCP (or the
Xen Cloud Platform), the Free Software release of Citrix/XenSource's
XenServer product.  Xapi is used by orchestration layers such as
OpenStack and CloudStack and recently went into wheezy as part of what
my colleagues call "project Kronos".  Another intended user of libxl
is libvirt.

So we are replacing xend with a C library known as "libxenlight" or
"libxl".  The command-line tool used by many Xen users to manage
domains, "xm", is being replaced by a mostly-compatible rewrite
called "xl".

This project is coming on nicely in the upstream tree.  We had a
preview of it in Xen 4.1 and in Xen 4.2 libxl will have an API which
we intend to support in theh future (in a backwards-compatibile way).
ABI forward compatibility is still beyond us, unfortunately.

Users of the Debian "xen-tools" package and its utilities such as
"xen-create-image" should find that the generated guest config files
etc. are all compatible with the new "xl" utility.  And you can use
"xl" pretty much like "xm".  So your ad-hoc higher-level machinery
probably doesn't need too much adjustment.

So we've spent considerable effort trying to figure out what xend does
and reproducing its behaviour so far as it's comprehensible and
reasonable.  The results aren't perfect from a compatibility point of
view, but we hope in any particular system the changes needed to use
xl rather than xm will be minor.

(An exception to this is the xend "managed domains" facility.  This is
where you say "xm new" rather than "xm create".  We think few Debian
users are using this; and this functionality is better provided in
other ways.  Either you can use the simple init script system, or if
that's not sufficient we would recommend using more sophisticated
management layers such as Xapi or Openstack.)

If you are a xend/xm user we would encourage you to take a look at the
new xl toolstack, a preview of which is available in the Xen 4.1 which
is in wheezy.  And if you use xend directly rather than via xm, or you
use xm in some way which you think might be unusual, you should check
that libxl and xl can meet your needs.  xend is pretty much dead right
now as far as development goes, and we expect to remove it in Xen 4.3
or 4.4 - probably, it won't be in wheezy+1.  So if there are important
features missing from libxl and xl we need to know about it.

libvirt is a more complicated problem.  libvirt isn't very big in
Debian - it's more of a Redhat thing - but I will mention it here
briefly.  I'm not much of an expert on it either.  If anyone here is
please do speak up.  Anyway, our intent is that the Xen libvirt driver
will be linked against libxl.  So the libvirt daemon itself will be
linked, transitively, against a particular Xen version.  It may be
necessary to provide multiple versions here too.

To try to make all of this somewhat less painful we have been working
hard to try to ensure that we can promise API stability in libxl
starting from the upcoming Xen 4.2 release - and to provide feature
parity with xend, and command line compatibility with xm.


So I hope I've given a flavour of some of the difficulties we face
as the upstream for a mature and sophisticated system - and the
different approaches we're taking for the various different interfaces
and layers.

If anyone has any suggestions for how either Debian or upstreamd
could do things better or make our and other people's lives easier, do
speak up!

Thanks.  Questions, comments ?