publication

[talk-2015-debconf-dgit.git] / talk.txt

==== overview slide


Hi.  I'm here to plug dgit, which is a system for treating the Debian
archive like a git remote.

I'm going to try to keep my talk fairly short to leave time for
questions.  Please save your questions for the end.


Firstly, a bit of framing.

When we work on Debian we take on different roles.  The biggest
difference is between the maintainer (or maintainers) of a package,
working on their own package, and everyone else.

I'm going to start by presenting dgit from the point of view of
everyone else: NMUers, sponsorship, bug squashers, downstream distros
and users, teams doing cross-archive work like transitions and
reproducible builds, and so on.  Maintainers, please be patient - I'll
get to you a bit later in the talk.


==== manpage slide

The point of dgit is that it lets everyone treat the archive as if it
were a git repository.

You can dgit clone any package, in any suite (so, for example, sid or
experimental) and you will get a git tree which is exactly the same as
dpkg-source -x.

So dgit always works the same, from the non-maintainer's point of
view, on any package: the operation is completely uniform.  You don't
need to know anything about the maintainer's verson control workflow
tools or source format preferences.

You can then work on the package in git, the way you would work in git
with any other project.  In particular, you can:

 * commit locally
 * cherry pick changes from other branches
 * git reset, git clean
 * git rebase -i to polish a more complex set of changes into
   a patch queue
 * all the other usual gitish stuff

If you have the right authority you can also dgit push, to upload.
That is, a DD can dgit push any package; a DM can dgit push the
packages that the archive thinks they can upload.

Before you push you still have to do a build.  dgit does not replace
existing ways of building source and binary packages, although it does
provide some helpful build rune wrappers (more about that later).


If you don't want to, or can't, upload to Debian, but do want so share
your work with other people, you can git push your dgit
branch anywhere suitable, so other can fetch it.  So, for example, you
could share your branch with your sponsor, who could then approve it
by running dgit push.  dgit's branches are ordinary git branches for
this purpose.

A downstream such as a derivative or partial derivative of Debian can
use the dgit branches directly as the upstreams for a git-based setup,
and work entirley without source packages.

==== data flow slide

Behind the scenes, dgit works by providing a set of git repositories
in parallel to the existing archive.

Every dgit push is actually two underlying operations: the first is a
git tag and push, to the dgit git server.  The second is a
conventional package upload - with a wrinkle: the .dsc of an upload
done with dgit contains an extra field with the git commit hash that
was pushed.

Likewise, fetch and clone combine information from the archive and the
git history.  If the most recent upload was done with dgit, the commit
hash in the dsc enables dgit fetch to find the right commit and
present it to you.  If the most recent upload was not done with dgit,
dgit imports the source package into git - and stitches it into the
existing dgit history, if there is one.

You do need to treat dgit git branches a bit specially is if you need
to build source pacakges (for upload, for example).  In this case dgit
needs the .orig tarballs for your source package.  If you are not
doing a new upstream version, dgit fetch will get the relevant .origs
for you.  If you /are/ doing a new upstream version, then presumably
you have obtained the origs as part of preparing your package, or you
can build them easily.

I should mention that currently only dgit push updates the history on
the dgit git server.  So until a package is pushed with dgit push for
the first time, doesn't exist on that server; and even after then,
non-dgit uploads are not recorded, so the history can be out of date.
I'm hoping to improve this in the future.


==== NMU linear history slide

As a general rule, the dgit history structure should be up to the
maintainer - at least, if they care.

If you are doing a straightforward NMU you should produce a
well-structured linear sequence of commits, as you would for any other
git upstream, based on the dgit suite branch.  Not only does this mean
that if the maintainer is using dgit, they can hopefully easily
include your changes; it also means that if they _aren't_ using dgit,
at least you have published a history which is suitable for rebasing
onto theirs, or whatever it is they might want to do with it.

If the source package is `3.0 (quilt)', you shouldn't touch
debian/patches; dgit will take care of that for you.  This is the
other reason why you should provide a tidy linear patch series: if the
maintainer likes quilt and is not using dgit, your changes will be
automatically presented to them in a fairly plausible format - a
series of extra patches - like they should expect from any other NMU.

An ordinary NMUer should not normally update a patch in the quilt
stack directly.  Ie, an NMUer shouldn't squash their changes into an
existing patch.  This is because while it's easy for the maintainer to
squash it themselves, if they want, it's a little harder for the
maintainer to disentangle a squashed patch.  Working directly on
patches can also result in people having to read interdiffs, which are
notoriously confusing.


So, in summary, if you are doing cross-archive work, or bug squashing,
you can start using dgit right away and I hope it will make your life
a lot easier.


==== NMU linear history on top of basic dgit history

Sadly, unless the maintainer uses dgit, the history you see in dgit
will not be the maintainer's history.

This is because maintainers' git branches often differ from the source
packages in the archive.

If you dgit clone a package and it has an X-Vcs-Git header, dgit will
set up a remote for that, so you can fetch the maintainer's history and use it if you
like.  So in that sense dgit clone encompasses debcheckout.

But, in the general case, the X-Vcs-Git tree may not be immediately
useable to someone not familiar with the package, so dgit doesn't use
it for the dgit suite branch.

For example, the maintainer's repo might contain only a debian/
directory, or be a quilty tree without patches applied.  And the tag
and suite naming conventions can vary too.  So while the maintainer's
history can be useful if you want to do archeaology, it's not in
general suitable for use by dgit.

There is also the problem that the maintainer's nominated git server
might be anywhere, so it might be down, or gone away, or compromised.


So, if the maintainer is not using dgit, dgit has to synthesise a git
history.  The history you see in dgit will then have a very basic
branch and commit structure, rather than representing the package's
actual history, something like you see here.


Which brings me onto the other side of this talk: dgit for
maintainers:

==== history comparison slide

For the reasons I've explained, downstream dgit users would like you
as a maintainer to use dgit push to do your uploads.  They will then
be able to see, and directly work with, your own history.

More generally, the point of using a dvcs like git is to publish your
work.  The existing ways of publishing git histories for Debian
packagess aren't uniformly useable for users: they require the user to
understand the maintainer's git working practices.

What dgit does is provide a way for you to publish a history which
users can rely on actually corresponding to the archive, and use
immediately without special knowledge.

But it's in your own selfish interest to upload with dgit, too:

If you use dgit, you will be able to directly merge NMUs, branches
proposed via pull-request emails, and other similar contributions:
Because, in this case, the dgit-using contributor will have based
their work on your own history.  Whereas, if you don't use dgit,
dgit-using contributors will be working on a stub history, and may
dgit push commits based on that stub.  You can dgit fetch that work
even if you're not using dgit for your own uploads, but if you're not
using dgit push yourself, at the very least you'll have to rebase the
NMUers' and other contributors' work onto your own history.

Another advantage of using dgit for your maintainer uploads is that it
will put your own history on browse.dgit.debian.org, rather than
advertising dgit's stub history (which can also be out of date).

If you use dgit push, you get an extra check that the source package
you are uploading is exactly the same as your git HEAD.  This can
save you some dsc-based checks.

And, of course, as I say, doing your uploads with dgit will improve
downstream dgit users' lives.


==== data flow slide

dgit is not a replacement for existing git packaging tools; it's
intended to complement them.  So (unlike git-dpm) dgit does not define
a git history structure.

Nor, unlike git-buildpackage, does dgit define or suggest a branch
structure for naming upstream or downstream branches, pristine tar
branches, etc.

dgit doesn't require a particular source format; it couldn't, since it
needs to work with any package.


==== data flow slide with EQUAL and FF

dgit push imposes only two requirements on your git trees, which stem
directly from dgit's objectives.

The most important requirement is that your git tree is identical to
the unpacked source package.  This is the whole point of dgit, because
it enables dgit to always present to the user a working tree which is
the same as they would get from apt-get source.


=== Status table slide

For all native and source format 1.0 packages (which will include
people using raw git to manage their delta from upstream), a git tree
identical to the source package is already the interchange git branch
format.  These maintainers can start using dgit right away for all
their maintainer uploads.  Please do!

The other requirement of dgit is simply that the dgit branches are
fast-forwarding.  So if your tools have made a rebasing branch, you
may need to make a fake merge (with git merge -s ours) before pushing.
I'm intending to provide some rather more cooked way to do this but I
haven't decided the exact shape yet.


For `3.0 (quilt)' packages, things are more complicated.  The purpose
of dgit is to provide a git view which is identical to the source
package, and can be worked on without knowledge of the maintainer's
chosen source format, patch system or git workflow.

This means that the dgit git tree for a `3.0 (quilt)' package is what
is sometimes called a `patches-applied packaging branch without .pc
directory'.

This means that all the changes made in Debian are represented in the
main source tree, as well as being contained within a patch in
debian/patches.

But: I think the most popular tool for working with `3.0 (quilt)' is
git-buildpackage, and git-buildpackage likes to work with
patches-unapplied tress.  I've spoken to Guido Guenther (the
git-buildpackage maintainer) about this.  We had a really good
conversation.  We have a plan for interoperating, which involves dgit
synthesising an extra git commit, to apply the patches, and then
pushing that to the dgit git server.  (The extra git commit wouldn't
appear on your own branch; if it did, it would cause trouble.)


There is another problem which affects both git-buildpackage and
git-dpm: namely, .gitignore files.  Neither git-buildpackage nor
git-dpm represent the .gitignore, which typically occurs in the git
tree, as a patch in debian/patches.  Rather, these tools tell
dpkg-source to ignore it when buildinng the source package.  So the
source package does not contain .gitignore, although the git tree
does.

I have spoken to Bernhard Link (the git-dpm maintainer) about this and
he wasn't keen on having git-dpm treat .gitignore normally, and record
.gitignores in a patch in the git-dpm branch.  So unless we can
persuade Bernhard to offer this, at least as an option, dgit is
probably going to have to synthesise a commit to add them, in much the
same way as it will add a commit to convert a git-buildpackage
patches-unapplied branch to a directly-editable patches-applied one.



==== data flow slide

There are a few other things I ought to cover, since they often come
up.  They're are relevant to maintainers and non-maintainers:


Firstly, some wrinkles.

The first wrinkle is that DMs currently need to email me a signed copy
of their ssh key, in order to be able to push.  This is because the
dgit repo server uses ssh as a transport and the project doesn't,
right now, have a record of DMs' ssh keys.


The second thing that's less than ideal is that the dgit git history
does not generally include the package upload history.
git-import-dscs can produce a git branch more or less representing the
upload history, but dgit doesn't run that itself.  It would be
difficult for dgit to do so because deciding which set of versions to
include is nontrivial and of course it would involve an awful lot of
downloading.

One could push such a branch to the archive with dgit push.  But, it
seems to me that the git history structure ought to up to the
maintainer, and if the maintainer chooses to use dgit, the
maintainers's existing git history is probably better.

So I think the real way to improve this is to persuade more
maintainers to use dgit.  Perhaps for maintainers who do not, we
should at some point consider providing centrally an archive-based
package history.


==== data flow slide with EQUAL and FF

But the most obvious challenge for a maintainer with an existing git
branch, but trying to use dgit, is dgit's insistence that the source
package and git tree are the same.

Sadly, quite a few source packages contain files not in the
maintainers' git branches, but which are needed to build: most
commonly, autotools output.  Such git branches are not useable with
dgit.

But nowadays most people recommend that the package build should
always rerun autotools.  If you do that, then neither your git tree
nor your source package need contain the autotools output and all is
well.

Alternatively, you can commit the autotools output to git.  Merge
conflicts, which do occur occasionally if you do this, are easily
resolved by rerunning autotools.


And a second way dgit's rule can bite is .gitignore.  Most existing
packaging tools remove .gitignore from source packages by default.
But, dgit requires that the source package and git tree are the same,
so if your git tree has .gitignore in it, your source package should
too.

==== manpage slide

So it is normally best to use one of dgit's build operations to build
for upload; in generally, all that those build operations do
differently to the underlying tool, is pass some appropriate -I
options, to exclude exactly and only the .git directory.

With `1.0' this is sufficient.  With `3.0 (quilt)', there is more work
to be done in the tools, as I mentioned earlier.


==== manpage clean slide

Finally, there is one compelling advantage of dgit's git-based
approach.

Many packages have strangely-behaved or plain buggy clean targets.
Because dgit knows that your git tree is canonical, it can help work
around this: you can tell dgit to use git-clean instead, avoiding the
package's clean target entirely.

If you're not in the habit of forgetting to say git-add, you can set a
configuration option to have dgit always use git-clean.  Then you will
never have to fight a buggy clean target, in a strange package, ever
again.



==== Future plans slide

I have a number of plans for the future, some of which I may even get
around to:


I need to implement the automatic adjustment of git-buildpackage and
git-dpm's git trees into the dgit branch format, and perhaps provide
some additional helper tools.


dgit can't currently upload to DELAYED queues, which makes it a bit
awkward for a fire-and-forget NMU.  One way to implement this would be
to push to the suite branch but send the package upload to DELAYED.
But such an upload which was deleted by the maintainer would still
present be in the suite branch, and the changes might be resurrected
by a future dgit user.  At the very least there should be a way for
the maintainer to revert a push which was accompanied by a DELAYED
upload.

The other implementation would have the pushed branch stashed away
somewhere and "revealed" only when it was ready, if it was still a
fast forward.  That makes it tricker for someone else to find it to
work on top of it, and is more work to implement, but it's probably
better.


It would be easy for dgit, when pushing an NMU, to construct the
NMUdiff and email it to the BTS.  Of course a maintainer can fetch
your NMU as git commits easily with dgit fetch, but maybe the
maintainer isn't using dgit (or, even, isn't using git at all), so you
for the benefit of those maintainers we should still be sending these
diffs.


dgit push is slightly more work than it needs to be when doing a
source-only upload.  In this case there is not really any need for a
separate `dgit build-source' step.  I think there should be a way to
have dgit push make the source package as well, since it easily can.


The dgit git server has most of the moving parts to become a
fairly-general-purpose git server with access control equivalent to
the Debian archive.  In this role it could replace some uses of
Alioth, which is good because Alioth is doing too many other things.

In particular, with the use of signed pushes, we could have
traceability of pushes, which makes a gitish packaging workflow more
practical - in particular, maintainers would not need to as thoroughly
audit work found on for-upload branches on the git server, before
pushing the code to the archive.  (I suspect many maintainers already
don't do this as thoroughly as they should.)


dgit requires the suite branches to be fast forwarding.  If you have a
workflow involving git rebase, you need to make a fake merge (ie, a
merge made with git merge -s ours), at the tip of your branch, to make
it be a fast forward from the previous state of the suite branch.  And
then when you want to work on the package again, you need to strip
that same fake merge.

This can be done with raw git operations (git merge and git reset)
quite easily, but it's ugly.  Also doing it by hand like this doesn't
provide many safety catches.

I intend to provide some tooling which will help with this, but I
haven't exactly decided what the UI should be and how it should work.


The final thing I can do myself is arrange for non-dgit uploads to be
automatically converted, and appended to the dgit suite branches on
the dgit git server.  That would make the dgit history, as shown on
browse.dgit.debian.org for example, for any package be more useful,
starting with the first dgit-based upload.  In particular it would
avoid the browse view ever being out of date.


There are also some things that I need help with.

The documentation for how dgit works together with existing git
management tools is rather sparse.

Ideally each git workflow tool would explain how (or whether) it works
with dgit.  Tasks that should be covered include how to incorporate
changes made in an NMU into the tool's favoured history format, and
how to upgrade a package to a new upstream version.  I have spoken to
the git-buildpackage and git-dpm maintainers about this.  I'm
optimistic particularly about git-buildpackage.


There is a difficulty with empty directories in source packages.  git
refuses to handle empty directories - they tend to just vanish from
commits and checkouts.  But I'm told there are some source packages
whose orig tarballs contain empty directories, and build systems
depend on those empty directories.

Currently if you dgit clone such a package, it won't build.  I don't
have a particularly good answer to this problem right now.  It's no
good to have dgit create these directories in the working tree, when
it notices, because the git branches might be transported by plain git
and used elsewhere.

This can be worked around in the source package by either adding
something to debian/rules to create these directories, or by adding
some files to the directories to make them nonempty.  But at the
moment a maintainer is not required to do either of these things.


And then there are a couple of exciting possibilities for new uses:

Firstly, it would be very nice to have a way for a sponsee to provide
a dgitish git branch, which the sponsor could work with directly and
ultimately dgit push.  I'm really need to talk about this with some
people more familiar than me with the sponsorship queue
infrastructure.



Secondly, dgit is not just for Debian.  Other distros currently using
only source packages can use it too.  Read-only support doesn't
necessarily depend on specific support at the distro end.  So, you can
already dgit fetch from ubuntu; I'd welcome patches for read-only
support for Mint, for example.

dgit push support requires a suitably set up git server, and maybe
some thought, depending on the distro's views on push access control.
I hear rumours that Ubuntu might grow a dgit git server and support
dgit push.  Patches very welcome!


==== Thanks and more info slide

Finally, there are some people I need to thank.

At least half of the design of dgit is Joey Hess's, at the now-famous
session in Vaumarcus.

Debian System Administration, and particularly Peter Palfrader, have
been absolutely brilliant.  The level of service and competence has
been outstanding.  Ever time I noticed some aspect I had forgotten,
and made a blatherous and ill-thought-out request, DSA would propose a
much better alternative and implement their end of it it almost
immediately.

Also, I'd like to thank the ftpmasters for setting up the new
ftpmaster data service.  This has enabled me to extend dgit's
useability to non-DDs.


So that's most of what I have prepared.  There's, of course, a lot
more detailed information in the manpages.

I'm going to take questions now.