Hi. I'm here to plug dgit, which is a system for treating the Debian
archive like a git remote.
-I'm going to talk for about 35 minutess and then I'll take questions.
+I'm going to talk for about 25 minutess and then I'll take questions.
+Firstly, a bit of framing.
+
When we work on Debian we take on different roles. The biggest
difference is between the maintainer (or maintainers) of a package,
working on their own package, and everyone else.
I'm going to start by presenting dgit from the point of view of
-everyone else: NMUers, sponsorship, bug squashers, downstreams,
-users, teams doing cross-archive work
-like transitions and reproducible builds, and so on. Maintainers,
-please be patient - I'll get to you later
+everyone else: NMUers, sponsorship, bug squashers, downstream distros
+and users, teams doing cross-archive work like transitions and
+reproducible builds, and so on. Maintainers, please be patient - I'll
+get to you a bit later in the talk.
==== manpage slide
If you don't want to, or can't, upload to Debian, but do want so share
your work with other people, you can git push your dgit
-branch anywhere suitable, so they can fetch it. So, for example, you
+branch anywhere suitable, so other can fetch it. So, for example, you
could share your branch with your sponsor, who could then approve it
by running dgit push. dgit's branches are ordinary git branches for
this purpose.
A downstream such as a derivative or partial derivative of Debian can
-use the dgit branches directly as the upstream for a git-based setup,
+use the dgit branches directly as the upstreams for a git-based setup,
and work entirley without source packages.
==== data flow slide
needs the .orig tarballs for your source package. If you are not
doing a new upstream version, dgit fetch will get the relevant .origs
for you. If you /are/ doing a new upstream version, then presumably
-you have obtained them as part of preparing your package, or you can
-build them easily.
+you have obtained the origs as part of preparing your package, or you
+can build them easily.
==== NMU linear history slide
If you are doing a straightforward NMU you should produce a
well-structured linear sequence of commits, as you would for any other
-git upstream. Not only does this mean that if the maintainer is using
-dgit, they can hopefully easily include your changes; it also means
-that if they _aren't_ using dgit, at least you have published a
-history which is suitable for rebasing onto theirs, or whatever.
+git upstream, based on the dgit suite branch. Not only does this mean
+that if the maintainer is using dgit, they can hopefully easily
+include your changes; it also means that if they _aren't_ using dgit,
+at least you have published a history which is suitable for rebasing
+onto theirs, or whatever it is they might want to do with it.
If the source package is `3.0 (quilt)', you shouldn't touch
debian/patches; dgit will take care of that for you. This is the
other reason why you should provide a tidy linear patch series: if the
maintainer likes quilt and is not using dgit, your changes will be
-automatically presented to them in a fairly plausible format like they
-should expect from any other NMU.
+automatically presented to them in a fairly plausible format - a
+series of extra patches - like they should expect from any other NMU.
An ordinary NMUer should not normally update a patch in the quilt
stack directly. Ie, an NMUer shouldn't squash their changes into an
existing patch. This is because while it's easy for the maintainer to
squash it themselves, if they want, it's a little harder for the
-maintainer to disentangle a squashed patch. This can also result in
-people having to read interdiffs, which are notoriously confusing.
+maintainer to disentangle a squashed patch. Working directly on
+patches can also result in people having to read interdiffs, which are
+notoriously confusing.
+
+
+So, in summary, if you are doing cross-archive work, or bug squashing,
+you can start using dgit right away and I hope it will make your life
+a lot easier.
==== NMU linear history on top of basic dgit history
packages in the archive.
If you dgit clone a package and it has an X-Vcs-Git header, dgit will
-set up a remote for it, so you can fetch the history and use it if you
+set up a remote for that, so you can fetch the maintainer's history and use it if you
like. So in that sense dgit clone encompasses debcheckout.
But, in the general case, the X-Vcs-Git tree may not be immediately
-useable to someone not familiar with the package.
+useable to someone not familiar with the package, so dgit doesn't use
+it for the dgit suite branch.
-The maintainer's repo might contain only a debian/ directory, or be a
-quilty tree without patches applied. And the tag and suite naming
-conventions can vary too. So while the maintainer's history can be
-useful if you want to do archeaology, it's not in general suitable for
-use by dgit.
+For example, the maintainer's repo might contain only a debian/
+directory, or be a quilty tree without patches applied. And the tag
+and suite naming conventions can vary too. So while the maintainer's
+history can be useful if you want to do archeaology, it's not in
+general suitable for use by dgit.
There is also the problem that the maintainer's nominated git server
might be anywhere, so it might be down, or gone away, or compromised.
So, if the maintainer is not using dgit, dgit has to synthesise a git
history. The history you see in dgit will then have a very basic
branch and commit structure, rather than representing the package's
-actual history.
+actual history, something like you see here.
Which brings me onto the other side of this talk: dgit for
as a maintainer to use dgit push to do your uploads. They will then
be able to see, and directly work with, your own history.
-In general, the point of using a dvcs like git is to publish your
+More generally, the point of using a dvcs like git is to publish your
work. The existing ways of publishing git histories for Debian
packagess aren't uniformly useable for users: they require the user to
understand the maintainer's git working practices.
But it's in your own selfish interest to upload with dgit, too:
-If you use dgit, you will be able to directly merge NMUs, patches
-proposed via pull-request emails, and so on: Because, in this case,
-the dgit-using contributor will have based their work on your own
-history. Whereas, if you don't, dgit-using contributors will be
-working on a stub history, and may dgit push commits based on that
-stub. You can dgit fetch it even if you're not using dgit for your
-uploads, but when you do at the very least you'll have to rebase the
-NMUer's work.
+If you use dgit, you will be able to directly merge NMUs, branches
+proposed via pull-request emails, and other similar contributions:
+Because, in this case, the dgit-using contributor will have based
+their work on your own history. Whereas, if you don't use dgit,
+dgit-using contributors will be working on a stub history, and may
+dgit push commits based on that stub. You can dgit fetch that work
+even if you're not using dgit for your own uploads, but if you're not
+using dgit push yourself, at the very least you'll have to rebase the
+NMUers' and other contributors' work onto your own history.
Another advantage of using dgit for your maintainer uploads is that it
will put your own history on browse.dgit.debian.org, rather than
intended to complement them. So (unlike git-dpm) dgit does not define
a git history structure.
-Nor does dgit define a branch structure distinguishing upstream or
-downstream branches, pristine tar branches, etc.
+Nor, unlike git-buildpackage, does dgit define or suggest a branch
+structure for naming upstream or downstream branches, pristine tar
+branches, etc.
dgit doesn't require a particular source format; it couldn't, since it
needs to work with any package.
directly from dgit's objectives.
The most important requirement is that your git tree is identical to
-the unpacked source package. (Technically, in the case of a `3.0
-(quilt)' package, it is what is sometimes called a `patches-applied
-packaging branch without .pc directory', which means that the upstream
-source files in the main package tree correspond to the actual source
-code that will be used when the package is built, rather than to the
-upstream versions.)
-
-For all native packages, and for users of git-dpm and raw git, this is
-already the interchange format. These maintainers can start using
-dgit right away. Please do!
+the unpacked source package.
-For those using git-buildpackage with `3.0 (quilt)', things are a bit
-more complicated. I'm told that gbp pq can be used to generate a
-patches-applied branch, and that some users prefer to use that as the
-interchange git branch, but I know this is far from universal. I'm
-talking to the git-buildpackage maintainers about gbp integration, so
-watch this space.
+For all native and source format 1.0 packages (which will include
+people using raw git to manage their delta from upstream), this is
+already the interchange format. These maintainers can start using
+dgit right away for all their maintainer uploads. Please do!
The other requirement of dgit is simply that the dgit branches are
fast-forwarding. So if your tools have made a rebasing branch, you
haven't decided the exact shape yet.
+==== 3.0 quilt applied slide
+
+For `3.0 (quilt)' packages, things are more complicated. The purpose
+of dgit is to provide a git view which is identical to the source
+package, and can be worked on without knowledge of the maintainer's
+chosen source format, patch system or git workflow.
+
+This means that the dgit git tree for a `3.0 (quilt)' package is what
+is sometimes called a `patches-applied packaging branch without .pc
+directory'.
+
+This means that all the changes made in Debian are represented in the
+main source tree, as well as being contained within a patch in
+debian/patches.
+
+But: I think the most popular tool for working with `3.0 (quilt)' is
+git-buildpackage, and git-buildpackage likes to work with
+patches-unapplied tress. I've spoken to Guido Guenther (the
+git-buildpackage maintainer) about this. We had a really good
+conversation. We have a plan for interoperating, which involves dgit
+synthesising an extra git commit, to apply the patches, and then
+pushing that to the dgit git server. (The extra git commit wouldn't
+appear on your own branch; if it did, it would cause trouble.)
+
+=== xxxx status table
+
+There is another problem which affects both git-buildpackage and
+git-dpm: namely, .gitignore files. Neither git-buildpackage nor
+git-dpm represent the .gitignore, which typically occurs in the git
+tree, as a patch in debian/patches. Rather, these tools tell
+dpkg-source to ignore it when buildinng the source package. So the
+source package does not contain .gitignore, although the git tree
+does.
+
+I have spoken to Bernhard Link (the git-dpm maintainer) about this and
+he wasn't keen on having git-dpm treat .gitignore normally, and record
+.gitignores in a patch in the git-dpm branch. So unless we can
+persuade Bernhard to offer this, at least as an option, dgit is
+probably going to have to synthesise a commit to add them, in much the
+same way as it will add a commit to convert a git-buildpackage
+patches-unapplied branch to a directly-editable patches-applied one.
+
+
+
+xxx empty directories
+
+
+ I'm told that gbp pq can be used to generate a
+patches-applied branch, and that some users prefer to use that as the
+interchange git branch, but I know this is far from universal. I'm
+talking to the git-buildpackage maintainers about gbp integration, so
+watch this space.
+
+
+, and for users of git-dpm and
+
+(more or less)
+ xxx
+
+
+
==== data flow slide
There are a few other things I ought to cover, since they often come
The second thing that's less than ideal is that the dgit git history
does not generally include the package upload history.
git-import-dscs can produce a git branch representing the upload
+
+xxx patches-applied
+
history, but dgit doesn't run that itself. It would be difficult for
dgit to do so because deciding which set of versions to include is
nontrivial and of course it would involve an awful lot of downloading.
package history.
+==== dgit requirements slide
+
But the most obvious challenge for a maintainer with an existing git
branch, but trying to use dgit, is dgit's insistence that the source
package and git tree are the same.
-However, some source packages contain files not in the maintainers'
-git branches, and which are needed to build: most commonly, autotools
-output. Such git branches are not useable with dgit.
+Sadly, quite a few source packages contain files not in the
+maintainers' git branches, but which are needed to build: most
+commonly, autotools output. Such git branches are not useable with
+dgit.
But nowadays most people recommend that the package build should
always rerun autotools. If you do that, then neither your git tree
well.
Alternatively, you can commit the autotools output to git. Merge
-conflicts etc. are easily resolved by rerunning autotools.
+conflicts, which do occur occasionally if you do this, are easily
+resolved by rerunning autotools.
+
+
+And a second way dgit's rule can bite is .gitignore. Most existing
+packaging tools remove .gitignore from source packages by default.
+But, dgit dgit requires that the source package and git tree are the
+same, so if your git tree has .gitignore in it, your source package
+should too.
+
+==== manpage slide
+So it is normally best to use one of dgit's build operations to build
+for upload; in generally, all that those build operations do
+differently to the underlying tool, is pass some appropriate -I
+options, to exclude exactly and only the .git directory.
-And a second way this can bite is that it is normally best to use one
-of dgit's build operations to build for upload. This is mainly
-because most other tools remove .gitignore by default. dgit requires
-that the source package and git tree are the same, so if your git tree
-has .gitignore in it, your source package should too.
+==== manpage clean slide
Finally, there is one compelling advantage of dgit's git-based
approach.
If you're not in the habit of forgetting to say git-add, you can set a
configuration option to have dgit always use git-clean. Then you will
-never have to fight a buggy clean target, in a strange package, ever again.
+never have to fight a buggy clean target, in a strange package, ever
+again.
==== Future plans slide
-I have a number of plans for the future, some of which I need help
-with. But I don't have time, I'm afraid, to go through them.
+I have a number of plans for the future, some of which I may even get
+around to:
+
+
+dgit can't currently upload to DELAYED queues, which makes it a bit
+awkward for a fire-and-forget NMU. One way to implement this would be
+to push to the suite branch but send the package upload to DELAYED.
+But such an upload which was deleted by the maintainer would still
+present be in the suite branch, and the changes might be resurrected
+by a future dgit user. At the very least there should be a way for
+the maintainer to revert a push which was accompanied by a DELAYED
+upload.
+
+The other implementation would have the pushed branch stashed away
+somewhere and "revealed" only when it was ready, if it was still a
+fast forward. That makes it tricker for someone else to find it to
+work on top of it, and is more work to implement, but it's probably
+better.
+
+
+It would be easy for dgit, when pushing an NMU, to construct the
+NMUdiff and email it to the BTS. Of course a maintainer can fetch
+your NMU as git commits easily with dgit fetch, but maybe the
+maintainer isn't using dgit (or, even, isn't using git at all), so you
+for the benefit of those maintainers we should still be sending these
+diffs.
+
+
+dgit push is slightly more work than it needs to be when doing a
+source-only upload. In this case there is not really any need for a
+separate `dgit build-source' step. I think there should be a way to
+have dgit push make the source package as well, since it easily can.
+
+
+The dgit git server has most of the moving parts to become a
+fairly-general-purpose git server with access control equivalent to
+the Debian archive. In this role it could replace some uses of
+Alioth, which is good because Alioth is doing too many other things.
+
+In particular, with the use of signed pushes, we could have
+traceability of pushes, which makes a gitish packaging workflow more
+practical - in particular, maintainers would not need to as thoroughly
+audit work found on for-upload branches on the git server, before
+pushing the code to the archive. (I suspect many maintainers already
+don't do this as thoroughly as they should.)
+
+
+dgit requires the suite branches to be fast forwarding. If you have a
+workflow involving git rebase, you need to make a fake merge (ie, a
+merge made with git merge -s ours), at the tip of your branch, to make
+it be a fast forward from the previous state of the suite branch. And
+then when you want to work on the package again, you need to strip
+that same fake merge.
+
+This can be done with raw git operations (git merge and git reset)
+quite easily, but it's ugly. Also doing it by hand like this doesn't
+provide many safety catches.
+
+I intend to provide some tooling which will help with this, but I
+haven't exactly decided what the UI should be and how it should work.
+
+
+The final thing I can do myself is arrange for non-dgit uploads to be
+automatically converted, and appended to the dgit suite branches on
+the dgit git server. That would make the dgit history, as shown on
+browse.dgit.debian.org for example, for any package be more useful,
+starting with the first dgit-based upload. In particular it would
+avoid the browse view ever being out of date.
+
+
+There are also some things that I need help with.
+
+The documentation for how dgit works together with existing git
+management tools is rather sparse.
+
+Ideally each git workflow tool would explain how (or whether) it works
+with dgit. Tasks that should be covered include how to incorporate
+changes made in an NMU into the tool's favoured history format, and
+how to upgrade a package to a new upstream version. I have spoken to
+the git-buildpackage and git-dpm maintainers about this. I'm
+optimistic particularly about git-buildpackage.
+
+
+There is a difficulty with empty directories in source packages. git
+refuses to handle empty directories - they tend to just vanish from
+commits and checkouts. But I'm told there are some source packages
+whose orig tarballs contain empty directories, and build systems
+depend on those empty directories.
+
+Currently if you dgit clone such a package, it won't build. I don't
+have a particularly good answer to this problem right now. It's no
+good to have dgit create these directories in the working tree, when
+it notices, because the git branches might be transported by plain git
+and used elsewhere.
+
+This can be worked around in the source package by either adding
+something to debian/rules to create these directories, or by adding
+some files to the directories to make them nonempty. But at the
+moment a maintainer is not required to do either of these things.
+
+
+And then there are a couple of exciting possibilities for new uses:
+
+Firstly, it would be very nice to have a way for a sponsee to provide
+a dgitish git branch, which the sponsor could work with directly and
+ultimately dgit push. I'm really need to talk about this with some
+people more familiar than me with the sponsorship queue
+infrastructure.
+
+
+
+Secondly, dgit is not just for Debian. Other distros currently using
+only source packages can use it too. Read-only support doesn't
+necessarily depend on specific support at the distro end. So, you can
+already dgit fetch from ubuntu; I'd welcome patches for read-only
+support for Mint, for example.
+
+dgit push support requires a suitably set up git server, and maybe
+some thought, depending on the distro's views on push access control.
+I hear rumours that Ubuntu might grow a dgit git server and support
+dgit push. Patches very welcome!
+
+
+==== Thanks and more info slide
+
+Finally, there are some people I need to thank.
+
+At least half of the design of dgit is Joey Hess's, at the now-famous
+session in Vaumarcus.
+
+Debian System Administration, and particularly Peter Palfrader, have
+been absolutely brilliant. The level of service and competence has
+been outstanding. Ever time I noticed some aspect I had forgotten,
+and made a blatherous and ill-thought-out request, DSA would propose a
+much better alternative and implement their end of it it almost
+immediately.
+
+Also, I'd like to thank the ftpmasters for setting up the new
+ftpmaster data service. This has enabled me to extend dgit's
+useability to non-DDs.
-Instead, I'm going to open the talk up to questions now.
+So that's most of what I have prepared. There's, of course, a lot
+more detailed information in the manpages.
-12 mins
+I'm going to take questions now.
~ijackson / talk-2015-debconf-dgit.git / blobdiff