-Hi. I'm here to plug dgit, which is a tool (and infrastructure) for
-treating the Debian archive like a git remote.
+==== overview slide
-I'm going to talk for about 35 minutess and then I'll take questions.
-----------------------------------------
+Hi. I'm here to plug dgit, which is a system for treating the Debian
+archive like a git remote.
+
+I'm going to talk for about 25 minutess and then I'll take questions.
+
+
+Firstly, a bit of framing.
When we work on Debian we take on different roles. The biggest
difference is between the maintainer (or maintainers) of a package,
working on their own package, and everyone else.
I'm going to start by presenting dgit from the point of view of
-everyone else: NMUers, sponsorship, teams doing cross-archive work
-like transitions and reproducible builds, bug squashers, downstreams,
-users, and so on.
+everyone else: NMUers, sponsorship, bug squashers, downstream distros
+and users, teams doing cross-archive work like transitions and
+reproducible builds, and so on. Maintainers, please be patient - I'll
+get to you a bit later in the talk.
-The point of dgit is that it lets you treat the archive as if it were
-a git repository.
+==== manpage slide
-You can dgit clone any package, in any suite in the archive (so, for
-example, sid or experimental) and you will get a git tree which is
-exactly the same as dpkg-source -x.
+The point of dgit is that it lets everyone treat the archive as if it
+were a git repository.
+
+You can dgit clone any package, in any suite (so, for example, sid or
+experimental) and you will get a git tree which is exactly the same as
+dpkg-source -x.
+
+So dgit always works the same, from the non-maintainer's point of
+view, on any package: the operation is completely uniform. You don't
+need to know anything about the maintainer's verson control workflow
+tools or source format preferences.
You can then work on the package in git, the way you would work in git
with any other project. In particular, you can:
- * commit on your local branch
+ * commit locally
* cherry pick changes from other branches
* git reset, git clean
* git rebase -i to polish a more complex set of changes into
a patch queue
+ * all the other usual gitish stuff
-If you want so share your work-in-progress with others, you can git
-push your branch anywhere where they will be able to fetch it.
+If you have the right authority you can also dgit push, to upload.
+That is, a DD can dgit push any package; a DM can dgit push the
+packages that the archive thinks they can upload.
-If you have the right authority you can also dgit push, to upload
-(after doing a build).
+Before you push you still have to do a build. dgit does not replace
+existing ways of building source and binary packages, although it does
+provide some helpful build rune wrappers (more about that later).
-dgit is not a replacement for existing git packaging tools; it's
-intended to complement them. So (unlike git-dpm) dgit does not define
-a git history structure, and (unlike git-buildpackage) it does not
-define a branch structure. Nor does it require a particular source
-format.
+If you don't want to, or can't, upload to Debian, but do want so share
+your work with other people, you can git push your dgit
+branch anywhere suitable, so other can fetch it. So, for example, you
+could share your branch with your sponsor, who could then approve it
+by running dgit push. dgit's branches are ordinary git branches for
+this purpose.
+A downstream such as a derivative or partial derivative of Debian can
+use the dgit branches directly as the upstreams for a git-based setup,
+and work entirley without source packages.
-The dgit history structure is up to the maintainer. If you are doing
-a straightforward NMU you should produce a well-structured linear
-sequence of commits, as you would for any git upstream. If the
-package is using a quilty source format, you should not touch
-debian/patches; dgit will take care of updating the quilt patch stack.
+==== data flow slide
-Unless the maintainer uses dgit, the history you see in dgit will not
-be the maintainer's history. This is because maintainers' git
-branches are often not identical to the source packages in the
-archive.
+Behind the scenes, dgit works by providing a set of git repositories
+in parallel to the existing archive.
-And, when dgit has to synthesise a git history because the maintainer
-is not using dgit, the history you see in dgit will have a very basic
-branch and commit structure, rather than representing the package's
-actual history.
+Every dgit push is actually two underlying operations: the first is a
+git tag and push, to the dgit git server. The second is a
+conventional package upload - with a wrinkle: the .dsc of an upload
+done with dgit contains an extra field with the git commit hash that
+was pushed.
+
+Likewise, fetch and clone combine information from the archive and the
+git history. If the most recent upload was done with dgit, the commit
+hash in the dsc enables dgit fetch to find the right commit and
+present it to you. If the most recent upload was not done with dgit,
+dgit imports the source package into git - and stitches it into the
+existing dgit history, if there is one.
+
+You do need to treat dgit git branches a bit specially is if you need
+to build source pacakges (for upload, for example). In this case dgit
+needs the .orig tarballs for your source package. If you are not
+doing a new upstream version, dgit fetch will get the relevant .origs
+for you. If you /are/ doing a new upstream version, then presumably
+you have obtained the origs as part of preparing your package, or you
+can build them easily.
+
+
+==== NMU linear history slide
+
+As a general rule, the dgit history structure should be up to the
+maintainer - at least, if they care.
+
+If you are doing a straightforward NMU you should produce a
+well-structured linear sequence of commits, as you would for any other
+git upstream, based on the dgit suite branch. Not only does this mean
+that if the maintainer is using dgit, they can hopefully easily
+include your changes; it also means that if they _aren't_ using dgit,
+at least you have published a history which is suitable for rebasing
+onto theirs, or whatever it is they might want to do with it.
+
+If the source package is `3.0 (quilt)', you shouldn't touch
+debian/patches; dgit will take care of that for you. This is the
+other reason why you should provide a tidy linear patch series: if the
+maintainer likes quilt and is not using dgit, your changes will be
+automatically presented to them in a fairly plausible format - a
+series of extra patches - like they should expect from any other NMU.
+
+An ordinary NMUer should not normally update a patch in the quilt
+stack directly. Ie, an NMUer shouldn't squash their changes into an
+existing patch. This is because while it's easy for the maintainer to
+squash it themselves, if they want, it's a little harder for the
+maintainer to disentangle a squashed patch. Working directly on
+patches can also result in people having to read interdiffs, which are
+notoriously confusing.
+
+
+So, in summary, if you are doing cross-archive work, or bug squashing,
+you can start using dgit right away and I hope it will make your life
+a lot easier.
+
+
+==== NMU linear history on top of basic dgit history
+
+Sadly, unless the maintainer uses dgit, the history you see in dgit
+will not be the maintainer's history.
+
+This is because maintainers' git branches often differ from the source
+packages in the archive.
+
+If you dgit clone a package and it has an X-Vcs-Git header, dgit will
+set up a remote for that, so you can fetch the maintainer's history and use it if you
+like. So in that sense dgit clone encompasses debcheckout.
+
+But, in the general case, the X-Vcs-Git tree may not be immediately
+useable to someone not familiar with the package, so dgit doesn't use
+it for the dgit suite branch.
+For example, the maintainer's repo might contain only a debian/
+directory, or be a quilty tree without patches applied. And the tag
+and suite naming conventions can vary too. So while the maintainer's
+history can be useful if you want to do archeaology, it's not in
+general suitable for use by dgit.
+
+There is also the problem that the maintainer's nominated git server
+might be anywhere, so it might be down, or gone away, or compromised.
+
+
+So, if the maintainer is not using dgit, dgit has to synthesise a git
+history. The history you see in dgit will then have a very basic
+branch and commit structure, rather than representing the package's
+actual history, something like you see here.
-----------------------------------------
Which brings me onto the other side of this talk: dgit for
-maintainers.
+maintainers:
+
+==== history comparison slide
+
+For the reasons I've explained, downstream dgit users would like you
+as a maintainer to use dgit push to do your uploads. They will then
+be able to see, and directly work with, your own history.
-For the reasons I've explained, other dgit users would like you to use
-dgit for your maintainer uploads. They will then be able to see, and
-directly work with, your own history.
+More generally, the point of using a dvcs like git is to publish your
+work. The existing ways of publishing git histories for Debian
+packagess aren't uniformly useable for users: they require the user to
+understand the maintainer's git working practices.
-But it's in your own interests, too:
+What dgit does is provide a way for you to publish a history which
+users can rely on actually corresponding to the archive, and use
+immediately without special knowledge.
-If you use dgit, you will be able to directly merge and/or cherry-pick
-NMUs, patches proposed via pull-request emails, and so on: Because, in
-this case, the dgit-using contributor will have based their work on
-your own history.
+But it's in your own selfish interest to upload with dgit, too:
-Using dgit for your maintainer uploads will put your own history on
-browse.dgit.debian.org, rather than showing dgit's stub history (which
-can also be out of date).
+If you use dgit, you will be able to directly merge NMUs, branches
+proposed via pull-request emails, and other similar contributions:
+Because, in this case, the dgit-using contributor will have based
+their work on your own history. Whereas, if you don't use dgit,
+dgit-using contributors will be working on a stub history, and may
+dgit push commits based on that stub. You can dgit fetch that work
+even if you're not using dgit for your own uploads, but if you're not
+using dgit push yourself, at the very least you'll have to rebase the
+NMUers' and other contributors' work onto your own history.
+
+Another advantage of using dgit for your maintainer uploads is that it
+will put your own history on browse.dgit.debian.org, rather than
+advertising dgit's stub history (which can also be out of date).
If you use dgit push, you get an extra check that the source package
you are uploading is exactly the same as your git HEAD. This can
save you some dsc-based checks.
-And, of course, non-maintainer dgit users will thank you.
+And, of course, as I say, doing your uploads with dgit will improve
+downstream dgit users' lives.
+
+
+==== data flow slide
+
+dgit is not a replacement for existing git packaging tools; it's
+intended to complement them. So (unlike git-dpm) dgit does not define
+a git history structure.
+Nor, unlike git-buildpackage, does dgit define or suggest a branch
+structure for naming upstream or downstream branches, pristine tar
+branches, etc.
-dgit push imposes only two requirements on your git trees. These stem
-directly from the need for the dgit view of package to be identical to
-the package as found in the archive. The dgit view of every package
-needs to uniformly and directly useable.
+dgit doesn't require a particular source format; it couldn't, since it
+needs to work with any package.
-The first, and most important, requirement is that your git tree is
-identical to the unpacked source package. (Technically, in the case
-of a `3.0 (quilt)' package, it is what is sometimes called a `patches
-applied packaging branch without .pc'.)
-For all native packages, and for users of git-dpm and raw git, this is
+==== data flow slide with EQUAL and FF
+
+dgit push imposes only two requirements on your git trees, which stem
+directly from dgit's objectives.
+
+The most important requirement is that your git tree is identical to
+the unpacked source package.
+
+For all native and source format 1.0 packages (which will include
+people using raw git to manage their delta from upstream), this is
already the interchange format. These maintainers can start using
-dgit right away. Please do!
+dgit right away for all their maintainer uploads. Please do!
+
+The other requirement of dgit is simply that the dgit branches are
+fast-forwarding. So if your tools have made a rebasing branch, you
+may need to make a fake merge (with git merge -s ours) before pushing.
+I'm intending to provide some rather more cooked way to do this but I
+haven't decided the exact shape yet.
+
+
+==== 3.0 quilt applied slide
+
+For `3.0 (quilt)' packages, things are more complicated. The purpose
+of dgit is to provide a git view which is identical to the source
+package, and can be worked on without knowledge of the maintainer's
+chosen source format, patch system or git workflow.
+
+This means that the dgit git tree for a `3.0 (quilt)' package is what
+is sometimes called a `patches-applied packaging branch without .pc
+directory'.
+
+This means that all the changes made in Debian are represented in the
+main source tree, as well as being contained within a patch in
+debian/patches.
+
+But: I think the most popular tool for working with `3.0 (quilt)' is
+git-buildpackage, and git-buildpackage likes to work with
+patches-unapplied tress. I've spoken to Guido Guenther (the
+git-buildpackage maintainer) about this. We had a really good
+conversation. We have a plan for interoperating, which involves dgit
+synthesising an extra git commit, to apply the patches, and then
+pushing that to the dgit git server. (The extra git commit wouldn't
+appear on your own branch; if it did, it would cause trouble.)
+
+=== xxxx status table
-For git-buildpackage users, things are a bit more complicated. I'm
-told that gbp pq can be used to generate a patches-applied branch, and
-that some users prefer to use that as the interchange git branch.
+There is another problem which affects both git-buildpackage and
+git-dpm: namely, .gitignore files. Neither git-buildpackage nor
+git-dpm represent the .gitignore, which typically occurs in the git
+tree, as a patch in debian/patches. Rather, these tools tell
+dpkg-source to ignore it when buildinng the source package. So the
+source package does not contain .gitignore, although the git tree
+does.
+I have spoken to Bernhard Link (the git-dpm maintainer) about this and
+he wasn't keen on having git-dpm treat .gitignore normally, and record
+.gitignores in a patch in the git-dpm branch. So unless we can
+persuade Bernhard to offer this, at least as an option, dgit is
+probably going to have to synthesise a commit to add them, in much the
+same way as it will add a commit to convert a git-buildpackage
+patches-unapplied branch to a directly-editable patches-applied one.
-I mentioned that there were two requirements. The other is simply
-that the dgit branch (one for each suite) is fast-forwarding. So if
-you (or your workflow tool) generated a rebasing branch, you may need
-to do something like
- git merge -s ours dgit/sid
-before pushing. I'm intending to provide some rather more cookied
-way to do this but I have decided the exact shape yet.
-----------------------------------------
+xxx empty directories
+
+
+ I'm told that gbp pq can be used to generate a
+patches-applied branch, and that some users prefer to use that as the
+interchange git branch, but I know this is far from universal. I'm
+talking to the git-buildpackage maintainers about gbp integration, so
+watch this space.
+
+
+, and for users of git-dpm and
+
+(more or less)
+ xxx
+
+
+
+==== data flow slide
There are a few other things I ought to cover, since they often come
-up:
+up. They're are relevant to maintainers and non-maintainers:
+
+
+Firstly, some wrinkles.
+
+The first wrinkle is that DMs currently need to email me a signed copy
+of their ssh key, in order to be able to push. This is because the
+dgit repo server uses ssh as a transport and the project doesn't,
+right now, have a record of DMs' ssh keys.
+
+The second thing that's less than ideal is that the dgit git history
+does not generally include the package upload history.
+git-import-dscs can produce a git branch representing the upload
-The dgit view of the archive does not include the package upload
-history. It is possible to use git-import-dscs to produce a git
-branch representing the upload history, but dgit does not do this
-itself.
+xxx patches-applied
-There is nothing stopping anyone from pushing such a branch to the
-archive with dgit push. But, it seems to me that the git history
-structure ought to up to the maintainer, and if the maintainer chooses
-to use dgit, normally the maintainers's existing git history is more
-useful and interesting.
+history, but dgit doesn't run that itself. It would be difficult for
+dgit to do so because deciding which set of versions to include is
+nontrivial and of course it would involve an awful lot of downloading.
+One could push such a branch to the archive with dgit push. But, it
+seems to me that the git history structure ought to up to the
+maintainer, and if the maintainer chooses to use dgit, the
+maintainers's existing git history is probably better.
-When using dgit, it is normally best to use one of dgit's build
-operations to build the package. This is mainly because most other
-build tools by default remove .gitignore when generating the source
-package. dgit requires that the source package and git tree are the
+So I think the real way to improve this is to persuade more
+maintainers to use dgit. Perhaps for maintainers who do not, we
+should at some point consider providing centrally an archive-based
+package history.
+
+
+==== dgit requirements slide
+
+But the most obvious challenge for a maintainer with an existing git
+branch, but trying to use dgit, is dgit's insistence that the source
+package and git tree are the same.
+
+Sadly, quite a few source packages contain files not in the
+maintainers' git branches, but which are needed to build: most
+commonly, autotools output. Such git branches are not useable with
+dgit.
+
+But nowadays most people recommend that the package build should
+always rerun autotools. If you do that, then neither your git tree
+nor your source package need contain the autotools output and all is
+well.
+
+Alternatively, you can commit the autotools output to git. Merge
+conflicts, which do occur occasionally if you do this, are easily
+resolved by rerunning autotools.
+
+
+And a second way dgit's rule can bite is .gitignore. Most existing
+packaging tools remove .gitignore from source packages by default.
+But, dgit dgit requires that the source package and git tree are the
same, so if your git tree has .gitignore in it, your source package
should too.
+==== manpage slide
+
+So it is normally best to use one of dgit's build operations to build
+for upload; in generally, all that those build operations do
+differently to the underlying tool, is pass some appropriate -I
+options, to exclude exactly and only the .git directory.
+
+
+==== manpage clean slide
+
+Finally, there is one compelling advantage of dgit's git-based
+approach.
Many packages have strangely-behaved or plain buggy clean targets.
Because dgit knows that your git tree is canonical, it can help work
-around this: you can tell dgit to use git clean instead, avoiding the
+around this: you can tell dgit to use git-clean instead, avoiding the
package's clean target entirely.
+If you're not in the habit of forgetting to say git-add, you can set a
+configuration option to have dgit always use git-clean. Then you will
+never have to fight a buggy clean target, in a strange package, ever
+again.
+
+
+
+==== Future plans slide
+
+I have a number of plans for the future, some of which I may even get
+around to:
+
+
+dgit can't currently upload to DELAYED queues, which makes it a bit
+awkward for a fire-and-forget NMU. One way to implement this would be
+to push to the suite branch but send the package upload to DELAYED.
+But such an upload which was deleted by the maintainer would still
+present be in the suite branch, and the changes might be resurrected
+by a future dgit user. At the very least there should be a way for
+the maintainer to revert a push which was accompanied by a DELAYED
+upload.
+
+The other implementation would have the pushed branch stashed away
+somewhere and "revealed" only when it was ready, if it was still a
+fast forward. That makes it tricker for someone else to find it to
+work on top of it, and is more work to implement, but it's probably
+better.
+
+
+It would be easy for dgit, when pushing an NMU, to construct the
+NMUdiff and email it to the BTS. Of course a maintainer can fetch
+your NMU as git commits easily with dgit fetch, but maybe the
+maintainer isn't using dgit (or, even, isn't using git at all), so you
+for the benefit of those maintainers we should still be sending these
+diffs.
+
+
+dgit push is slightly more work than it needs to be when doing a
+source-only upload. In this case there is not really any need for a
+separate `dgit build-source' step. I think there should be a way to
+have dgit push make the source package as well, since it easily can.
+
+
+The dgit git server has most of the moving parts to become a
+fairly-general-purpose git server with access control equivalent to
+the Debian archive. In this role it could replace some uses of
+Alioth, which is good because Alioth is doing too many other things.
+
+In particular, with the use of signed pushes, we could have
+traceability of pushes, which makes a gitish packaging workflow more
+practical - in particular, maintainers would not need to as thoroughly
+audit work found on for-upload branches on the git server, before
+pushing the code to the archive. (I suspect many maintainers already
+don't do this as thoroughly as they should.)
+
+
+dgit requires the suite branches to be fast forwarding. If you have a
+workflow involving git rebase, you need to make a fake merge (ie, a
+merge made with git merge -s ours), at the tip of your branch, to make
+it be a fast forward from the previous state of the suite branch. And
+then when you want to work on the package again, you need to strip
+that same fake merge.
+
+This can be done with raw git operations (git merge and git reset)
+quite easily, but it's ugly. Also doing it by hand like this doesn't
+provide many safety catches.
+
+I intend to provide some tooling which will help with this, but I
+haven't exactly decided what the UI should be and how it should work.
+
+
+The final thing I can do myself is arrange for non-dgit uploads to be
+automatically converted, and appended to the dgit suite branches on
+the dgit git server. That would make the dgit history, as shown on
+browse.dgit.debian.org for example, for any package be more useful,
+starting with the first dgit-based upload. In particular it would
+avoid the browse view ever being out of date.
+
+
+There are also some things that I need help with.
+
+The documentation for how dgit works together with existing git
+management tools is rather sparse.
-Some maintainers' source packages contain files not in their git
-branches: most commonly, autotools output. Because the point of dgit
-is to provide a git view of what's actually in the archive, this does
-not work with dgit.
+Ideally each git workflow tool would explain how (or whether) it works
+with dgit. Tasks that should be covered include how to incorporate
+changes made in an NMU into the tool's favoured history format, and
+how to upgrade a package to a new upstream version. I have spoken to
+the git-buildpackage and git-dpm maintainers about this. I'm
+optimistic particularly about git-buildpackage.
-But nowadays most people seem to recommend that the package build
-should rerun autotools and regenerate these files, anyway. If you do
-this, then neither your git tree nor your source package contain the
-autotools output and all is well.
-Alternatively, you can commit the autotools output to git (and resolve
-any conflicts by simply rerunning autotools), or maintain a separate
-post-autotools git branch.
+There is a difficulty with empty directories in source packages. git
+refuses to handle empty directories - they tend to just vanish from
+commits and checkouts. But I'm told there are some source packages
+whose orig tarballs contain empty directories, and build systems
+depend on those empty directories.
+Currently if you dgit clone such a package, it won't build. I don't
+have a particularly good answer to this problem right now. It's no
+good to have dgit create these directories in the working tree, when
+it notices, because the git branches might be transported by plain git
+and used elsewhere.
+This can be worked around in the source package by either adding
+something to debian/rules to create these directories, or by adding
+some files to the directories to make them nonempty. But at the
+moment a maintainer is not required to do either of these things.
+And then there are a couple of exciting possibilities for new uses:
-The point, of course, of using
+Firstly, it would be very nice to have a way for a sponsee to provide
+a dgitish git branch, which the sponsor could work with directly and
+ultimately dgit push. I'm really need to talk about this with some
+people more familiar than me with the sponsorship queue
+infrastructure.
+Secondly, dgit is not just for Debian. Other distros currently using
+only source packages can use it too. Read-only support doesn't
+necessarily depend on specific support at the distro end. So, you can
+already dgit fetch from ubuntu; I'd welcome patches for read-only
+support for Mint, for example.
+dgit push support requires a suitably set up git server, and maybe
+some thought, depending on the distro's views on push access control.
+I hear rumours that Ubuntu might grow a dgit git server and support
+dgit push. Patches very welcome!
+==== Thanks and more info slide
+Finally, there are some people I need to thank.
+At least half of the design of dgit is Joey Hess's, at the now-famous
+session in Vaumarcus.
+Debian System Administration, and particularly Peter Palfrader, have
+been absolutely brilliant. The level of service and competence has
+been outstanding. Ever time I noticed some aspect I had forgotten,
+and made a blatherous and ill-thought-out request, DSA would propose a
+much better alternative and implement their end of it it almost
+immediately.
+Also, I'd like to thank the ftpmasters for setting up the new
+ftpmaster data service. This has enabled me to extend dgit's
+useability to non-DDs.
- defer talking about
+So that's most of what I have prepared. There's, of course, a lot
+more detailed information in the manpages.
+I'm going to take questions now.
~ijackson / talk-2015-debconf-dgit.git / blobdiff