minor edits from sean

[talk-2018-dc18-gdr.git] / talk.txt

===== context slide

Hi.

Sean Whitton and I are here to present a new git workflow tool for
Debian packaging.

Before I tell you all about it I need to show where it fits into the
ecosystem of Debian package management software.  On this slide we
have you, the maintainer, just off of the left of the slide.  On the
right we have the Debian repositories.

You may have heard me plugging dgit once or twice.  You should all use
dgit!  dgit push publishes your git history so Debian's users can use
it!  But, that's not actually what I am here to talk about today.
git-debrebase does not need dgit, and dgit does not need
git-debrebase.

git-debrebase is an alternative to gbp pq and to git-dpm, and to
other tools in the same kind of area like gitkpkg.  And, of course,
it's an alternative to using raw quilt.

git-debrebase is a tool to help manage your git branch containing the
Debian version of a package you maintain.  git-debrebase helps you
maintain a useful git branch with the contents you need (for building
and uploading).

git-debrebase is mainly concerned with maintaining your Debian delta
queue.  That is, that is, the changes you make, for Debian, to the
upstream parts of the package.  Other tools tend to call this your
"quilt patch series".  But with git-debrebase there is no quilt, and
you don't work with patches.  Instead, there is a series of commits.
So, we prefer the term "delta queue".

It's primarily intended for Debian package maintainers, although you
could use it outside of or alongside Debian.

git-debrebase does not deal with building at all.  You use whatever
your existing build tools are.

Nor does git-debrebase deal with source packages or orig tarballs.  It
does not do uploads.  Of course when you actually upload to Debian you
need to produce a source package.  Getting a source package is, of
course, as easy or easier than with other workflows.  Usually, it is
done automatically for you by dgit push-source, so you usually don't
need to concern yourself with .dsc source packages even when uploading
to Debian.

Of course you can share your git branch on a service like salsa,
without building or uploading.

===== usp slide 1

git-debrebase offers a standard git-rebase workflow, where you edit
the whole of the source code for your package, including your changes
to upstream files and your changes to packaging, all together.
The experience is very like using plain git-rebase to edit a topic
branch.  Delta queue editing can be done at any time, interleaved with
packaging work.

As far as I know there are no other tools that offer these features.
Both gbp pq and git-dpm require you to switch to a separate view in
order to edit the delta queue.  Some tools have specific functions for
cherry-pick, but with none of them you can just use plain git cherry
pick or git-am onto your usual branch.  With git-debrebase, you can
just edit the code and commit it, with git, in the completely usual
way.

Specifically, at any point, you may make commits to upstream files,
and commits to packaging, in any order.  So you can just
git-cherry-pick from upstream.  You can make fixup commits, and use
the git-rebase autosquash syntax to have them automatically folded in
by the next rebase.  If you wish, you may make "mixed" commits
containing both changes to upstream files and changes to packaging
files.

Of course you can always directly edit the source if you use a plain
git merge workflow and non-quilt source package - for example, as
described in the dgit-maint-merge tutorial manpage.  But of course
that does not maintain the Debian delta as a broken-down series;
and in the source package such a merge based workflow squashes all the
changes into a single debian-changes patch.

So supporting you maintaining a delta queue - that is, a linear series
of changes - is what git-debrebase is for.


Also, unlike git-dpm and some other tools, git-debrebase has no
in-tree metadata, so it can't get out of date or be desynchronised,
or need any manual changing or fixup.

===== usp slide 2

As I said, unlike with gbp pq and git-dpm, there is no need to ever
switch branches.  git-debrebase only uses one branch to handle all for
your Debian work.

Of course usually you will have an upstream remote-tracking branch as
well.  And if you are working in multiple Debian releases, backports
for example, you will have branches for those.  But it's only one
branch for each line of Debian development, and no temporary branches
or alternative views.

With git-debrebase, you can always immediately build binaries out of
your working tree, with dpkg-buildpackage (or whatever other build
tool you prefer).  And your working tree is never made dirty by
git-debrebase or any of the other tooling: Because your working tree
always has the delta queue applied, it is never dirtied by patch
application.  Because there is no metadata, you can never get a
metadata conflict.

Because git-debrebase treats the quilt patches in debian/patches/ as
an output, and handles them entirely automatically, your tree is never
dirtied by the generation of patches.  And you never need to read any
diffs of diffs.


===== usp slide 3

And, the final part of my plug:

With git-debrebase, git-blame, and git-log on a file, work entirely
properly.  For example, if you do git-log on a file from upstream
which was changed in the Debian delta queue, git-log will show the
Debian delta queue commits, preceded by the upstream history.
We'll show an example of this in the demo.

If you run git-blame you will see a correct indication of which
upstream and/or delta queue commits introduced each line.  Or, for a
file in the debian directory, you will see a correct reporting of
which commits in the package's packaging history introduced each line.

With git-debrebase, you never need to use the quilt program.  You can
mostly ignore the 3.0 quilt source format.

Not having to learn much about 3.0 (quilt) is all really good for
newbies.  Particularly, for people from other sofware development
communities who don't know about Debian.

Unfortunately it is not possible to paper over the cracks completely:
you will still get trouble if you make changes in git which 3.0 quilt
cannot represent.

On the other hand, when you use git-debrebase with 3.0 quilt, the
generated 3.0 quilt source package is perfect pretty, with your delta
queue commits converted nicely into pathes - just as other people
consuming .dscs have come to expect.

And finally: of course, git-debrebase is compatible with dgit.  You do
not need to pass any quilt mode option to dgit.  And, you always can
upload right away.  All necessary bureaucracy is done automatically by
dgit push-source.

OK, that concludes the marketing spiel.  Now we're going to have a
quick demo.  Sean ?

===== demo

===== data model slide "commit history structure" showing master=2

Now you've seen it in action, I'm going to quickly run through the
data model and history structure.

There are some important detail I'm going to be glossing over, so if
you actually want to know what's really going on, please read the
reference documentation in the section 5 manpage, where everything is
fully and formally defined.

So.  This slide shows a likely situation, which you might find in the
middle of an editing session.

The horizontal part near the bottom is called the Breakwater.  This
branch contains unpatched upstream source code, plus the Debian
packaging in the debian directory.  It does not contain /any/
representation of the delta queue.  So it contains neither any
of your Debian changes to upstream files, nor any debian/patches.

In the example, commits A and B are packaging work.

The Debian delta queue sits on top of that.  In this example there are
two Debian delta queue commits, 1 and 2.  These are commits touching
upstream files.  In the diagram your current HEAD (ie, your local
master branch) is at 2.  So your tree contains the patched source code
plus the packaging: ie, it is your actual, patched, source package.

You could build it with dpkg-buildpackage -uc -b, to produce binaries
for testing.  You can git grep for things and be told where they are -
even if they are in the upstream source but in your delta queue.  You
can git log -G for things, and be told where they came from and shown
the relevant commit (whether that's upstream, or one of yours).

OK then, suppose you make a change like the one Sean made in the demo:

===== data model slide "commit history structure" showing master=C3

I'm calling this C3.  The reason for this name will be clear in a
moment.

Sean's commit edited an upstream file and also debian/changelog.  So
if you do that, your tree is, of course, still fine: you can build and
test it right away.

But suppose you want to tidy things up, and, in particular, that you
wanted the new upstream change to actually come before 2.  Maybe it
just makes more sense there, or maybe you are going to change patch 3
to make use of it.

===== data model slide with git-debrebase -i overlay

So, you run git-debrebase -i.  The usual git-rebase todo list comes
up, and you will see in it what looks like commit C3, and reorder
that.  Assuming there are no conflicts, the result looks like this:

===== data model slide "commit history structure after git-debrebase"

You can see that C3 has been split into two commits: C', which
contains the changelog change, and 3' which contains the upstream
change.  The upstream change is now in the delta queue in the proper
place.

C', the packaging part of your new commit, has been pushed down to the
bottom of the stack and become part of the breakwater.

This is the general scheme of things: we have a fast-forwarding
breakwater containing packaging and unchanged upstream files.  It
doesn't have a ref to itself; instead, it is contained within your
master branch.  Each time you git-debrebase, the rebase starts on the
breakwater.

What about a new upstream version ?

===== data model slide "commit history structure after new-upstream"

To rebase onto a new upstream version, you run git-debrebase
new-upstream.  git-debrebase expects the upstream code in the form of
a git commit, of course.  (Actually, by default, it hopes to find a
tag named after the new upstream version number, but you can tell it
explicitly if that's not right.)

git-debrebase arranges to include the new upstream source into the
breakwater, and then rebases your delta queue series onto that.

There are new commits on the breakwater: firstly, a special
merge. which folds the new upstream source code, unchanged, into your
breakwater branch.  This is called an anchor merge.  The most recent
anchor merge is the backstop for rebase processing by git-debrebase.

The second commit is simply adding a new changelog entry for you.

Having provided the new base for your delta queue, it then uses
git-rebase --onto to rebase the delta queue commits.  If you didn't
ask for an interactive rebase, and there are no merge conflicts,
that's it as far as the code in git is concerned.

Of course if you are going to upload to the Debian archive you'll also
have to make an orig tarball of the new upstream.  If you're using the
workflow I've been describing so far, that's generally just a single
call to git-deborig.

===== data model slide "commit history structure after upload"

So, indeed, let's consider an upload to Debian.  (And let's imagine
you made or obtained a suitable orig tarball.)

There's a certain amount of bureaucracy to be done.  In the usual case
of an upload with dgit, this is all done for you automatically, so you
don't really need to worry about it.

But it's useful to understand what's going on, so I'm going to explain
it.

Firstly, you're going to publish your history, so your history has to
be made fast forward from the previous version of the package.  To
achieve this, git-debrebase will make a pseudomerge.

===== data model slide - pseudomerge box overlay

A pseudomerge is a merge commit which takes its contents from only one
of its parents.  You would make one by hand with git-merge -s ours,
if you wanted to make your HEAD fast forward, and know that all the
wanted changes from the other branch are included.

git-debrebase records the previous branch state, so that it can make
the right pseudomerge.  Your new branch is derived from the previous
one, so it is right to declare that it is fast forward, too.

The branch with the pseudomerge is suitable for pushing to any git
server.  It's what you would push to salsa, say.

===== data model slide "commit history structure after upload"

Secondly, when you upload a 3.0 quilt package, the contents of
debian/patches need to be right.  Again, that is taken care of
automatically: a commit is made adding a patch representation of the
delta queue to debian/patches.

You can ignore these autogenerated commits.

After uploading you'll want to push your branch to salsa, if you have
a team repository there.  That makes sure all the views of your
package are up to date, so that other members of your team won't
accidentally base their work on an old version.

You can just push a git-debrebase branch which has had the pseudomerge
made, called a `stitched' branch, with git push.  It's a normal fast
forwarding git branch.

If you want to push without uploading that's fine too: git-debrebase
stitch will just make the pseudomerge for you, giving you a
fast-forwarding branch suitable for pushing to salsa or whereever.

After upload, next time you come to the package, you can work directly
by adding commits on master.  If you want to rebase, or just want to
tidy the branch up, you can run git-debrebase:

===== data model slide "commit history structure rebasing again"

It strips off the bureaucracy commits.  Those remain published, of
course, but they are removed from your own master branch.

If you made any commits on top on top of the pseudomerge (or you
pulled any such commits from salsa, or whereever), it folds those back
into the breakwater and the delta queue.  So you once again a nice
delta queue to edit.

git-debrebase makes a note of where you were previously, so that when
you next want to push or upload, it can stitch the history back in
with another pseudomerge.  At the start of this walkthough, that ref
was indeed present - I glossed over that a bit.

===== data model slide "commit history structure" again

More common is a different kind of merge problem.

What happens if you one of your delta queue commits doesn't apply
during the upstream rebase - that is, your patch does not apply to the
new upstream source code ?

===== data model slide  "commit history structure  failed rebase"

git-rebase users will have seen this kind of situation before.
git-rebase stops at the first commit which can't be applied in
the new context, and asks the user for help.

This looks quite bad.  Of course, it's not good.  But, this is an
irreducible aspect of maintaining a delta queue on top of a moving
target.  Sometimes, you'll need to fix up conflicts.

At least with git-rebase, you at least get good tools to help you fix
it up.  Some of the other workflows can involve trying to resolve
merge conflicts during quilt apply.  Much less fun.

Also, git-debrebase new-upstream is quite low commitment.  Imagine,
like on the diagram here, git-rebase has applied commit 1, and stopped
because it can't apply commit 3'.

Now, if you decide that this is too difficult to deal with today, you
can just say git rebase --abort and everything just gets put back.

===== data model slide  "commit history structure  aborted rebase"

The autogenerated special breakwater merge, and changelog entry, are
discarded, leaving you just where you were before.  You've wasted no
effort because everything you're throwing away was machine-generated.

There is one caveat I should mention:

===== data model slide  "commit history structure after upload" AGAIN

Right now, if two git-debrebase branches diverge, it is not trivial to
merge them again.  The data model I'm describing does not currently
allow general merge commits.

If git-debrebase encounters a normal `git merge' it will stop and
fail.  In the general case, sorting out such a merge is not a trivial
problem.

gbp pq sometimes handles this kind of situation by expecting you to
merge the actual patches: ie, you can end up resolving merge conflicts
in diffs.  Other tools don't always handle this well either.

I have ideas about how to do better at this, so watch this space.
But, for now, teams should coordinate to avoid creating diverging
git-debrebase branches.  git-debrebase will help you with that by
often spotting when it is about to occur.

===== status and references slide

git-debrebase is available in testing and stretch-backports and is in
good shape.  Since early versions it has been battle-tested to help
with security updates to the Debian Xen packages.  The documentation
is comprehensive.

No doubt the user interface and documentation will improve, and new
features will will be added.  But you can start using it right away.

The best starting point is probably the tutorial manpage
dgit-maint-debrebase (7) (which is in the dgit package).

Sean and I are holding a workshop on Tuesday morning, where anyone is
invited to come and get help with git-debrebase - and also with dgit.
So if your questions dun't get answered before then, do drop in.

And of course I should refer to the reference documentation.
git-debrebase(5) has the data model and definition of terminology.
git-debrebase(1) is the command line reference.

But, really, start with the dgit-maint-rebase tutorial, or come to the
workshop.