wip

[talk-2018-dc18-gdr.git] / talk.txt

===== title slide

Hi.

===== context slide

Sean Whitton and I are here to present a new git workflow tool for
Debian packaging.

Before I tell you all about it I need to show where it fits into
the ecosystem of Debian package management software.  On this slide we
have you, the maintainer, on the left.  On the right we have the
Debian repositories.

You may have heard me plugging dgit once or twice.  You should all use
dgit!  dgit push publishes your git history so Debian's users can use
it!  But, that's not actually what I am here to talk about today.
git-debrebase does not need dgit, and dgit does not need
git-debrebase.

git-debrebase is an alternative to gbp pq and to git-dpm, and to
other tools in the same kind of area like gitkpkg.  And, of course,
it's an alternative to using raw quilt.

git-debrebase is a tool to help manage your git branch containing the
Debian version of a package you maintain.  git-debrebase helps you
maintain a useful git branch with the contents you need (for building
and uploading).

It maintains your Debian delta queue - that is, that is, the changes
you make, for Debian, to the upstream parts of the package - as a
series of commits.  It's primarily intended for Debian package
maintainers, although you could use it outside of or alongside Debian.

git-debrebase does not deal with building at all.  You use whatever
your existing build tools are.

Nor does git-debrebase deal with source packages or orig tarballs.  It
does not do uploads.  Of course when you actually upload to Debian you
need to produce a source package; usually, this would be done
automatically for you by dgit push-source.

Of course you can share your git branch on a service like salsa,
without building or uploading.

===== usp slide 1

git-debrebase offers a standard git-rebase workflow, where you edit
the whole package all together.  The experience is very like using
plain git-rebase to edit a topic branch.  Delta queue editing can be
done at any time, interleaved with packaging work.

As far as I know there are no other tools that offer these features.
Both gbp pq and git-dpm require you to switch to a separate view in
order to edit the delta queue.  Some tools have specific functions for
cherry-pick, but with none of them you can just use plain git cherry
pick or git-am onto your usual branch.  With git-debrebase, you can
just edit the code and commit it, with git, in the completely usual
way.

Specifically, at any point, you may make commits to upstream files,
and commits to packaging, in any order.  So you can just
git-cherry-pick from upstream.  You can make fixup commits, and use
the git-rebase autosquash syntax to have them automatically folded in
by the next rebase.  If you wish, you may make "mixed" commits
containing both changes to upstream files and changes to packaging
files.

Of course you can always directly edit the source if you use a plain
git merge workflow and non-quilt source package - for example, as
described in the dgit-maint-merge tutorial manpage.  But of course
that does not maintain the Debian delta as a broken-down series.
Supporting you maintaining such a delta queue is what git-rebase is
for.

Also, unlike git-dpm and some other tools, git-debrebase has no
in-tree metadata, so it can't get out of date or be desynchronised,
or need any manual changing or fixup.

===== usp slide 2

As I said, unlike with gbp pq and git-dpm, there is no need to ever
switch branches.  git-debrebase only uses one branch to handle all for
your Debian work.

Of course usually you will have an upstream remote tracking branch as
well.  And if you are working in multiple Debian releases, backports
for example, you will have branches for those.  But it's only one
branch for each line of Debian development, and no temporary branches
or alternative views.

With git-debrebase, yowr working tree is always immediately buildable
with dpkg-buildpackage (or whatever other build tool you prefer).  And
it is never made dirty by git-debrebase or any of the other tooling:
Because your working tree always has the delta queue applied, it is
never dirtied by patch application.  Because there is no metadata, you
can never get a metadata conflict.

Because git-debrebase treats the quilt patches in debian/patches/ as
an output, and handles them entirely automatically, your tree is never
dirtied by the generation of patches.  And you never need to read any
diffs of diffs.


===== usp slide 3

And, the final part of my plug:

With git-debrebase, git-blame, and git-log on a file, work entirely
properly.  For example, if you do git-log on a file from upstream
which was changed in the Debian delta queue, git-log will show the
Debian delta queue commits, preceded by the upstream history.

If you run git-blame you will see a correct indication of which
upstream and/or delta queue commits introduced each line.  Or, for a
file in the debian directory, you will see a correct reporting of
which commits in the package's packaging history introduced each line.

With git-debrebase, you never need to use the quilt program.  You can
mostly ignore the 3.0 quilt source format.  Unfortunately it is not
possible to paper over the cracks completely: you will still get
trouble if you make changes in git which 3.0 quilt cannot represent.

On the other hand, when you use git-debrebase with 3.0 quilt, the
generated 3.0 quilt source package is perfect pretty, with your delta
queue commits converted nicely into pathes - just as other people
consuming .dscs have come to expect.

And finally: of course, git-debrebase is compatible with dgit.  You do
not need to pass any quilt mode option to dgit.  And, you always can
upload right away.  All necessary bureaucracy is done automatically by
dgit push-source.

OK, that concludes the marketing spiel.  Now we're going to have a
quick demo.  Sean ?

===== demo

===== data model slide up to "2" [1]

Now you've seen it in action, I'm going to quickly run through the
data model and history structure.

There are some important detail I'm going to be glossing over, so if
you actually want to know what's really going on, please read the
reference documentation in the section 5 manpage, where everything is
fully and formally defined.

So.  This slide shows a likely situation, which you might find in the
middle of an editing session.

The horizontal part near the bottom is called the Breakwater.  This
branch contains unpatched upstream source code, plus the Debian
packaging in the debian directory.  It does not contain /any/
representation of the delta queue.  So it contains neither any
of your Debian changes to upstream files, nor any debian/patches.

In the example, commits A and B are packaging work.

The Debian delta queue sits on top of that.  In this example there are
two Debian delta queue commits, 1 and 2.  These are commits touching
upstream files.  In the diagram your current HEAD (ie, your local
master branch) is at 2.  So your tree contains the patched source code
plus the packaging: ie, it is your actual, patched, source package.

You could build it with dpkg-buildpackage -uc -b, to produce binaries
for testing.  You can git grep for things and be told where they are -
even if they are in the upstream source but in your delta queue.  You
can git log -G for things, and be told where they came from and shown
the relevant commit (whether that's upstream, or one of yours).

OK then, suppose you make a change like the one Sean made in the demo:

===== data model slide up to "D3" [2]

I'm calling this C3.  The reason for this name will be clear in a
moment.

Sean's commit edited an upstream file and also debian/changelog.  So
if you do that, your tree is, of course, still fine: you can build and
test it right away.

But suppose you want to tidy things up, and, in particular, that you
wanted the new upstream change to actually come before 2.  Maybe it
just makes more sense there, or maybe you are going to change patch 3
to make use of it.

===== data model slide with git-debrebase -i overlay

So, you run git-debrebase -i.  The usual git-rebase todo list comes
up, and you will see in it what looks like commit C3, and reorder
that.  Assuming there are no conflicts, the result looks like this:

===== data model slide up to "2'" [3]

You can see that C3 has been split into two commits: C', which
contains the changelog change, and 3' which contains the upstream
change.  The upstream change is now in the delta queue in the proper
place.

C', the packaging part of your new commit, has been pushed down to the
bottom of the stack and become part of the breakwater.

This is the general scheme of things: we have a fast-forwarding
breakwater containing packaging and unchanged upstream files.  It
doesn't have a ref to itself; instead, it is contained within your
master branch.  Each time you git-debrebase, the rebase starts on the
breakwater.

What about a new upstream version ?

===== data model slide up to "2''" [4]

To rebase onto a new upstream version, you run git-debrebase
new-upstream.  git-debrebase expects the upstream code in the form of
a git commit, of course.  (Actually, by default, it hopes to find a
tag named after the new upstream version number, but you can tell it
explicitly if that's not right.)

git-debrebase arranges to include the new upstream source into the
breakwater, and then rebases your delta queue series onto that.

There are new commits on the breakwater: firstly, a special
merge. which folds the new upstream source code, unchanged, into your
breakwater branch.  This is called an anchor merge.  The most recent
anchor merge is the backstop for rebase processing by git-debrebase.

The second commit is simply adding a new changelog entry for you.

Having provided the new base for your delta queue, it then uses
git-rebase --onto to rebase the delta queue commits.  If you didn't
ask for an interactive rebase, and there are no merge conflicts,
that's it as far as the code in git is concerned.

Of course if you are going to upload to the Debian archive you'll also
have to make an orig tarball of the new upstream.  There are tools
like git-deborig to help with that.

===== data model slide up to PM [5]

So, indeed, let's consider an upload to Debian.  (And let's imagine
you made or obtained a suitable orig tarball.)

There's a certain amount of bureaucracy to be done.  In the usual case
of an upload with dgit, this is all done for you automatically, so you
don't really need to worry about it.

But it's useful to understand what's going on, so I'm going to explain
it.

Firstly, you're going to publish your history, so your history has to
be made fast forward from the previous version of the package.  To
achieve this, git-debrebase will make a pseudomerge.

===== data model zoom in on pseudomerge

A pseudomerge is a merge commit which takes its contents from only one
of its parents.  You would make one by hand with git-merge -s ours,
if you wanted to make your HEAD fast forward, and know that all the
wanted changes from the other branch are included.

git-debrebase records the previous branch state, so that it can make
the right pseudomerge.  Your new branch is derived from the previous
one, so it is right to declare that it is fast forward, too.

The branch with the pseudomerge is suitable for pushing to any git
server.  It's what you would push to salsa, say.

===== data model zoom in on pseudomerge and add patches 4b

Secondly, when you upload a 3.0 quilt package, the contents of
debian/patches need to be right.  Again, that is taken care of
automatically: a commit is made adding a patch representation of the
delta queue to debian/patches.

You can safely ignore these autogenerated commits

===== data model slide up to PM [5] AGAIN

And indeed, they will be stripped out:

Next time you stark work, using git-debrebase, both the pseudomerge
and the patch addition will be stripped, so that you once again have a
nice delta queue to edit.

That'll put you back in a situation like I introduced at the start.

===== data model slide up to PM [5]

After uploading you'll want to push your branch to salsa, if you have
a team repository there.  That makes sure all the views of your
package are up to date, so that other members of your team won't
accidentally base their work on an old version.

You can just push a git-debrebase branch which has had the pseudomerge
made, called a `stitched' branch, with git push.  It's a normal fast
forwarding git branch.

If you want to push without uploading that's fine too: git-debrebase
stitch will just make the pseudomerge for you, giving you a
fast-forwarding branch suitable for pushing to salsa or whereever.

There is one caveat I should mention: right now, if two git-debrebase
branches diverge, it is not trivial to merge them again.  If
git-debrebase encounters a normal `git merge' it will stop and fail.
Sorting this out is not a trivial problem.  gbp pq sometimes handles
this kind of situation by expecting you to merge the actual patches:
ie, you can end up resolving merge conflicts in diffs.  The other
tools can handle this badly too.

I have ideas about how to do better at this, so watch this space.
But, for now, avoid allowing your git-debrebase branch to diverge.
git-debrebase will help you with that by often spotting when it is
about to occur.

===== data model slide merge conflict failure

What if you have a merge conflict during the upstream rebase ?

git-rebase users will have seen this kind of situation before.
git-rebase stops at the first commit which can't be applied in
the new context, and asks the user for help.

This looks quite bad.  Of course, it's not good.  But, this is an
irreducible aspect of maintaining a delta queue on top of a moving
target.  Sometimes, you'll need to fix up conflicts.

At least with git-rebase, you at least get good tools to help you fix
it up.  Some of the other workflows can involve trying to resolve
merge conflicts during quilt apply.  Much less fun.

Also, git-debrebase new-upstream is quite low commitment.  Imagine,
like on the diagram here, git-rebase has applied commit 1, and stopped
because it can't apply commit 3'.

Now, if you decide that this is too difficult to deal with today, you
can just say git rebase --abort and everything just gets put back.

===== data model slide merge conflict failure, new stuff greyed out

The autogenerated special breakwater merge, and changelog entry, are
discarded, leaving you just where you were before.  You've wasted no
effort because everything you're throwing away was machine-made.

===== status and references slide

git-debrebase is available in testing and stretch-backports and is in
good shape.  Since early versions it has been battle-tested to help
with security updates to the Debian Xen packages.  The documentation
is comprehensive.

No doubt the user interface and documentation will improve, and new
features will will be added, but you can start using it right away.

The best starting point is probably the tutorial manpage
dgit-maint-debrebase (7) (which is in the dgit package).

Sean and I are holding a workshop on Tuesday morning, where anyone is
invited to come and get help with git-debrebase - and also with dgit.
So if your questions dun't get answered before then, do drop in.

And of course I should refer to the reference documentation.
git-debrebase(5) has the data model and definition of terminology.
git-debrebase(1) is the command line reference.

But, really, start with the dgit-maint-rebase tutorial, or come to the
workshop.