git-debrebase: convert-from-gbp; on stitch failure, clarify message

Clarify what the preceding failure messages mean, and what the
consequences are.

This is part of #906641.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gdr-convert-gbp: Break the start out into lib-gdr

This will allow us to test other things to do with this conversion
without making tests/setup/gdr-convert-gbp more complicated.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: convert-from-gbp: Print a success message.

This is part of #906641.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Add close note for #905400 to changelog entry for 6.5.

The commit
  git-debrebase: Properly reject bare dgit dsc imports.
fixed #905400.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gdr-convert-gbp: test snag on unused patches

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: convert-from-*: Snag on unused patches

Patches in d/patches which are not in series will be deleted, and not
represented in the output.

So call them a snag.

Closes:#904997.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: get_tree: Support $precheck and $recurse

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gdr-subcommands: Test raw conversion from gbp

Make a branch which strips off the pseudomerge generated by gdr to
stitch the dgit history, and then run some tests on that.

These new tests detect #905975.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gdr-subcommands: Use $peel, not $work, for a few checks

The definition of P says
    P        result is add-patches, remaining letters apply to result~

But this was not hnoured by N n Z.  Honour it.

No change with existing tests, which do not follow P with any of
those.  But, this will be important in a moment,

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gdr-convert-gbp: Save converted-from-gbp

This new branch will be useful in a moment.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: On rebase, always save ffq-prev

Even if we were both stitched and laundered.

Closes:#905975.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: `prepush' is now a silent no-op if unstitched.

This is more friendly.  Update the docs accordingly.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Add perf numbers

This is from tests on my laptop with the branch the Debian kernel
folks shared with me, ie c6eb63c6ed0694b0d22fbe5aaff953d209a3fc4e.

                                     avoid        new
   runtimes in [s]         before    read-tree   get_differs

  git-debrebase             78.574    3.699       1.419
  git-debrebase status       3.359    unchanged   0.818

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Rewrite get_differs in terms of trees_diff_walk

As noted, git-diff-tree is quite slow.  We can do much better, by
stepping across the two trees ourselves.

There should be no particularly significant change: the answers should
be the same for reasonable inputs.  Unreasonable inputs (particularly,
executable files etc.) may be handled better, even.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: get_tree: Support '' for nonexistent objects

This is going to be convenient in a moment.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: New get_tree machinery: Fix debugging

When we localise, we get a new variable, rather than localising the
binding imported from Debian::Dgit.  How annoying.

Also, add two new debug prints at level 3.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Fix comment documenting D_DEB_*

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Actually avoid read-tree in walk when not rewriting

See
   git-debrebase: Prep for avoid read-tree in walk when not rewriting
for the motivation.

Here we try to check that the commit we would make, if we were
actually calling read-tree, write-tree and hash-object, would be
identical to the one we are reusing.

In particular, we check that the upstream files, and the debian/
directory, have the contents we would feed to read-tree.  And we check
that the parent list is the same.

With these checks this new algorithm is, I think, safe enough to just
use always.   So set $opt_careful to 0.  We will abolish this
variable in a moment.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Provide new get_tree and trees_diff_walk

These are ways to avoid calling git-diff-tree, which is very slow even
with --name-only --no-renames and without recursion.  I think it is
doing all the work to make nice actual diffs, and then throwing them
away.

Two calls to git-ls-tree is about 20x faster than git-diff-tree
on the Linux kernel, for example.

No functional change in this commit, since no callers yet.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: unset GIT_EDITOR, so it works if user has that set.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Rename $want_tree_ to $want_

NFC

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Prep for avoid read-tree in walk when not rewriting

Because the git index is a flat list of files, not a directory
hierarchy, git-read-tree can be very slow.  We want to avoid it
if possible.

Introduce a new way of working, where we defer calls to read-tree.

In detail:
 * $read_tree_upstream and $read_tree_debian, which are the only ways
   that walk invokes read-tree, simply record their argument;
 * The actual read-tree is done just before the write-tree and
   commit generation.  read_tree_upstream, conveniently, does both
   halves, if it gets both arguments.
 * Put all of the read-tree and commit regeneration in a branch
   triggered if we are rewriting or "trying to be careful".

Right now we are always careful, but I have tested this with a hack
setting $opt_careful to 0, and the performance on the example branch
in #905995 is much improved: 3.4s rather than 77s for `git-debrebase'.

This is not suitable for enabling in its current form.  I want to
actuallly abolish the $opt_careful, and, instead, check with
get_differs or git-cat-file that the tree segments, and the parent
lists, are identical.

When we do that we won't be checking any more that the commit
generation can generate identical commits when not rewriting.  But it
doesn't really matter much any more provided the commits are
well-formed and right, and the test suite will check that.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit(1): Encourage --overwrite rather than --overwrite=version.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: walk: Introduce $read_tree_debian

We are going to make these do something more efficient, sometimes.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: walk: Move $read_tree_upstream into the loop

NFC.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Drop a redundant call to $read_tree_upstream

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: conflict: Some xxx's now done

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: test conflict: Test resolution via patch

This is a really horrible way to do things.  We need a better
way to let the user help resolve this kind of merge.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Actually use merge resolutions

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Actually record merge resolutions

And, destroy it on scrap.

Nothing consumes this yet.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: scrap works properly when it does only rebase --abort

We use @deferred_updates to see if we actually did anything for
--noop-ok purposes.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: test conflict: Break out no-wreckage

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Change wreckage to refs/debrebase/wreckage/*

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Dgit.pm: Move reflog_cache_* from dgit

We are going to want this for git-debrebase merge resolutions.

NFC other than the usual change to literal `git'.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Dgit.pm: Move $maindir* up the file.

We're going to want these for the reflog cache functions.

NFC.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Dgit.pm: Move make_commit_text from dgit

NFC other than the usual change to literal `git'.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: New record-resolved-merge subcommand, WIP

Right now this seems to be able to generate some commit, but
it doesn't know how to record it.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Split off merge_series_patchqueue_convert

We're about to reuse this.

NFC.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Record the new-base in the wreckage.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Use a fresh branch and check out again

We are going to want this in a moment.

No overall functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Introduce $merged_pq

We are going to want this in a moment.

NFC.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Use a couple of globals instead

We are going to make some of this into its own sub, and copying the
convenience functions seems annoying, so do this.

NFC.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Introduce merge_attempt_cmd global

NFC

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Do wreckage cleaning separately

git-update-ref hates us if we delete a ref and reintroduce it, in the
same transaction.  In this case we don't mind doing it in two goes.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: fresh_workarea: Support $subdir

And use this in merge_series.

NFC.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Prettify subprocess failures

If we print STDERR in advance, there is a newline beforehand, which is
undesirable.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: test gdr-merge-conflicts: Test late wreckage

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: test gdr-merge-conflicts: Break out wreckage-*

NFC

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Minor style and error message change

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Call merge_failed in merge_series

Introduce $attempt_cmd for this purpose.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Defeat evals which might try to trap merge_failed

That would be dangerous because we're in the wrong directory and other
things may be wrong.  For example, if we continue we might overwrite
wreckage with other wreckage.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Save wreckage in $maindir

We should change to $maindir before exiting.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Always save wreckage

There is no point configuring this.  We tidy it up again soon enough.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: test suite: Drop check of no wreckage save

We are going to make this nonconfigurable.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: test suite: Check wreckage tidying

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Wreckage: tidy it up on successful commands

Whenever we do a final ref update, discard any wreckage.  The wreckage
can be autoregenerated, so it is not very important.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Break out merge_wreckage_cleaning (nfc)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: test suite: Check wreckage saving

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Save wreckage if desired

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Introduce new merge_failed sub

And call it from $nomerge in walk.  Now we produce a sane error
message.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Make mwrecknote call printdebug

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Miscellaneous mwrecknote fixes

Add the missing commitish for the -patchqueue wrecknote.

Defend against undef (or too few) args to mwrecknote.
(The way we end up calling it means perl doesn't notice.)

In walk, ensure that we initialise $cl->{MergeWreckNotes}.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Break out run_ref_updates_now

NFC

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Dgit.pm: failmsg: Remove all double newlines, not just the first

I don't think this makes any difference with the current error
messages.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: gdr-merge test: Test an early failure too

And sort out the section comments.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Set DEBFULLNAME

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Introduce global mwrecknote sub

NFC

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: gdr-merge-conflicts test: New test

Will be improved later.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: gdr-merge test: sort out section comments

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Do not crash on branches with no delta queue

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Do not bail making patches if there aren't any.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: test: Introduce t-merge-conflicted-*

NFC

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: test: Drop stuff cloned from gdr-viagit

We don't much are about this here.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Add wreck note recording to merge_series

We also add a LeftRight to the merge base delta queue hashref, so that
it can be treated the same as the left and right delta queues.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Clarify an internal commit message

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Break out $mdebug

NFC.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Pass wreckage hashref to merge_series

NFC so far.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Introduce $mwrecknote in walk analysis half

We collect various git objects that we may want to present to the user
as wreckage.  Currently we dump them in $cl and print them out in
$nomerge.  We'll want to do better than that of course.

Also, we don't collect any wreckage for the actual series merge yet.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Whitespace changes

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Refactor interchange merge-base computation

We need to unpack this a bit so that we can insert some wrecknote
stuff, which will come in a moment.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Fix a $nomerge message

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Add a comment about rewriting

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Remove a redundant check

MergedBreakwaters commits are now handled by a completely other branch
of the if, so this check is now redundant.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: mergedbreakwaters_anchor: Check consistency

Check that the result contains all the other anchors.  This is
ensured by walk, and required of VanillaMerges.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Avoid needlessly walking MergedBreakwaters

We will not ever need to rewrite one of these, so we can just ask
keycommits for its anchor.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: test suite: Use t-ref-same-start

This is better than t-gdr-good laundered because (i) it is faster (ii)
it checks that the second laundry was a no-op (and did not rewrite
unnecessarily).

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: test suite: test avoiding re-merge of debian/

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Avoid asking user to re-merge debian/

If it had conflicts, they will have been resolved earlier and we can
just take that resolution.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: test suite: Do a bit more testing

We can now call t-gdr-good laundered, because it works.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: lib-gdr: t-gdr-good: Handle MergedBreakwaters

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Add another xxx comment

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: walk: Reintroduce the local $read_tree_upstream sub

Now that we have the $tree_with_debian parameter to the global
read_tree_upstream, we can use it to avoid the git-write-tree call
and pass $build as the $tree_with_debian.

This thus partially reverts
  dfc092e0ec3d932e51a071000519477d0e33525c
  git-debrebase: Make read_tree_upstream global (nfc)

The combined effect is now that $read_tree_upstream does exactly the
git commands it did before.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Get debian/ parts of new patch queue right

We want the newly generated delta queue commits to have the $newbase's
debian/.  Previously we let read_tree_upstream save and restore
debian/, but the contents of debian/ in the index are not right:
they're the result of the git merge, which is whatever is left over
from earlier.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: read_tree_upstream: New $tree_with_debian parameter

If not passed, this function does the same as it did before.

NFC.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Generate some more debugging commits

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: keycommits: Handle merged-breakwater commits

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: keycommits: New $claimed_bw optional argument

No users yet, so NFC.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: merge: Generate merged-breakwaters with correct metadata

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: keycommits: Break out $found_anchor

We are going to want to reuse this.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: lib-gdr: t-gdr-good: Introduce t-gdr-analyse

NFC

This will allow us to recurse.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>