test suite: test-list-uptodate: Drop imports and dependencies

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gdr-fresh: Drop use of git-deborig and fix dependencies

b765388a4cea
  "test suite: Test dgit calling git-debrebase on new debianisation"
failed to update the test dependencies, and does not work in a formal
autopkgtest run because dgit is not installed.

Additionally, Because scripts in devscripts have their own individual
dependencies, and the dependencies for git-deborig differ between
stretch and sid, It is not reasonably possible to write a dependency
which allows git-deborig to work.  This is another reason why this
test does not work in a formal run (this time, failing only on
sid/buster).

So, drop use of git-deborig, and the resulting dependency on
devscripts.  (dgit and git-debrebase pull in devscripts anyway, so
that is academic.)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase(5): Document new understanding of debrebase-last

Bug #907208 shows that the previous statement wasn't true.

debrebase-last is still marginally useful: we use it for deciding what
hints to print in some error messages; detecting and rejecting
attempts to convert already-converted branches; and in
`git-debrebase status'.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: branch_is_gdr: Add a cautionary comment

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Test dgit calling git-debrebase on new debianisation

Now all the cases in dgit's branch_is_gdr history walker are tested:

  DGIT_TEST_DEBUG=-DD tests/using-intree tests/run-all
  cat tests/tmp/*.log |perl -ne 'next unless s/^branch_is_gdr  \w+ //; print' |sort -u |less

Produces the same list as:

  git-grep 'branch_is_gdr  ' | perl -pe 's/^^dgit:\s+//' |sort |less

(With the list of gdr commit annotations the same as in the source.)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebse: merge handling: Handle absence of patches via $any

This saves testing for $seriesfile existence and makes the code
slightly less tangled.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: make-patches: Handle case when there are no patches

Previously, we would bomb out.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: keycommits: Fix doc comment

There's no $dgitimport; that's $trouble.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: convert-to-gbp: Handle no-patches case better

In this case it's very like git-debrebase forget-was-debrebase.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: convert-to-gbp: Actually start from breakwater

Contrary to the documentation, this operation would start from HEAD.
It should rewind to the breakwater.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: make_patches_staged: Return flag, are there any patches?

All callers currently ignore the return value, so NFC.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: gbp_pq_export: Return flag, are there any patches?

All callers currently ignore the return value, so NFC.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: branch_is_gdr: Put gdr commit annotations in alphabetical order

This makes it slightly easier to see our test coverage.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Separate t-gdr-gbp-import-core from -with-queue

NFC

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Test dgit branch_is_gdr walker on a merge resolution

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Test dgit detection of fresh branch for gdr make-patches

This tests that #907208 is fixed.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gdr-diverge-nmu-dgit: Insist on gdr-generated patches

Now dgit runs git-debrebase make-patches, with the result that the
patch commit in this conflict will be made by gdr.

This is what we want, so now that we have it, insist on it.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Honour DGIT_TEST_RUN_PFX env var

One might set this to `eatmydata', for example.

(It's not a good idea to wrap run-all or the Makefile in eatmydata,
because you actually want the stamp files etc. at least.)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Replace branch_is_gdr with a history walker

The debrebase-last approach is fundamentally wrong, because, for
example, cloning a stitched laundered gdr branch from a git server
does not establish debrebase-last, so dgit would make patches
itself (slowly).

So, instead, use git-debrebase make-patches in situations where it
will succeed.  We could just run it but that's slower, and it is a bit
awkward because we have to consider whether gdr is installed.  So
provide our own implementation.  It can be simpler because it only has
to handle the easy cases;

On testing of the new algorithm: we want to be confident that this
doesn't misfire; otherwise dgit users could be inconvenienced.  And we
want it to work for gdr users of course.

We can analyse the test coverage of the logic in branch_is_gdr
by running the whole test suite and then comparing this:

  cat tests/tmp/*.log |perl -ne 'next unless s/^branch_is_gdr  \w+ //; print' |sort -u |less

with this:

  git-grep 'branch_is_gdr  ' | perl -pe 's/^^dgit:\s+//' |sort |less

Noting that we should ideally have one each of all the gdr kinds we
try to recognise.  Currently that shows that we are missing only:

  gdr merged-breakwater
  unmarked BreakwaterStart YES

Closes: #907208.
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gdr-diverge-nmu-dgit: Tolerate gdr-generated patches

We are about to extend the set of situations where dgit runs
git-debrebase make-patches, with the result that the patch commit in
this conflict will be made by gdr.

This is good, so allow it.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Introduce get_tree_of_commit

NFC

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Do not abandon quilt fixup at gdr split commits

One of these might well be in the delta queue somehow.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Make all commit annotations have a COMMIT-TYPE

Omitting this sometimes turns out to have a mistake.  We are going to
want to distinguish at least split from other commits.  Add a
COMMIT-TYPE to the others as future-proofing.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: lib-gdr: Check that we made patches with gdr

Conveniently, the places where we don't want this, and are
deliberately looking for dgit-self-made patches, are gdr-makepatches7
and mix-it in gdr-subcommands; there, we don't push, so don't end up
checking t-gdr-made-patches.

We also add this check to the appropriate place in gdr-subcommands.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test-suite: lib-gdr: Be more defensive about unexpected states/args

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase(5): Warn against renaming branch while unstitched

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Do not try split brain gdr make-patches

Print a helpful error message instead.

Closes: #906908.
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: test suite: gdr-subcommands: test forget

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: forget-was-ever-debrebase: New subcommand

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gdr-import-nostitch: Test -fseries-comments

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: convert-from-*: snag on discarding comments in series

series files can contain #-comments.  The conversion process will
throw them away, so complain about that.

Closes: #907198.
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Make example_1.1 contain a patch with Name and Topic

This triggers #907202 and #907206.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: classify: Do not misclassify patches in subdirs

The old get_differs algorithm would recurse in debian/patches.
The new one failed to do that.  Fix that.

Closes: #907206.
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: trees_diff_walk: Support recursion

No-one specifies this yet, so NFC.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Fix get_tree's memoisation wrt $recurse

$recurse needs to be part of the key to the memo hash!

Luckily, there is only one caller which calls with $recurse==1,
check_series_has_all_patches.  It ends up mishandles this situation by
complaining about the directory names, as if they were unused patches.

And after that no-one needs to call get_tree.

Closes: #907202.
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: trees_diff_walk: Refactor $all to \%opts

No callers pass this so NFC.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 6.10

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: fianlise 6.9

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Remove unsubstituted $changesfile from message

Closes:#906787.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Document docs changes

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit-maint-debrebase(7): Add crossreference to "ILLEGAL OPERATIONS"

There are three items under "ILLEGAL OPERATIONS":

1. We already refer to "OTHER MERGES".

2. This commit adds a crossreference to the second item.

3. The information under "Editing debian/patches" is already basically
   included in dgit-maint-debrebase(7).

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

dgit-maint-debrebase(7): Rewrite "Converting an existing package"

Closes:#905573

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

test suite: Update debian/tests/control following dependency fix.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: do not crash on push of a new gdr package.  Closes:#906784.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 6.9

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Fix dependencies of new gdr-merge-conflicts test.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 6.8

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Finalise 6.7

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Reorganise and tidy up the current entry

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: quilt fixup: Maybe mention gitattributes

These are a potential cause of quilt linearisation failure.

Closes:#906199.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit(1), dgit(7): Document that we do not suppress export attributes

Ie, ones which affect git-archive.  This is related to #906199.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: quilt fixup errors: Refactor and reword very slightly

Make this two calls to push so we can put things in between.
Also remove `to specify'; now the message fits in 80 columns.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: quilt fixup: Do not suggest quilt modes if we see gdr

If `stopped at' is a git-debrebase commit, then a non-default quilt
mode is probably not the answer.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: quilt fiuxp: linearisation failure: Add a blank line

The `stopped at' commit report can be long, if there are several, or
complicated, commits.  Add a blank line.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: quilt fixup: @failsuggestion: Include a machine-readable reason

Each entry is now an arrayref.  This will let quilt_fiuxp filter them
later.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit(7): Mention bad origs as possible cause of quilt fixup failure

And document our error message changes.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: quilt fixup error messages: Change hint emphasis

* No longer hint --quilt=smash or dpkg source --commit.  These are
  rarely the right thing.

* Hint about bad origs, instead.  This is fairly common.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Remove a spurious space

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gdr-unprocessable-hints: New test

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Provide hints for unprocessable commits

Depending on the apparent branch ffq state.  Closes:#905005.

In particular, if it doesn't look like this branch was ever used with
gdr, suggest that the user is on the wrong branch, or needs to use
convert-from-*.  Closes:#905279.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Introduce new `fail_unprocessable'

This is called when we want to call `fail' because we have found an
unprocessable commit.

NFC.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: convert-from-*: leave debrebase-last refs

To hint to everyone that this is now a gdr branch.  This will make
dgit push use gdr make-patches, for example.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: begin_convert_from: Break out $gdrlast_obj

NFC

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Snag on repeated conversions

If debrebase-last says we are ahead of some previous gdr work, it is
carrying on is unlikely to be helpful.  It would only be sensible if
the package is vacillating between branch formats, and this is
reflected and visible in the branch history.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Refuse to start conversion if ffq-prev exists

Trying to rerun git-debrebase convert-from-something, on an unstitched
gdr branch, would end in tears.  Forbid it utterly.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: record_gdrlast: Honour $oldvalue

NFC for existing callers.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Move ffq_prev_branchinfo up the file

NFC

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Break out begin_convert_from & complete_...

$gdrlastinfo is undef for now, and will be used in a moment.

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Break out record_gdrlast

NFC

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: get_tree: Check we got an object name

Otherwise the cache is dangerous.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Merge: Make a todo about gbp not round tripping

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Merges: Enable with command-line option instead of env var

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: New walk mechanism: Drop obsolete $opt_careful

NFC

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Merge: Put off some xxx's, making them MERGE-TODO

We are going to release with this machinery in this experimental
state.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Document merged-breakwater metadata tag

This is generated only by the experimental merge machinery and is
subject to possible future change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Merge: Remove some xxx's

git-debrebase(5) already has ILLEGAL OPERATIONS (which forbids many
git-rebase operations) and also of course LEGAL OPERATIONS.

The limitations on what kind of merges we can resolve and how will be
documented when we document the merge machinery properly.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Document changes to convert-from-gbp messages.

This is part of #906641.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: convert-from-gbp: Check separately for $dtag

This allows us to print a more sensible error message, Previously
git would print a fatal error, after which we carry on.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: convert-from-gbp: Add a missing newline to a die

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: convert-from-gbp: handle missing maintainer tag

Look for / in $mtag, not in $mtag_pat.  Obviously it will be in the
latter.  The result, before, is that we would look for a completely
bogus $dtag and print daft error messages.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gdr-import-nostitch: New test

Generate this situation and check it's OK.  Also check that the
messages contain the expected hint.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: convert-from-gbp; on stitch failure, clarify message

Clarify what the preceding failure messages mean, and what the
consequences are.

This is part of #906641.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gdr-convert-gbp: Break the start out into lib-gdr

This will allow us to test other things to do with this conversion
without making tests/setup/gdr-convert-gbp more complicated.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: convert-from-gbp: Print a success message.

This is part of #906641.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Add close note for #905400 to changelog entry for 6.5.

The commit
  git-debrebase: Properly reject bare dgit dsc imports.
fixed #905400.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gdr-convert-gbp: test snag on unused patches

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: convert-from-*: Snag on unused patches

Patches in d/patches which are not in series will be deleted, and not
represented in the output.

So call them a snag.

Closes:#904997.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: get_tree: Support $precheck and $recurse

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gdr-subcommands: Test raw conversion from gbp

Make a branch which strips off the pseudomerge generated by gdr to
stitch the dgit history, and then run some tests on that.

These new tests detect #905975.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gdr-subcommands: Use $peel, not $work, for a few checks

The definition of P says
    P        result is add-patches, remaining letters apply to result~

But this was not hnoured by N n Z.  Honour it.

No change with existing tests, which do not follow P with any of
those.  But, this will be important in a moment,

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gdr-convert-gbp: Save converted-from-gbp

This new branch will be useful in a moment.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: On rebase, always save ffq-prev

Even if we were both stitched and laundered.

Closes:#905975.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: `prepush' is now a silent no-op if unstitched.

This is more friendly.  Update the docs accordingly.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Add perf numbers

This is from tests on my laptop with the branch the Debian kernel
folks shared with me, ie c6eb63c6ed0694b0d22fbe5aaff953d209a3fc4e.

                                     avoid        new
   runtimes in [s]         before    read-tree   get_differs

  git-debrebase             78.574    3.699       1.419
  git-debrebase status       3.359    unchanged   0.818

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Rewrite get_differs in terms of trees_diff_walk

As noted, git-diff-tree is quite slow.  We can do much better, by
stepping across the two trees ourselves.

There should be no particularly significant change: the answers should
be the same for reasonable inputs.  Unreasonable inputs (particularly,
executable files etc.) may be handled better, even.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: get_tree: Support '' for nonexistent objects

This is going to be convenient in a moment.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: New get_tree machinery: Fix debugging

When we localise, we get a new variable, rather than localising the
binding imported from Debian::Dgit.  How annoying.

Also, add two new debug prints at level 3.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Fix comment documenting D_DEB_*

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Actually avoid read-tree in walk when not rewriting

See
   git-debrebase: Prep for avoid read-tree in walk when not rewriting
for the motivation.

Here we try to check that the commit we would make, if we were
actually calling read-tree, write-tree and hash-object, would be
identical to the one we are reusing.

In particular, we check that the upstream files, and the debian/
directory, have the contents we would feed to read-tree.  And we check
that the parent list is the same.

With these checks this new algorithm is, I think, safe enough to just
use always.   So set $opt_careful to 0.  We will abolish this
variable in a moment.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Provide new get_tree and trees_diff_walk

These are ways to avoid calling git-diff-tree, which is very slow even
with --name-only --no-renames and without recursion.  I think it is
doing all the work to make nice actual diffs, and then throwing them
away.

Two calls to git-ls-tree is about 20x faster than git-diff-tree
on the Linux kernel, for example.

No functional change in this commit, since no callers yet.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>