test suite: Use nproc(1) rather than Sys::CPU.

This is more portable and does not depend on libsys-cpu-perl being
installed.

Closes:888496.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Non-noop git fetch is not available with --dry-run, so fail

When fetch or push wants git fetch (other than in a situation where it
happes to be a noop) but --dry-run was specified, we would loop with a
false coplaint about git fetch.  False, because we didn't actually run
git fetch so of course it didn't do anything.

Instead, fail with an explanation.  Closes:#871317.

This does not occur, and we do not fail, if git-fetch would be a
no-op.  We can tell when this is the case, because we use
git-ls-remote too.  This leads to one suggestion for a workaround for
the user, which is to do a not --dry-run fetch first.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit, Dgit.pm: Clarify error framing a bit

Add a newline and "error:" before most error messages.  Thius makes
the output clearer, by distinguishing informational output (which
might or might not relate to any error which is occurring) from things
which are definitely wrong.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Improve message for discrepant quilt fixup

When quilt fixup fails because of discrepancies, print a git diff rune
which will show them.  Closes:#865446.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: quiltify_splitbrain: Pass $oldtiptree (o+d/p) too

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: test suite: quilt-singlepatch: Test that we can remove everything

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: commit_quilty_patch: Handle other file statuses too

In particular, R can occur if all Debian changes have been removed in
a package with single-debian-patch, in which case we would fail to
commit the patch queue removal.  I don't think dpkg-source will remove
files in other cases, so this is probably the only actually buggy
case.

I don't think the other letters are particularly likely but if they
occur they should be committed too.  I have deliberately excluded U
which should definitely not occur.

Closes:#893263.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit build: Better message when network is offline.

When build fails because the archive access fails, mention that this
was tried because --since-version was not specified.

Closes:#883340.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit(1): Mention under `dgit build' that it uses the network.

Part of the fix for #883340.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 5.2~

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 5.1

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: document changes so far

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: suppress gbp pq export output

Except in case of error.  Closes:#901809.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Dgit.pm: Move shell_cmd from dgit

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Trigger on indirect dependencies

As discussed on debian-ci.  Bug report to request documentation for
this in the autopkgtest spec is TBD.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

control file: Fix a typo in the git-debrebase description

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gitattributes: check that dotfiles work properly too

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gitattributes: Test that old macros are updated

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gitattributes: Rename a sums file to the right name

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: gitattributes: Defuse working-tree-encoding

* Add -working-tree-encoding to $negate_harmful_gitattrs.

* Add new arrangements for updating an existing dgit-defuse-attrs
  macro definition:
    - is_gitattrs: say whether the macro is up to date
    - setup_gitattrs: update an existing out-of-date macro
    - minor message changes
    - document the new behaviour

One consequence is that we have had to breach the promise about how to
stop future dgit setup-new-tree runs messing with this: it is now
necessary to disapply the macro, rather than redefine it.

Closes:#901900.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: gitattributes: Minor refactoring to prep for adding new squash

* is_gitattrs_setup: Break out $gitattrs_ourmacro_re
* is_gitattrs_setup: Reorganise very slightly
* is_gitattrs_setup: Add a debug print
* setup_gitattrs: Introduce $new
* setup_gitattrs: Add a blank line

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gitattributes: Do not try to set working-tree-encoding to true

Part of the fix to #901900.  This suppresses the sort-of-spurious test
failure which occurs because the working-tree-encoding attribute is
discovered and then set to an inappropriate value.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gitattributes: Test working-tree-encoding suppression

Put a UTF-8 BOM in the middle of the test files, and some gitattribute
settings for working-tree-encoding.  `ISO-8859-1' produces file
corruption.  `UTF-16' produces an error since it doesn't specify BE or
LE, and the file lacks a BOM at the start.

Part of the fix to #901900.  This introduces the desired test failure.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: new-upstream: fix ff check handling of multi-piece upstreams

Multi-piece combine commits may have n or n+1 parents, depending on
whether the previous new-upstream invocation was snag-free.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase manpages: Fix typos and etc.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 5.1~

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 5.0

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Delete some obsolete NOTES

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: git-debrebase: Rename gdr-newupstream-v0 test

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Rename new-upstream-v0 command to new-upstream

This UI is pretty good now, I think, for single-piece upstreams.

For multi-piece ones, which are quite tricky anyway, it is tolerable,
and we it doesn't steal much of the available syntax space.  So we
don't need the -v0 any more.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit(1): Refer to dgit-maint-debrebase

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit(1): Abbreviate some of the tutorial descriptions

This will avoid linewrap from `man dgit' in an 80-column terminal,
when we introduce a reference to dgit-maint-debrebase.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Merge branch 'gdr-manpages' into wip.rebase

dgit-maint-debrebase(7): Add cross-refs to git-debrebase(1) and (5)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: add dgit-maint-debrebase.7 to .gitignore

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: git-debrebase: Do some tests without dgit installed

We provide a new psuedo-dependency NO-DGIT which throws away the usual
default dependency list.

We abolish NO-DEFAULT which could not be useful because the tests
pretty much all depend on setup/gnupg which (currently, as a
workaround for the gnupg2 races) requires chiark-utils-bin for
with-lock-ex.

There are two tests which don't actually need any interaction with the
archive.  Mark them accordingly.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Split into its own package

* Provide and use Debian::Dgit::GDR which does the same for @INC as
  Debian::Dgit:Infra, only for GDR.  It's not *entirely* similar as
  its doc comment talks about ExitStatus too.

* In Makefile, split off lots of GDR_* variables and the
  two targets install-gdr and installdirs-gdr
  We take no care that only the required manpages are built for
  each install target; instead, they all go in MANPAGES and `all'.

* Add a control file stanza.  git-debrebase takes over the dependenc
  on libfile-fnmatch-perl.  We Recommend dgit and gbp.  (Even though
  if you're working with neither the Debian archive nor `3.0 (quilt)',
  you need neither.)

* In rules, add specpkg_install_gdr, to run the Makefile machinery.

* In the test dependencies, GDR now means to ask for git-debrebase and
  also git-buildpackage (for make-patches).  libfile-fnmatch-perl is
  handled via the dependencies of git-debrebase.deb.

* Add a .gitignore for the staging area in debian/

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: git-debrebase: Use gdr-convert-gbp-noarchive

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gdr-convert-gbp-noarchive: new setup kind

Introduce $ifnoarchive in gdr-convert-gbp, which allows us to run it
without any of the operations which touch the simulated archive.

This is faster.  Also in the future after we split the packages this
will make it possible to run this new setup without having dgit or the
archive management tools installed.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Get perl -I for source tree right

We wanted the Dgit.pm in the source tree (ie, the test suite's).
Previously this would work by accident, because Dgit.pm would be
installed in /usr.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: t-setup-done: get $import right setup using t-chain-test

We need to honour DGIT_TEST_NAME, rather than just using $0, in that
case.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: divergence checking: Do not mind if a ref we are checking does not exist

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: git-debrebase: provide GDR "macro" for dependencies

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: enumerate-tests: Refactor dependencies, support NO-DEFAULT

This will allow more flexibility in a moment.

No users of NO-DEFAULT yet, and no functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

debian/rules: Refactor dgit-infrastructure perl arrangements

We are going to want to do this for another package too.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Document gdr, and bump to 5.0~

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

documentation: Discuss gdr make-patches vs dgit quilt-fixup, etc.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Use git-deborig style upstream commitish finding

As discussed in email with Sean.

We could use git-deborig --just-print --version= but its error
handling is rather troublesome.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Add some plans for future, in a comment

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gdr-edits: Rely on dgit interop, and test that it DTRT

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: test suite: t-gdr-good: Introduce etypes, a sequence

Rather than having the next type recorded in netype, record it in
etypes.  This is more orthogonal, and will allow more subtle
specifications of what is expected.

No functional change for now.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Interoperate with git-debrebase, automatically

* Recognise gdr branches, by the presence of corresponding
  ffq-prev or debrebase-last refs.

* Before trying to do our own multi-patch linear quilt fixup, on a gdr
  branch, see if gdr can do it.  If so that is faster and better.

* If our branch is an unstitched gdr branch, which would be ff from
  the archive if we were to stitch it, then do so.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Prepare for calling git-debrebase

Provide the --git-debrebase= option, and pass it in the test suite.
Nothing uses this yet.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Dgit.pm: gdr_ffq_prev_branchinfo: move from git-debrebase

dgit is going to want this too, for interoperability.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: make-patched: Provide --quiet-would-amend

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: test suite: gdr-subcommands: provide X checkletter

No user yet.  I wanted this for a test which actually isn't going to
work like that, but I thought I would commit it anyway.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: test suite: gdr-subcommands: use xopts for N*

This stylistic change improves the orthogonality.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: test suite: gdr-subcommands: split off XX

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: test suite: gdr-subcommands: honour new $xopts

This allows the introduction of chaining checkletters which modify
subsequent attempts.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: test suite: gdr-subcommands: document XX

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Change exit status for patch amendments, to 7

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: make_patches; Move get_differs into cmd_...

We want to make the error handling dependent on options, etc.  Moving
the differs analysis into cmd_make_patches avoids having a complex
calling convention for make_patches.

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Dgit.pm: Provide failmsg, previously buried in fail

Most of `fail' is error message construction.  This change allows
callers to do something other with the message, than pass it to die.

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: test suite: run gdr status

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Provide status subcommand

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: keycommits: Now $fatal hook

This allows a caller to avoid having keycommits crash on
unrecognisable stuff.  This will be used in a moment.

No functional change, because the default implementation calls fail
the same way it did before.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: keycommits abolish use of last etc.

Previously, presumably, this little fragment was within an anonymous
sub.  Now it isn't any more, and we can simply return.

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: keycommits: Pass $cl to callbacks

No overall functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: keycommits: Move up $cl (nfc)

We are going to want this within the scope of the anon sub $x.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Fix multi-orig handling

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Fix snagging

* Rename $snags_checked to $snags_summarised and make it a counter
  of the snags we have summarised in snags_maybe_bail.
* Introduce all_snags_summarised, which uses arithmetic to see if
  we have had new snags since the most recent snags_maybe_bail,
* Use this in run_deferred_updates rather than the existing
  approach; this means we can crash if there were new snags for
  which we should have bailed.
* Make snags_maybe_bail not produce a redundant identical summary.
* Initialise counters to 0 so arithmetic does not give undef warnings.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Some notes on anchor parent ordering

This is the result of some playing around with gitk.  Notably,
for example,
  git-branch -f t dgit/stretch-security && ~/things/Dgit/play/swap-pseudomerges -f RELEASE-4.8.0..t
where dgit/stretch-security is c9517ee45383271e90f40b7a9773d1e6fefaeca7
and swap-pseudomerges is something like this

  #!/bin/sh
  #
  # runs
  #   git-filter-branch --some-filters "$@"

  set -e

  git-filter-branch --parent-filter '
  case "$(git cat-file commit $GIT_COMMIT)" in
  *"git-debrebase new-upstream breakwater"*|*"git-debrebase anchor"*)
  perl -ne '\''
  chomp;
  print join " ", reverse split / +(?=-p )/;
  print "\n";
  '\''
  ;;
  *)
  cat
  esac
  ' "$@"

or
  ....
*"git-debrebase new-upstream breakwater"*|*"git-debrebase anchor"*)
  ....

and then runes like

  gitk --first-parent dgit/stretch-security &
  gitk dgit/stretch-security -- ':/' ':!/debian' &

etc.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Commentary: explain why pseudomerges are the way round they are

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Bail if single-debian-patch needs committing with --quilt=nofix

Also, if the user invokes dgit push with uncommitted debian/patches/.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Break out quiltify_nofix_bail (nfc)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Change badusage exit status to 8

This matches dgit.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

exit status: Document in changelog.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

exit status: Fix dgit-badcommit-fixup to use Debian::Dgit::ExitStatus

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

exit status: Fix git-debrebase to use Debian::Dgit::ExitStatus

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

exit status: Fix dgit to use Debian::Dgit::ExitStatus

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

exit status: Introdude Debian::Dgit::ExitStatus

No callers yet.

Also, no version for dgit-infrastructure.  The one perl script in
dgit-infrastructure that cares a lot about its exit status
(dgit-repos-policy-debian) already has an END { } and uses _exit,
which is a bit like our approach here, but simpler.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Implement make-patches

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: test suite: gdr-subcommands: Add some t-git-next-date

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Run gbp pq export only from (perhaps private) laundered branch

When the branch is stitched, gbp pq will descend into both sides of
the pseudomerge.  This will duplicate patches and be generally bad.

So instead, use walk to generate, privately, a laundered version of
HEAD, to use to generate the patches.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Break out make_patches_staged (nfc)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: conclude, quick: uncleanness after laundry is fatal

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Defend against missing snags_maybe_bail calls

Almost the only thing gdr does as an output is to update HEAD and
other refs.  That's done with run_deferred_updates.  So, there,
check that (i) we have called snags_maybe_bail at least once;
(ii) we have not had any snags since.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Add a several missing snags_maybe_bail calls

That these were missing is a bit alarming.  I am about to add some
more defences against missing this check.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Fix a message to use "" properly

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: test suite: gdr-subcommands: Better error handling/debugging

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Do not mind stitching an unlaundered branch

 <Diziet> I discover that stitch treats non-launderedness as a snag.
 <Diziet> This is not quite compatible with these newfangled
          push-your-unlaundered-stuff workflows.
 <Diziet> It would be possible to make one of prepush or stitch
          (currently synonyms) behave differently in this respect.

 <spwhitton> do you know why stitch treats non-launderedness as a snag?
 <spwhitton> given that we expect [most people] to use `git debrebase
          conclude`, which launders, and never invoke `git debrebase
          stitch` explicitly, it would be okay to change that such
          that `git debrebase stitch` does not consider
          non-launderedness to be a snag.

 <Diziet> I think it does that just because I am the kind of person
          who thinks, when writing some routine, "what could I check
          here?" :-)
 <Diziet> I think you are perhaps right that it ought not to.
 <Diziet> "conclude" didn't exist then of course.

 <spwhitton> okay.  git-debrebase(1) could note "you probably want
          conclude because you probably want to launder"

 <Diziet> Mmmm.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: do_stitch: Introduce $unclean argument

Update all call sites.  No functional change just yet.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Provide "prepush"

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Provide and test "conclude"

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase(1): Fix to say that noop is always ok for "quick"

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase(1): Very minor clarification to "quick"

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: test suite: New test for subcommands

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Rename `conclude' to `quick'

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: comment: explain why tagged linear anchors are evil

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Only make finalised changelog entries

Unfortunately, finalised UNRELEASED has become the dominant convention
in Debian.  So conform to that.

Also this breaks a test, which assumed that the operation date did not
leak into the output tree.  But, sadly, now it does.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>