git-debrebase: Improve error message for convert-from-gbp upstream discrepancy

Closes:#905078.

I don't intend to print `small' diffs because I think that kind of
dwim is probably more annoying than helpful.

I don't intend to factor this kind of pattern out throughout because:
The messages for these commands are generally not the actual diff rune
dgit or gdr ran, for various reasons (notably that dgit/gdr often use
object hashes).  The result is that there is little duplication.

So I think the situation now is OK.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Improve error messages for bad options.

* GetOptions calls warn().  So we need a wrapper which disables
  or $SIG{__WARN__} (which prints a stack trace).
* Put the call to badusage in the wrapper.
* Change the messages to be clearer about what is meant.
* Add the program name to the badusage message.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: convert-from-gbp: Use same algorithm for finding upstream

The version is in the changelog, and thus upstream commitish can then
be found from as new-upstream.

This also means we now give the same error message, which is much
better than the one we had before.

Closes:#905062.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Break out resolve_upstream_version (nfc)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: new-upstream: Fix handling of epochs.

Dpkg::Version->version() is only the middle part.  ->epoch() doesn't
have the colon.  This is inconvenient enough that we convert it to a
string and use a regexp :-/.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gdr-unprocessable: new test

Test git-debrebase messages about unprocessable commits.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gdr-convert-gbp: Provide some unprocessable branches

We'll add a test which uses these, in a moment.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: new-upstream: Fix error message for new version with bad syntax

Calling
  new Dpkg::Version , check => 1
does not die or anything if it's wrong; it just returns undef.

The result is an error message about calling is_valid on an undefined
value.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: keycommits: Pass correct argument to fail

We got the array index wrong and would print HASH(0x55be4dba1f88) or
whatever.

Closes:#905003.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: walk: unprocessable: use fail, not die, and clean up

Make this message like the one printed by $x->($fatal, ...) in
keycommits.  That is more consistent and also nicer.  There is no need
for the line number etc. either.

This is mentioned in #905003.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 6.3~

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 6.2

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit-maint-debrebase(7): fix command to just launder

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

changelog: document manpage updates (gbp dch)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit-sponsorship(7): add references to push-source

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

dgit-maint-gbp(7): update references to --*-dirty

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

dgit-maint-gbp(7): discuss dgit.default.build-products-dir

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

manpages: add references to pbuilder and cowbuilder

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

manpages: alioth->salsa

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

dgit build: Warn if --build-products-dir is not `..'.

Closes:#904859.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit(1): Improve and correct --build-products-dir description.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Test dgit import-dsc --build-products-dir.

Add t-buildproductsdir-config to import-nonnative, which will do as a
test I think.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: lib-import-chk: Honour $bpd

No functional change because none of the import tests set this.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Remove // from inside conditionals

This idiom is terse, but quite confusing.  Instead, handle the undef
case explicitly.

While we are editing these:
 * Change two internal errors from die to confess
 * Fix a die in git-debrebase which ought to be badusage

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 6.2

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 6.1

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

changelog: close bug requesting pbuilder support

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

dgit(1): replace use of 'recalled'

Suggested-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

dgit(1): don't have pbuilder description refer to sbuild description

Suggested-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

test suite: add stubbed test for pbuilder subcommand

pbuilder requires root, but we want to be able to test the subcommand
outside of autopkgtest too.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

test suite: add t-pushed-good-check-changes

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

test suite: add t-push-included

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

debian: suggest sbuild | pbuilder | cowbuilder

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

dgit(1): document new pbuilder and cowbuilder subcommands

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

dgit: move cmd_sbuild

No functional change.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

dgit: new 'pbuilder' and 'cowbuilder' subcommands

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

dgit: remove dead $need_split_build_invocation code path

No functional change.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

dgit: factor out build_with_binary_builder()

The pbuilder and cowbuilder subcommands will reuse this code.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

dgit: fix a reference to parent directory

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

dgit: set $dscfn in build_or_push_prep_early()

No functional change.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

dgit: opts_opt_multi_cmd: New @$extra feature

Adjust the existing call sites.  NFC.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Update copyright years

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>
Acked-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 6.1

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 6.0, and document remaining things

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: categorise, and add a few missing things

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Abolish move_dsc_to_bpd

This subroutine only has one caller.  It also contains a bit of
duplication both within itself and compared to its caller - the calls
to mv and its error handling - which is annoying to remove as it is.

Instead, put its code into build_source, and break out a private
subroutine $mv for the rename.

No functional change other than to messages.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Add t-buildproductsdir-config to some sbuild tests

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Revert "dgit: Move up $sourcechanges"

This reverts commit 07437a3fe514da7990300b2fd594adff49685de5.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Move cmd_push_source down the file

Pure code motion.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Add a couple of t-buildproductsdir-config

Add t-buildproductsdir-config to all build-modes tests and to
push-source.  This will probably catch almost all of the ../
bugs (#863582).

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Use $bpd in several places where it is needed

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Provide a gbp.conf which sets --export-dir to $bpd

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Introduce $bpd variable (nfc)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite lib: add t-buildproductsdir-config

No users yet so NFC.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: push_source no longer cleans the tree.

This is no longer needed because we always build in a playtree.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: build_source mostly uses a playtree, not $maindir

In particular, it uses a playtree unless we are trying to include
uncommitted changes in the source package (which is not supported in
split brain mode).

This change allows build_source to respect $buildproductsdir.  It will
also enable, later, building a source package from HEAD, or arbitrary
git ref, when the working tree is dirty.

This change also requires modifying build_prep to only conditionally
clean $maindir.  It should clean $maindir only when it's $maindir that
we're packing into a source package, or when we are running any
builder in $maindir.

When we are using a playtree we must build the changes file there too,
because dpkg-genchanges cannot be persuaded to look for the .dsc
anywhere but ../.

Right now build-source still cleans the tree needlessly.

We need to change the test suite to not expect a clean unless running
a builder:

dgit now cleans the tree only when including uncommitted changes in a
source package, and when running a builder in $maindir.  The code in
lib-build-modes does not test the including of uncommitted changes in
a source package.  Thus, it should expect the tree to be cleaned only
when running a builder.

Also, we need to not expect a clean with sbuild, for the same reason.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Drop alwayssplit tests

Since
  dgit: make $need_split_build_invocation always true
this option has had no effect, so these tests are redundant.

And change the comment in dgit near the option parsing.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: quilt-splitbrains: Drop non-wgf dpm test of gbp package

Soon, dgit push-source will no longer clean the tree.  So this test
will be of no use.  Also, it will fail because the error message changes.

Delete it.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: build_prep: Provide it with $wantsrc

No functional change yet as it doesn't use it.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: build_source: drop call to build_prep

We want the callers to do this so they can control it better.  They
mostly already do.  In detail:

* cmd_push_source

  This one is most complicated.  It already calls prep_push, which
  does a variety of things which overlap with build_prep.  The things
  which are in build_prep but not in prep_push are:
   - notpushing (via build_prep_early), which seems to be a mistake
   - clean_tree(), which is needed, so we add it (for now...)
   - build_maybe_quilt_fixup, which is needed, so we add it
   - remove old changes, which is not actually needed

* cmd_build, cmd_gbp_build

  build_prep was already called by the if arm which didn't call
  build_source; so we can just move it up outside the if.  No
  functional change here.

* cmd_build_source, cmd_sbuild

  We need to replace the call.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: build_source: Drop call to build_prep_early

build_prep starts with a call to build_prep_early so this is redundant
(and confusing).

Effect is to sometimes no longer duplicate some effort.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Abolish build_source_for_push

This sub has only one caller and the indirection is not really
helpful IMO.

NFC.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: build_source_for_push does not try to unapply patches

build_source_for_push should now always build in a worktree, so
patches should never be applied dirtily to $maindir.  Die if it turns
out they were.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

dgit: Move up $sourcechanges

This is not really ideal, but cmd_push_source needs it.  For now do it
like this.  I may move cmd_push_source further down the file, later,
and then we can revert this.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Support run-all -p, DGIT_TESTS_PROGRESSIVE

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Drop dependency arrangement for tests/tmp

run-all makes the directory now adays so this is not needed.  It's
going to be harmful for --progressive, since it would make make want
to rerun things every time tests/tmp/ is updated.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: run-all: Minor refactoring etc.

* Makefile: Move the DGIT_TEST_RETRY_COUNT if/else into the rule.
  This removes a small amount of duplication.

* run-all: Document the option to pass explicitly list of scripts.

* run-all: Add an option parser, which currently supports no options.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Honour DGIT_SCHROOT_CHROOT to set the schroot to use for the sbuild tests.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: make $need_split_build_invocation always true

dgit will always build the source package.

One minor consequence is that dgit will usualliy generate
*_multi.changes rather than *_$arch.changes, so we need to update the
one test that makes the contrary assumption.

Bump the dgit major version number as this is quite a significant
change in implementation and also a behavioural change.

This change makes a lot of code dead.  Removing that is left as a task
for the future.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Move WANTSRC_* definitions up the file

We are going to want these a bit earlier.  NFC.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Disregard *_multi.changes for .changes ambiguity purposes

The changes file ambiguity problem arises because dgit does not know
what architecture changes file the build is going to generate.  (To
know that it would have to delve even more into the command line
options the user is passing through dgit to the builder.)

dgit --always-split-build generally makes a _multi.changes file,
because it merges source changes with binaries from the build.  We are
going to make --always-split-build the only way things are done.  This
would result in lots more situations where --rm-old-changes is needed.

However, actually, we can assume that the builder does not generate a
*_multi.changes.  That will allow us to spot the builder-generated
changes file even if there is already a dgit-generated *_multi.changes
file from a previous build.

So: disregard _multi.changes, both when pre-checking for confusing
files, and when actually figuring out what the build produced.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: push-source dies if user tried to include uncommitted changes

This does not make sense because you cannot dgit push uncommitted
changes.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

dgit: don't let the user pass --include-dirty in split brain mode

It is not supported.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Rename --ignore-dirty to --include-dirty

Leaving the old name supported as a deprecated alias.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: linkorigs: Fix error handling when bpd missing

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: bpd_abs(): Fix return value when bpd is absolute.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: massage_dbp_args: Make names for $wantsrc values

   0 - source will NOT need to be built separately by caller
  +1 - source will need to be built separately by caller
  +2 - source will need to be built separately by caller AND

So:

   0 => WANTSRC_BUILDER
  +1 => WANTSRC_SOURCE | WANTSRC_BUILDER
  +2 => WANTSRC_SOURCE

No functional change except to debugging output.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Document some recent changes

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: rename $ignoredirty -> $includedirty

This better reflects the effects that the option has.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

dgit: rename two quilt_fixup_ functions

They are useful for doing things other than quilt fixup.

No functional change.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

dgit: actually respect $buildproductsdir in many places

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Provide $bpd_glob

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: add dgit.default.build-products-dir git configuration key

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

messages: Be a lot more friendly about NEW in particular

And also add a couple of `please's.  Closes:#904448.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

apt-get method: when apt does not update release files, print hint

Unconditionally print hint about noatime.  This problem seems to occur
quite rarely (noatime is out of fashion nowadays, and I haven't seen
any other causes) so it doesn't seem worth fiddling about writing code
to determine the mount options.  Instead, print a hint and let the
user investigate.

Closes:#851873.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: unset VISUAL, which interferes.

Closes:#904308.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 5.11

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 5.10

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

infrastructure: Run git gc --auto before mirroring.  Closes:#841414.

This would not have avoided the trouble reported here
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=841414#74

Empirically, I managed to work around the problem by running `git gc'
(rather than `git gc --auto') and then rsyncing the result to the
mirror.  I will request that DSA increase the RAM (or swap) size on
the relevant host.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Drop a couple of useless test log output lines.

We don't need these (((((((((( )))))))))) now that it's just one line
inside them.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: rework for merging experimental tests into master

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 5.9+exp4

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: tar up the artifacts, and name the tarball after the test

Naming the output after the test works around #903696.

tarring the whole lot up turns out to make it a lot smaller.
Before:
    $ du -sk ; du -sk --apparent-size
    1640    .
    853     .
After:
    236     .
    235     .

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: chmod a+r $AUTOPKGTEST explicitly.

test suite: Do not save non-world-readable artifacts

Save things to $AUTOPKGTEST_ARTIFACTS without preserving access
control information.

Works around #903680.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 5.9+exp1

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Save much stuff to $AUTOPKGTEST_ARTIFACTS.

We take some care to make this of reasonable size.  In particular,
we delete raw tarballs (typically, IMPORT) and .git's sample hooks.

And, we git gc the git trees.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Introduce playtree-save-refs

Arranges for all the refs in each playtree to be saved in the parent's
ref namespace.

This is going to be used when running git-gc when saving artifacts.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Drop t-ok from setup/*

t-ok is going to save artifacts, which we don't want these to do.

It is not actually needed.  All it does that is useful is the fsck,
which the actual tests will do.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Use  dch -r -D sid ''  not  dch -r sid.

Bizarrely,
   dch -r [-D suite]
runs $EDITOR but
   dch -r [-D suite] weevils
ignores `weevils' but does not run $EDITOR.  This is #903598.

Apparently, this is well known, and convention is to pass ''.
See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=903598#8.

So use that.

Closes:#903441.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>