dgit: Provide $bpd_glob

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: add dgit.default.build-products-dir git configuration key

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

messages: Be a lot more friendly about NEW in particular

And also add a couple of `please's.  Closes:#904448.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

apt-get method: when apt does not update release files, print hint

Unconditionally print hint about noatime.  This problem seems to occur
quite rarely (noatime is out of fashion nowadays, and I haven't seen
any other causes) so it doesn't seem worth fiddling about writing code
to determine the mount options.  Instead, print a hint and let the
user investigate.

Closes:#851873.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: unset VISUAL, which interferes.

Closes:#904308.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 5.11

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 5.10

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

infrastructure: Run git gc --auto before mirroring.  Closes:#841414.

This would not have avoided the trouble reported here
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=841414#74

Empirically, I managed to work around the problem by running `git gc'
(rather than `git gc --auto') and then rsyncing the result to the
mirror.  I will request that DSA increase the RAM (or swap) size on
the relevant host.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Drop a couple of useless test log output lines.

We don't need these (((((((((( )))))))))) now that it's just one line
inside them.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: rework for merging experimental tests into master

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 5.9+exp4

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: tar up the artifacts, and name the tarball after the test

Naming the output after the test works around #903696.

tarring the whole lot up turns out to make it a lot smaller.
Before:
    $ du -sk ; du -sk --apparent-size
    1640    .
    853     .
After:
    236     .
    235     .

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: chmod a+r $AUTOPKGTEST explicitly.

test suite: Do not save non-world-readable artifacts

Save things to $AUTOPKGTEST_ARTIFACTS without preserving access
control information.

Works around #903680.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 5.9+exp1

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Save much stuff to $AUTOPKGTEST_ARTIFACTS.

We take some care to make this of reasonable size.  In particular,
we delete raw tarballs (typically, IMPORT) and .git's sample hooks.

And, we git gc the git trees.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Introduce playtree-save-refs

Arranges for all the refs in each playtree to be saved in the parent's
ref namespace.

This is going to be used when running git-gc when saving artifacts.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Drop t-ok from setup/*

t-ok is going to save artifacts, which we don't want these to do.

It is not actually needed.  All it does that is useful is the fsck,
which the actual tests will do.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Use  dch -r -D sid ''  not  dch -r sid.

Bizarrely,
   dch -r [-D suite]
runs $EDITOR but
   dch -r [-D suite] weevils
ignores `weevils' but does not run $EDITOR.  This is #903598.

Apparently, this is well known, and convention is to pass ''.
See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=903598#8.

So use that.

Closes:#903441.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Factor out t-dch-commit-r

The pattern `dch -r sid' is actually wrong and we will want to change
it.  See #903441 and #903598.  For future-proofing, factor this out
before we fix it.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 5.9+exp1

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 5.9

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase(5): Suggest `gitk -- :.' too

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase(5): Document best gitk options.  Closes:#901927.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase(5): Add new subsection DISCUSSION

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

NOTES.dgit-downstream-dsc.7.pod: WIP for #810829

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit-downstream-dsc(7): New manpage.

Closes:#842643,#851194.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Document reorganisations

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gdr-convert-gbp: drop --overwrite

With #903132 properly fixed, this is no longer needed.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase convert-from-gbp: Add ffq check

Check that the result will not count as having diverged.  This will
usually turn failures to make the ff pseudomerge into -fdiverged.

Related to #903132.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase convert-from-gbp: Make it ff of dgit view, if we can.

Look for dgit-generated tags so we can usually make the new branch ff
of the dgit view.

Specifically, we fish the most recent released version out of
debian/changelog and look for a DISTRO/VSN tag and a corresponding
archive/DISTRO/VSN tag.  If this all looks good, we declare ff of
archive/.

This should avoid the need for dgit push --overwrite in most cases.

We do not check the DISTRO name.  In principle dgit could find the
distro name from the suite name, but that makes the software layering
quite complicated.  (For example, we would have to provide a way to
tell gdr what program to run for dgit, for the benefit of the test
suite.)  I think the current approach is good enough; there are not
likely to be any tag pairs that look like the right kind of thing but
are in fact something else entirely, so we're not likely to
accidentally trash some wanted changes.

Closes:#903132.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Set $|

This will avoid pepeering everything with calls to flush.  Without it
some messages from ffq_check, say, can arrive after the stderr output
from a subsequent snags_maybe_bail, which is quite confusing.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Provide snags_maybe_bail_early

No callers yet.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Replace `die badusage' with `badusage'.

badusage contains a call to finish.  So the die is redundant and never
gets called.

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: ffq_check: Move defaults for $ff and $notff into sub

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Dgit.pm: Move debinatag_maintview from dgit

git-debrebase is about to want this for convert-from-gbp dgit split
brain fixup.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Dgit.pm: Break out parsechangelog_loop from dgit

git-debrebase is going to want this.

No functional change except to some debugging messages.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Dgit.pm: Move parsecontrol etc. out of dgit

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Fix a typo in a comment

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Break out ff_check (for reuse by convert-from-gbp)

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: record_ffq_prev_deferred: Move get_head();

No functional change.  This will make ff check easier to split out.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 5.9

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 5.8

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: Improve grammar if one blocking snag.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gdr-newupstream: Test adding revision

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase new-upstream: Add a -1 revision if the user didn't supply one.

Closes:#903127.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: status: Reorganise commit reporting.

Replace the references to @_ with a hash.  Using a hash is much less
confusing than all this numerical indexing.  Also, \@_ is wrong
because there is only one @_, so by the time it is dereference, the
commit being printed is the anchor.

This is why in #903131 we see

      branch is unlaundered
      found mixed upstream/packaging commit (7badba627162337c7057002f37e9a1a593d07d08)
        abcc6260 Update to upstream 2.9.7

which appears to sugggest that the anchor is a mixed commit.  Actually
the mixed commit is 7badba627162 but the subsequent message is based
on @_ which has different information in it by then.

And, set $kcmsg from the main message, not including the commit
information.  We are the more sophisticated caller mentioned in the
previous commmit.

Closes: #903131.
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debrebase: keycommits: Callbacks get separate $mainwhy

Separate out the extra contextual info, naming the hash of the commit
being complained about, into a separate argument to $x->().  This
allows us to pass the message without that as an additional argument
to the callbacks.

This way we can still pass \&snag as a callback but other callers can
do something more sophisticated involving $cl.

No functional change for any existing callers.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: makepatches7: Add a nasty .gitignore

Prior to the fix for #903130 this prevents both dgit and gdr from
making patches properly.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit, git-debrebase: Properly make patches with nasty .gitignores.

Specifically:

 * When running git status --porcelain, pass --ignored so it
   lists ignored files, and handle the ! output that it then
   produces.

 * Add missing -f to a few git add invocations.

I have done some greps ('git.*add', 'git.*status', 'porcelain', etc.)
to try to find other missed cases and none turned up.

git diff is OK because normally we pass two treeish arguments, in
which case the ignores are ignored by git diff.  When we are asking it
to look at the working tree, we are expecting it to ignore untracked
files (whether ignored or not), and diff tracked ones, which is what
it does.

Closes:#903130.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Document dgit(1) build unscrambling

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
squash! changelog: Document manpage fix

dgit(1): Unscramble push and push-source descriptions

These got mixed up in a recent commit.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

changelog: start 5.8

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 5.7

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: sourceonlypolicy: New test

Test the fix for #801435.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Check that entirely-new uploads to Debian are not source-only-uploads.

Avoids the user pushing things that will inevitably be REJECTed.

We end up introducing some new config and force options to support
this.

We do not reuse test_source_only_changes, as it doesn't have quite the
right shape (in particular, it sometimes blithers on stdout).  Also
arguably it is better to look, specifically, for .debs, for the
purpose of this test.

This requires a new archive protocol method.  We implement it for
ftpmasterapi (where it is actualliy needed) and dummycatapi (where we
need it for tests.)  Implementing it for madisonish methods would be
easy.  For aptget, it would probably involve iterating over suites,
and not be adviseable.

Closes:#801435.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Test dput-ng compatibility.

Add dput-ng as a dependency to trustingpolicy-replay.

This is the test which went wrong with dput-ng before.  It also has a
number of pushes, so it will do.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Fix trustingpolicy-replay & dput-ng.

Specifically:

 * Only dput-ng actually complains about the existence of .upload
   files.  dput itself just says there is nothing to do and calls that
   success.

 * The call to t-rm-dput-dropping was in the wrong place.  t-commit
   bumps the version to 1.1 but it's 1.0 that needs removing.

Closes:#903007.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: checkout: Check we end up on the right branch each time

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: checkout: Do not leave detached HEAD!

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Always pass LC_COLLATE=C to sort(1).

Specifically, by replacing every invocation of sort with the new
function t-sort.  The one case where we already took care of this, we
change LC_ALL to LC_COLLATE, which should be sufficient.

Closes:#903006.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit(7): Mention git-debrebase and gbp pq alongside git-dpm

in the comment about handling patch stacks.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: checkout: new test

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit(1): checkout: Document when this acceses the network

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: checkout: New command.

Closes:#878443.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: fetch: Introduce new dofetch()

This actually does all that is needed.  It replaces two now-identical
calling patterns.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: fetch: Reorganise cmd_fetch

This slightly odd construction is actually how the return value from
fork_for_multisuite should be handled.  It now looks much more like
the calling pattern in pull().

No functional change: if $multi_fetched is falseish, we call
fetch_one, and then, no matter what, we finish 0.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: fetch: Slightly refactor pull()

When for_for_multisuite returns '0', meaning we are a child and should
do one suite, use finish 0 rather than returning, after doing the
fetch.

This has no functional change because the one call site is at the end
of cmd_pull, after which we finish 0 anyway.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: fetch: Rename fetch_one from fetch (nfc)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit(1): Document as-yet-unwritten `dgit checkout'

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: update-vcs-git: Honour --package properly.

Specifically, use package_from_d_control rather than open-coding
a substandard implementation.

The consequent abolition of $sourcep does mean that if the user
specifies the suite `.', and does not specify -p, we needlessly parse
debian/control twice.  This doesn't really matter.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Break out package_from_d_control (nfc)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 5.7

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 5.6

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Document empirical effectiveness of new gnupg workaround

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: manpages-format: Test -Tps too

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: manpages-format: work properly when no expected warnings

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: manpages-format: Remove some pointless redirects

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: manpages-format: Prep for other formats (nfc)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: manpages-format: New test

Test that manpages format with only expected roff warnings.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Merge remote-tracking branch 'dgit/dgit/experimental'

Conflicts:
debian/changelog

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gpg agent workaround: Fix bugs in log neatening.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gpg agent workaround: Neaten the logging.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Provide DGIT_TEST_BYHAND if appropriate

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gnupg workaround: Remove a sprurious $tmp

This is not set here, which is just as well because it would have been
quite wrong.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gpg-agent workaround: Send ps output to stderr (oops)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gpg-agent workaround: Cope if ps not found.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 5.5+exp5

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gpg-agent workaround: Kill unexpected leftover agents.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gpg-agent workaround: Add more debugging output.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Add bc to appropriate test Depends.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: 5.5+exp3 to placate version checking

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Merge remote-tracking branch 'dgit/dgit/sid' into experimental

changelog: start 5.6

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 5.5

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit(1): Fix a wrong reference to \fp, which should be \fP.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

debian/control: Add missing comma

Closes:#902578.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

New gnupg agent thing

before (5.3)

1044.27user 126.77system 6:26.89elapsed 302%CPU (0avgtext+0avgdata 402420maxresident)k
1778718inputs+5123720outputs (2370major+57693524minor)pagefaults 0swaps

after (5.3 + these changes)

1072.81user 131.38system 6:25.02elapsed 312%CPU (0avgtext+0avgdata 402448maxresident)k
1777558inputs+5124952outputs (2370major+59176578minor)pagefaults 0swaps

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

No-change upload, for testing.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Merge remote-tracking branch 'dgit/dgit/experimental' into HEAD

-s ours