Merge branch 'stable'

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Regularise patch filenames, and defend against funny commit subjects

* Do not specified patch names which look like series filenames
* When we invent a filename based on a commit message, add ".patch".

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

using-these: New script to help with ad-hoc testing

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 4.1~

dgit: Pass --no-renames to git diff-tree -z, avoiding potential trouble

Without this option, git-diff-tree might detect a rename (or possibly
even a copy).  If it does it prints a different output format with a
status of C or R *and a separate filename*.  The latter is an
additional nul-terminated record and would get dgit's interpreter of
the git diff-tree output out of step.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 3.12~

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 3.11

Actually understand foo,-security (!)

There does not in fact seem to have been any code which implements
this line from dgit(7):
    If a specified subsuite starts with - then mainsuite is prepended.
The code which canonicalises it back to the version with ",-" does
work, though.

Closes:#867189.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: mkdir .git/info in setup_gitattrs

This might be necessary if setup_mergechangelogs were disabled.
(This is something of a latent bug, since `git init' creates
.git/info.)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Garden and reformat

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Test multisuite clone without --rm-on-error.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: clone multisuite works even without --no-rm-on-error.

This is slightly subtle.

We need $rmonerror in cmd_clone to be cleared.  cmd_clone does that
only when clone itself returns.  The multisuite plumbing means that
currently, clone returns only in the child which set up the tree and
set up the first suite.  Unsetting $rmonerror there is correct because
we want to keep the tree on success, and the parent will remove it if
there is a later failure.  (Ie, the child remains responsible for
removing the tree if it itself fails.)

In the parent, we get $multi_fetched==1 if not only our original clone
child succeeded, but all the other fetches worked too.  We are now
responsible for the tree.  If our final tasks are successful, again,
we need not to delete the tree.

Closes:#867434.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: badcommit-fixup: Check core.sharedRepository works

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit-badcommit-fixup: Honour core.sharedRepository.

Closes:#867603.

In principle it might be nicer to copy more options.  But we don't
want to duplicate the logic in prep_ud in dgit, and we don't want to
make this script too standalone.  I'm not aware of other options that
are important, rather than nice-to-have tuneables.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: honour more pre-tree git config options in our private trees

These share the user's object store and we should manipulate the
object store the way the user wants.  In particular,
core.sharedRepository is important.

Prompted by #867603, which is the same bug in dgit-badcommit-fixup.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: tolerate compressor terminating with SIGPIPE.

Closes:#857694.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: avoid dying with wrong message if compression fails

When dgit intended to report a decompressor had failed, it would
instead crash with no useful message.  This was due to
generate_commits_from_dsc's @compr_cmd being shadowed inside
the if $compr_ext.

This is part of #857694.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

infrastructure: Cope with new git-receive-pack which has quarantine feature

Ie, work around #867702.  See the bug there for discussion.

(We are perhaps flying a little close to the wind with our wrapper
script location, but it doesn't seem likely to break, to me.)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

infra: dgit-repos-server: Break out mkscript

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Add many pre_* to call no_local_git_cfg

For each operation which can meaningfully be run outside a git tree,
arrange to call no_local_git_cfg and thus avoid running
  git config --local

There is one slight infelicity: some subcommands (notably
archive-api-query) could in theory be run within a git tree and expect
that git tree to influence their output.  However, this seems
unlikely.  In fact, I think there are probably only in-tree callers
and the in-tree callers do not do this.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Provide no_local_git_config

Part of the fix to #865863.  No callers yet, so no functional change.

This function not only arranges not to call git config --local.
Removing 'local' from @gitcfgsources also prevents various config
lookup machinery from trying to find information in $cfg{local}
now of course does not exist.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: rpush: Do argument parsing and chdir in pre_...

The only thing in between pre_... and cmd_... os git_slurp_config.

The code now in pre_... does the following things, all of these should
be done before git_slurp_config:

 * Sets some global variables affecting debugging and logging output.
   If git_slurp_config is told to print debugging, it should be
   affected by these changes.

 * Rearranges its file descriptors so that stdin/stdout are as the
   rest of dgit expects, and the protocol is on PI/PO.  If
   git_slurp_config were to use stdin/stdout, it ought to use the
   "normal" versions, and not access the protocol streams.

 * Checks and calculates the negotiated protocol version.  This is not
   affected by the config, only by the arguments form the caller and
   our own idea of the protocol versions we support.

 * Changes to the appropriate working tree.  Doing this before
   slurping the config arranges to honour the local git config from
   build host working tree.  (It also avoids rpush failing on newer
   git due to asking for git config --local in the wrong place.)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Postpone call to git_slurp_config

This is needed to fix #865863: we need to know what our operation is,
before we can decide whether to look for --local git config.

The code which now runs earlier is:
 * The messages about $dryrun_level (which is set only by
   the command line, and not by configuration - verified by
   searching for $dryrun_level).
 * Usage failure if @ARGV empty.  This is not affected by
   configuration.  (parseopts does the argument parsing and
   already runs before git_slurp_config.)
 * Extracting the $cmd from @ARGV.
 * Calling $pre_fn.  There is only one pre_* sub, which is
   pre_gbp_build.  It provides the default for $quilt_mode.
   $quilt_mode is indeed somewhat entangled with the git config,
   but this takes place in parseopts_late_defaults, which is
   called much later.

Therefore there is no functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: curl --proto-redir settings: do not use qw() for ...,...

This syntax generates the following spurious warning with buster's
perl (but not with stretch's):
  Possible attempt to separate words with commas at /home/ian/things/Dgit/2dgit/dgit line 103.

It seems better to keep this warning everywhere else, and using a
different syntax is easier than disabling it around this construct.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Cope with git restricting ext:: protocols.

buster's git rejects ext:: by default.  See #XXXX and
  man git-config |less +/'protocol.*allow'

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Cope if the archive server sends an HTTP redirect

We achieve this by passing -L to curl.

We also pass an appropriate-seeming --proto-redir, because the curl
manual is not entirely reassuring that following redirections with the
default configuration is safe.

This finally fixes #867185/#867309.  What happens there is that curl
gets a redirect, along with an HTML error document.  curl then exits
with status zero, effectively pretending that the error document is
the resource which was requested.  dgit notices that something is
wrong because the file does not have the expected cryptographic
checksum.

I suspect that there are other download problems which would give a
similar effect.  Sadly the curl manpage doesn't seem to suggest a way
to avoid this.  At least, dgit will never carry on in such a
situation, since it insists that the file has the right hash.  And if
it does have the right hash we don't really care how it was obtained.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: avoid "Use of uninitialized value $got in concatenation"

When dgit wanted to report that a file it downloaded had the wrong
checksum, it would instead crash with this message.  This was due to
complete_file_from_dsc's $got (which is relied on by the callers of
$checkhash) being shadowed inside $checkhash.

This is part of #867185/#867185.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: fix rpush+buildinfo: Transfer buildinfos for signing.

buildinfos are supposed to be signed.  And, indeed, if they are
present, debsign wants to sign them.  That means they need to be
transferred to the signing end, and back again.

We check that the filename is not totally unreasonable, but do not
attempt to verify it completely.  If there are situations where
unwanted or confusing buildinfos are generated, this is the fault of
the build process.  dgit rpush should, in this respect, do the same as
debsign+dput - ie faithfully sign and upload what the build has
provided.

We do check that the buildinfo doesn't look too much like a .changes,
and mentions the same files as the .changes (insofar as they mention
files in common).  This is a rather nugatory defence against some
kinds of bait and switch attacks.

This is in some sense an incompatible protocol change: if the build
host has a new dgit, and sends buildinfos, an old dgit on the
initiator will declare a protocol violation.  However, the new
protocol elements occur only when needed.  in this situation, the only
way to get things to work at all with the old dgit at either end would
be to strip out the buildinfos, which is obviously undesirable.

Closes:#867693.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

rpush: break out @rfiles in dopush

This very slight refactoring makes it easier to add additional files
here, and also factors out the appending of $dryrunsuffix to each one.

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Declare fast forward from 0.22-experimental2

[dgit --overwrite=0.22-experimental2]

changelog: finalise 4.0

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: --deliberately-not-fast-forward works properly in split view quilt modes (suppressing the pseudomerge).

changelog: start 4.0~

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 3.11~

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 3.10

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Update for docs fixes

Signed-off-by: Nicholas D Steeves <nsteeves@gmail.com>

docs: Fix typos

Signed-off-by: Nicholas D Steeves <nsteeves@gmail.com>

import-maintmangle: New test for changelog Maintainer mangling.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

lib-import-chk: New import_chk_changelog_massage hook

Nothing sets this yet, so no functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: importing: Better handle commas in changelog maintainer fields

Some maintainers have written commas in the maintainer field of their
changelog entries.  Such changelog entries could not be imported.

clogp_authline had code to replace multiple maintainers (which are
hypothetical right now) into just the first, but that trips on these
maintainers with commas.

Ultimately we should expect that the Maintainer: field from
dpkg-parsechangelog is in (a subset of) RFC5322 recipient field
format.  In that format, any commas would need to be quoted.

So:

If the Maintainer field from dpkg-parsechangelog has a comma which has
no @ or " before it, then we consider it a single old-school
comma-containing maintainer.  If it were intended as multiple
maintainers, then the first maintainer has no email address.  Not
coping properly with that very-hypothetical future seems OK.  We
simply delete the comma, so that the things we record in the git
history are more conservative.

If there is a " we leave things untouched in the hope that this is a
single address, albeit with some quoting.  The alternative would be to
try to use a full RFC5322 parser.  That's quite a risky change.

Perhaps we will revisit this after stretch.

For now this Closes:#852661.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: lib-import-chk: Test commit authorship

Check that commits have smae authorship as appears in the changelog.
(Or, at least, the same authorship set.)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Do not fail when run with detached HEAD.  Closes:#853022.

Specifically:

* Pass -q to git-symbolic-ref.  That means that it doesn't print
  an error message when HEAD is not a symbolic ref (ie, a
  detached head), and it means that the exit status is then 1
  rather than 128.

* If the return value from cmdoutput_errok is undef, check $? (which
  is the exit status <<8) and then simply pass on the undef.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: branchsuite: Break out @cmd

We are going to need to reuse this to print a proper error message.

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Strip initial newline from Changes line from dpkg-parsechangelog

so as to avoid blank line in commit messages.  Closes:#853093.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: quilt-useremail: New test for user config copying

Test for #853085.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Copy several user.* settings from main tree git local config

Copy to dgit private workarea.  Closes:#853085.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 3.10~

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 3.9

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit-maint-merge(7): Quote sample clone commands

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit-maint-merge(7): Get git clone url right.  Closes:#852609.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: overwrite-chkclog: test UNRELEASED handling.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit --overwrite: Check $gf->('Distribution')

Check that the overwritten version's changelog entry is not
UNRELEASED.

This could easily happen if this release was being made from a git
branch which predates the previous package upload, with working
practices which commit finalised UNRELEASED changelog entries with the
complete version number for the next upload.  (Such practices seem
quite common.)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

pseudomerge_version_check; Break out $gf

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 3.9~

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 3.8

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: defdistro-setup: Test that setup-* functions distro selection works.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: t-defdistro: Set distro='' too

This means that t-defdistro works without t-chain-test and
reinvocation.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

config and suite handling: Make dgit-setup-* work in default distro.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 3.8~

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

finalise 3.7

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Documemnt dgit-maint-merge changes

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit-maint-merge(7): updates re #852090: suggestions by Sean

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit-maint-merge(7): updates re #852090: suggestions by me

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit-maint-merge(7): updates re #852090: suggestions by me

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit-maint-merge(7): updates re #852090: suggestions by me

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit-maint-merge(7): Fix a typo `detachs'

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Makefile: New %.view target:

`make dgit-maint-merge.7.view' runs `man -l ...'

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: defdistro-dsd-clone-drs: New test

Would have detected #851906 (and hopefully #850521).

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Document fixes to repos_server_url

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: repos_server_url: Set $access_forpush to 1

In 468edf05661e "dgit: clone-dgit-repos-server: Set $access_forpush"
we set this to 0.

But this is wrong.  The dgit-repos-server is used for pushing, not for
readonly access.  The readonly url may be entirely wrong, and indeed
it is, for Debian.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: repos_server_url: Set $isuite

We need to set this or access_*() fails.  Set it to a dummy value.
The user will need to specify -d to get the server for a non-default
distro, which is hopefully obvious.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: repos_server_url(): Break out

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 3.7~

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 3.6

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: aptget archive access method

Add dummy implementation of file_in_archive_aptget copied from
file_in_archive_dummycat.  Re:#851697.

infrastructure: Do not reject commits with no author/committer name

But still insist on email address and date).

Peter Green reports that eg 71e128629ec786f3 in upstream xen.git is
such a commit (and is accepted by github).

Closes:#851716.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

infrastructure: Properly honour NOCOMMITCHECK policy hook exit status.

Closes:#851800.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: downstream-gitless: Test import of .dsc with unsafe url.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: downstream-gitless: Test import of .dsc from unknown distro.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: git_lrfetch_sane: Take $url, and pass it right value in import

This means we actually use the url from a Dgit .dsc field naming an
unknown distro.  Closes:#851728.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: git_get_config: Use confess, not croak

Using croak was simply a mistake.  I always wanted a stack trace.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Dgit.pm: Call `confess' when shellquote gets an undef arg

This can happen if a command (eg passed to debugcmd) has an undef
argument.  This turns an undefined warning into an explicit internal
error.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 3.6~

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 3.5

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: gardening

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gitattributes: Call t-ok.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gitattributes: More testing of clone: check for message (#851624)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: clonee: Do setup_new_tree earlier.

This avoids printing a spurious warning about actually-defused
gitattributes when cloning.  Closes:#851624.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gitattributes: More testing of clone: check good clone

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gitattributes: work around git attribute updates bug

touch af/*, to work around #851713

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gitattributes: More testing of clone: check sums

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gitattributes: Start to test clone

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gitattributes: Break out sums_check_broken

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Defuse gitattributes in private working area, always

Even if we don't do it in the user's tree because config
setup-gitattributes=false.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Improve comment left in .git/info/attributes.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gitattributes: Check that warning does not appear when unexpected

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gitattributes: Break out t-dgit-warn-check

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gitattributes: Break out wd= setting

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: gitattributes: Test that checkout is good too

test suite: gitattributes: Break out sums_check

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>