http[s] tests: Add http test case

Maybe we will turn this into a TLS test case in due course.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

http[s] tests: Fix stdout handling in ftpmaster test server

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

http[s] tests: More-or-less functional ftpmaster http server

Not used anywhere yet.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

http[s] tests: Import and slightly hack example script

This is basically a copy of some of the example code from the
HTTP::Server::Simple docs, hacked up with my own (so far) nonsense.

Add copyright info to file and d/copyright.  Upstream licence is
Artistic or GPL-1+.  We choose GPL-1+ and upgrade to GPL-3+.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: lib: Provide $tmp/must-clean directory to tests

This is a directory for things which should be removed on exit.
Currently there are none.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Slightly more debugging for check_for_git with http[s]

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Fix a typo in the entry for 9.6

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Document changes to enumerate-tests

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: enumerate-tests: Spot t-alt-test

And find the dependencies of the referred-to test.  No functional
change since currently no chained-to test has any nonstandard
dependencies.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: enumerate-tests: gencontrol-add-things: restrictions

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: enumerate-tests: gencontrol-add-things: delim feature

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: enumerate-tests: Break out gencontrol-add-things

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Start 9.7

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Write and finalise for 9.6

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-playtree-create: Drop an unwanted set -x

Closes: #932959
Reported-by: Sean Whitton <spwhitton@spwhitton.name>
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

debian/control: Drop now unused dependency on libwww-perl.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit(1): Drop mention of LWP::UserAgent since we don't use it now

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Abolish url_get in favour of url_fetch

This removes use of LWP from dgit.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Use libcurl, not @curl, for git check

This means that when we sort out our TLS problems this will be fixed
too.

The remaining use of command-line curl is just to download additional
.dsc compoments.  I intend to leave that as-is.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: url_fetch: Honour new CurlOpts

No functional change with the existing callers.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: url_fetch: Honour new AccessBase

No functional change with the existing callers.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Reorganise url_fetch

This was archive_api_query_curl.  We're working on making it good for
other things too.

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: libcurl usage: Install SIGPIPE handler ourselv

We don't want a global SIGPIPE setting.  In particular we do not want
to run any of subprocesses with SIGPIPE ignored.

Just in case libcurl gets this wrong, tell it CURLOPT_NOSIGNAL, and do
it ourselves.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Set $SIG{INT} to DEFAULT

There is a bug in libdpkg-perl, where Dpkg::Source::Package installs a
handler on loading.  This has the effect of making in-process C
calls (eg to libcurl) uninterruptible by ^C.

Work around it by unconditionally resetting SIGINT.  (Anyone who
deliberately invoked us with SIGINT ignored deserves to keep all the
resulting pieces.)

See #932841.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: archive_api_query_curl: Add a bit of debug

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Use WWW::Curl rather than invoking curl(1) for archive api

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Drop support for old attempts at cert/key pinning

I don't think anyone is using these.  We are going to do something
different to try fo fix #932570.

When #790093 is fixed we may need something like archive-query-tls-key
(or even exactly that) but we would have to reimplement it for
WWW::Curl anyway.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

tag2upload: Use the non-readonly method for dgit fetch

The principal effect in the Debian configuration is to access the
master dgit git server via ssh, rather than the mirror via https.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Provide --for-push option

This is much more conveient and in some cases sensible than
 -cdgit-distro.DISTRO.readonly=0

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Make cmd_archive_api_query call api_query_raw

In support of #932570.  Now there is only one caller of
archive_api_query_cmd.

It is OK to pass `undef' because
  - archive_

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Move $data check into api_query

In support of #932570.  Our intended other caller doesn't have a
$data.  No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: Break out api_query_raw

In support of #932570.  No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Add missing Closes of #932476 to changelog entry for 9.5

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 9.6

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

changelog: finalise 9.5

changelog: Bug close syntax fix

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

changelog: Minor fixes

Mention two omitted bug Closes.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Mention Vcs-Git change

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Update

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: Check for a detached head when pushing HEAD

See discussion in #932612.

Suggested-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: Check for unstitched git-debrebase branch

Closes: #932612
Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

debian/control: Name salsa as our Vcs-*

That is mostly what we are using now.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Wrap a line and mention HACKING.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Update

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: Check that patches are (un)applicable

Closes: #932477
Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: check_treesame: Show diffstat when there is a diff

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: check_treesame: Also pass --quiet to 'git diff'

Otherwise, when using git-debpush in a terminal, the user will get a
pile of unwanted diff output each time check_treesame is called.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

HACKING: First cut

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debpush: Do not spuriously print -sn/-sk

When looking for -sn or -sk in debian/source/options, we need to not
print it to stdout even if grep finds it.  So use -q.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Acked-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: use -m rather than -F- <<EOF

This means that a run with set -x shows the tag data that is being
signed.  The user can even then try it again.  It will make it easier
to do some kind of dry run, too, if we want to.

We are really not in danger of hitting a command line length limit.

(It also involves fewer syscalls, less use of /tmp etc., although that
is really irrelevant here.)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Acked-by: Sean Whitton <spwhitton@spwhitton.name>

dgit clone: No longer create an "origin" remote

This was a normal git remote pointing to the dgit git server.

This is not really sensible because
1. it can't simply be pushed to
2. sometimes (if the package was never pushed with dgit)
    it doesn't even exist
3. it may be out of date with respect to the archive

The effect of 2 is that it can break `git remote update'.

This "git remote add" was introduced in 2013 in
  39c6c123ab730d42ec7c9ed01c30e0175c7691e7
  fix branch usage
and was first included in 0.2.  There doesn't seem to be any evidence
of me having deeply considered the merits of this at the time.

Closes: #932694
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Rename debian/3-1 tag to test-dummy/ in a test git

Our distro is test-dummy.  We are going to stop randomly fetching tags
from the dgit git server, so the debian/ one would be omitted.

Replace the tag with a fresh equivalent one and change the one use
site.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-playtree-setup: Rename from git-playtree-create

The name is from when I thought it would actually mkdir.  But it
doesn't.  No overall functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-playtree-create: Fix with tests/using-intree

Reported-by: Sean Whitton <spwhitton@spwhitton.name>
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

copyright: credit Genome Research Limited

I work for the Wellcome Sanger Institute, but am employed by Genome
Research Limited, so it is they who should be credited in Copyright
notices.

Signed-off-by: Matthew Vernon <mv3@sanger.ac.uk>
Closes: #932630

git-debrebase(1): Make this manpage a less intimidating entry point

- Nicer short description right in the first section.

- Don't use the term 'delta queue' until it's needed.

- Don't talk about other manpages so early.

- Cater for people who type `man git-debrebase` looking for a
  barebones tutorial, not (just) a command line reference.

  - That tutorial should minimise references to dgit, especially since
    users might want to use git-debpush to upload.

Thanks to Sam Hartman for the user feedback which prompted this and
Ian Jackson for feedback on the content of "QUICK REFERENCE".

Closes: #926656
Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Dgit.pm: Abolish now-unused $t_local_git_cfg playtree_setup arg

This was there so that dgit's -c options would be honoured here.
But now, we get this information via @git, which dgit adds -c
options to, which we pass to git-playtree-create by invoking it
via git.

So this is no longer used and can be abolished.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-playtree-setup: Provide to git-debpush

Not used yet, so no overall functional change to the script.
The .deb has a new file.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-playtree-setup: Rewrite in shell and call it from Perl

We want this because git-debpush is going to want this functionality
but doesn't want to include (or have a copy of) Dgit.pm.

The installation arrangements are rather ugly.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

Dgit.pm: Provide @git

This clears the way for some parts of Dgit.pm to honour dgit's
git invocation conventions.

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

i18n-commit - autogenerated

git-debpush: Check upstream source is identical in the upstream tag

Closes: #932476
Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>
Acked-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit-maint-pbo(7): Add to .gitignore

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit-maint-bpo(7): new manpage

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>
Closes: #857490

git-debrebase(*): Fix references to dgit-maint-debrebase(7)

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>
Closes: #932536

git-debpush: Support forcing individual checks

While using git-debpush interactively, if the user sees that checks
have failed which they know to be safe to override, they typically
just use --force or -f.

However, it is useful for scripts to be able to skip single checks,
and if the user knows that a check will fail in advance of running
git-debpush, it is convenient to be able to specify that but still see
whether other checks fail, where those failures were unexpected.  This
can avoid the user having to run git-debpush more than once.

The list of any checks which failed but were overridden is not stored
in the generated tag.  git-debpush's checks are for the convenience of
the local user only, and the list of failed but overridden checks is
not considered to be metadata for the upload.  Not recording failed
but overridden checks in the git tag keeps git-debpush's checking
independent of its wrapping of git-tag and git-push, which makes
git-debpush simpler and easier to understand.  The complexity is on
the server side, in tag2upload.

git-debpush(1): We want the list of checks that can be overridden to
be at the bottom of the list of options because most users will not
need to look at it.  We also want the description of --force|-f to be
adjacent to the description of --force=<check>.  So move the
description of --force|-f to the end of the list of options.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>
Closes: #932459

git-debpush: Subheaders

No functional change.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush(1): Correct description of --force

Checks are still performed, but turned from errors into warnings.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

changelog: start 9.5

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 9.4

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Update copyright notices in many files

Mostly, adding 2019.  Also adding some missing credits to Sean

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

debian/copyright: Add some missing credtis

  Frans Spiesschaert (.nl translation)
  Matthew Vernon / Sanger Institute
  Paul Hardy

Information obtained by hand from git.  This should have been done
much earlier.  If only we had some tool for this.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: read-only access to the Ubuntu Cloud Archive (Closes: #932322)

This adds -d ubuntucloud for the Ubuntu Cloud Archive (see
https://wiki.ubuntu.com/OpenStack/CloudArchive for details thereof).

Access is by the aptget: query mechanism, suites are spelled like
bionic-updates:train or bionic-proposed:train, additionally the
shorthand bionic:train for bionic-updates:train is allowed.

Closes: #932322
Signed-off-by: Matthew Vernon <mv3@sanger.ac.uk>
Acked-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: move suite name check to after suite rmap

The Ubuntu Cloud Archive has Codenames of the form
bionic-updates/train (the / in which suitere doesn't match); this can
be corrected by suite rmap, so apply that before doing the suitere
sanity check.

Signed-off-by: Matthew Vernon <mv3@sanger.ac.uk>
Acked-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: tag-to-upload tests: Tolerate a changed git message

git changed
  Couldn't find remote ref
to
  couldn't find remote ref
and this broke our test case.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Closes: #932475

test suite: tag-to-upload tests: Make message checks regexps

No functional change yet.  All the existing message strings are
already equivalently valid when interpreted by egrep.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: tag-to-upload tests: Set LC_MESSAGES in with-mangled

t-expect-fail would do this but we aren't using it here, so we need to
do it explicitly.

This will avoid the tests failing in some locales.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 9.4

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalis 9.3

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Document changes for tagupl-gbp

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: tagupl-gbp: New test

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: t-debpolicy: mkdir -p the git repo

This makes t-debpolicy compatible with (eg) setup imports that already
made a $tmp/git.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: splitbrain: "bad b/p" prints more info

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: setup/gbp: Make the result the master branch

This means we can drop --git-ignore-branch from gbp-orig.

No overall functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: break out tests/setup/gbp

This creates a fairly usual set of gbp style git branches/tags.

No overall functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: t-gbp-example-prep: Drop t-tstunt-parsechangelog

And hoist it into all callers.  In fact there is only one caller that
doesn't have it already.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Document dgit(1) archive-query doc

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit(1) document the presence of the aptget archive-query method

This is already used for debian-security, and might be useful for
other archives that like madison or similar (e.g. the Ubuntu Cloud
Archive).

Closes: #932321
Signed-off-by: Matthew Vernon <mv3@sanger.ac.uk>

git-debpush: refactor to avoid more than one push command

No functional change.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

changelog: update for tag-to-upload server-side fixes

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: update for git-debpush improvements

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: check that upstream tag is ancestor of $branch

Closes: #932096
Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: gather some information earlier

For use by sanity checks.

Pure code motion.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: cope with spaces in user-supplied upstream tag

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: check for target suite change since last upload

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: check for pushing the dgit view to the maintainer view

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: factor out find_last_tag()

Also determine the last debian/ tag earlier, for use by sanity checks.

No functional change.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: introduce and call fail_check

Without this, passing --force to override an earlier sanity check
might cause the user to miss the output of a later sanity check, which
would never get run.

We don't want to have multiple forcing options to override different
sanity checks, as that is too much complexity for a script like this.

So always run all checks, and error out afterwards if at least one of
them failed.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

test suite: tagupl-baredebian: New test

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: baredebian: Break up baredebian-test-core

For a baredebian tag-to-upload test, we're going to want to replace
the push.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>