git-debpush: Do not spuriously print -sn/-sk

When looking for -sn or -sk in debian/source/options, we need to not
print it to stdout even if grep finds it.  So use -q.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Acked-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: use -m rather than -F- <<EOF

This means that a run with set -x shows the tag data that is being
signed.  The user can even then try it again.  It will make it easier
to do some kind of dry run, too, if we want to.

We are really not in danger of hitting a command line length limit.

(It also involves fewer syscalls, less use of /tmp etc., although that
is really irrelevant here.)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Acked-by: Sean Whitton <spwhitton@spwhitton.name>

dgit clone: No longer create an "origin" remote

This was a normal git remote pointing to the dgit git server.

This is not really sensible because
1. it can't simply be pushed to
2. sometimes (if the package was never pushed with dgit)
    it doesn't even exist
3. it may be out of date with respect to the archive

The effect of 2 is that it can break `git remote update'.

This "git remote add" was introduced in 2013 in
  39c6c123ab730d42ec7c9ed01c30e0175c7691e7
  fix branch usage
and was first included in 0.2.  There doesn't seem to be any evidence
of me having deeply considered the merits of this at the time.

Closes: #932694
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: Rename debian/3-1 tag to test-dummy/ in a test git

Our distro is test-dummy.  We are going to stop randomly fetching tags
from the dgit git server, so the debian/ one would be omitted.

Replace the tag with a fresh equivalent one and change the one use
site.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-playtree-setup: Rename from git-playtree-create

The name is from when I thought it would actually mkdir.  But it
doesn't.  No overall functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-playtree-create: Fix with tests/using-intree

Reported-by: Sean Whitton <spwhitton@spwhitton.name>
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

copyright: credit Genome Research Limited

I work for the Wellcome Sanger Institute, but am employed by Genome
Research Limited, so it is they who should be credited in Copyright
notices.

Signed-off-by: Matthew Vernon <mv3@sanger.ac.uk>
Closes: #932630

git-debrebase(1): Make this manpage a less intimidating entry point

- Nicer short description right in the first section.

- Don't use the term 'delta queue' until it's needed.

- Don't talk about other manpages so early.

- Cater for people who type `man git-debrebase` looking for a
  barebones tutorial, not (just) a command line reference.

  - That tutorial should minimise references to dgit, especially since
    users might want to use git-debpush to upload.

Thanks to Sam Hartman for the user feedback which prompted this and
Ian Jackson for feedback on the content of "QUICK REFERENCE".

Closes: #926656
Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Dgit.pm: Abolish now-unused $t_local_git_cfg playtree_setup arg

This was there so that dgit's -c options would be honoured here.
But now, we get this information via @git, which dgit adds -c
options to, which we pass to git-playtree-create by invoking it
via git.

So this is no longer used and can be abolished.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-playtree-setup: Provide to git-debpush

Not used yet, so no overall functional change to the script.
The .deb has a new file.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-playtree-setup: Rewrite in shell and call it from Perl

We want this because git-debpush is going to want this functionality
but doesn't want to include (or have a copy of) Dgit.pm.

The installation arrangements are rather ugly.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

Dgit.pm: Provide @git

This clears the way for some parts of Dgit.pm to honour dgit's
git invocation conventions.

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

i18n-commit - autogenerated

git-debpush: Check upstream source is identical in the upstream tag

Closes: #932476
Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>
Acked-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit-maint-pbo(7): Add to .gitignore

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit-maint-bpo(7): new manpage

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>
Closes: #857490

git-debrebase(*): Fix references to dgit-maint-debrebase(7)

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>
Closes: #932536

git-debpush: Support forcing individual checks

While using git-debpush interactively, if the user sees that checks
have failed which they know to be safe to override, they typically
just use --force or -f.

However, it is useful for scripts to be able to skip single checks,
and if the user knows that a check will fail in advance of running
git-debpush, it is convenient to be able to specify that but still see
whether other checks fail, where those failures were unexpected.  This
can avoid the user having to run git-debpush more than once.

The list of any checks which failed but were overridden is not stored
in the generated tag.  git-debpush's checks are for the convenience of
the local user only, and the list of failed but overridden checks is
not considered to be metadata for the upload.  Not recording failed
but overridden checks in the git tag keeps git-debpush's checking
independent of its wrapping of git-tag and git-push, which makes
git-debpush simpler and easier to understand.  The complexity is on
the server side, in tag2upload.

git-debpush(1): We want the list of checks that can be overridden to
be at the bottom of the list of options because most users will not
need to look at it.  We also want the description of --force|-f to be
adjacent to the description of --force=<check>.  So move the
description of --force|-f to the end of the list of options.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>
Closes: #932459

git-debpush: Subheaders

No functional change.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush(1): Correct description of --force

Checks are still performed, but turned from errors into warnings.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

changelog: start 9.5

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 9.4

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Update copyright notices in many files

Mostly, adding 2019.  Also adding some missing credits to Sean

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

debian/copyright: Add some missing credtis

  Frans Spiesschaert (.nl translation)
  Matthew Vernon / Sanger Institute
  Paul Hardy

Information obtained by hand from git.  This should have been done
much earlier.  If only we had some tool for this.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: read-only access to the Ubuntu Cloud Archive (Closes: #932322)

This adds -d ubuntucloud for the Ubuntu Cloud Archive (see
https://wiki.ubuntu.com/OpenStack/CloudArchive for details thereof).

Access is by the aptget: query mechanism, suites are spelled like
bionic-updates:train or bionic-proposed:train, additionally the
shorthand bionic:train for bionic-updates:train is allowed.

Closes: #932322
Signed-off-by: Matthew Vernon <mv3@sanger.ac.uk>
Acked-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit: move suite name check to after suite rmap

The Ubuntu Cloud Archive has Codenames of the form
bionic-updates/train (the / in which suitere doesn't match); this can
be corrected by suite rmap, so apply that before doing the suitere
sanity check.

Signed-off-by: Matthew Vernon <mv3@sanger.ac.uk>
Acked-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: tag-to-upload tests: Tolerate a changed git message

git changed
  Couldn't find remote ref
to
  couldn't find remote ref
and this broke our test case.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Closes: #932475

test suite: tag-to-upload tests: Make message checks regexps

No functional change yet.  All the existing message strings are
already equivalently valid when interpreted by egrep.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: tag-to-upload tests: Set LC_MESSAGES in with-mangled

t-expect-fail would do this but we aren't using it here, so we need to
do it explicitly.

This will avoid the tests failing in some locales.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 9.4

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalis 9.3

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Document changes for tagupl-gbp

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: tagupl-gbp: New test

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: t-debpolicy: mkdir -p the git repo

This makes t-debpolicy compatible with (eg) setup imports that already
made a $tmp/git.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: splitbrain: "bad b/p" prints more info

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: setup/gbp: Make the result the master branch

This means we can drop --git-ignore-branch from gbp-orig.

No overall functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: break out tests/setup/gbp

This creates a fairly usual set of gbp style git branches/tags.

No overall functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: t-gbp-example-prep: Drop t-tstunt-parsechangelog

And hoist it into all callers.  In fact there is only one caller that
doesn't have it already.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Document dgit(1) archive-query doc

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit(1) document the presence of the aptget archive-query method

This is already used for debian-security, and might be useful for
other archives that like madison or similar (e.g. the Ubuntu Cloud
Archive).

Closes: #932321
Signed-off-by: Matthew Vernon <mv3@sanger.ac.uk>

git-debpush: refactor to avoid more than one push command

No functional change.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

changelog: update for tag-to-upload server-side fixes

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: update for git-debpush improvements

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: check that upstream tag is ancestor of $branch

Closes: #932096
Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: gather some information earlier

For use by sanity checks.

Pure code motion.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: cope with spaces in user-supplied upstream tag

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: check for target suite change since last upload

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: check for pushing the dgit view to the maintainer view

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: factor out find_last_tag()

Also determine the last debian/ tag earlier, for use by sanity checks.

No functional change.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: introduce and call fail_check

Without this, passing --force to override an earlier sanity check
might cause the user to miss the output of a later sanity check, which
would never get run.

We don't want to have multiple forcing options to override different
sanity checks, as that is too much complexity for a script like this.

So always run all checks, and error out afterwards if at least one of
them failed.

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

test suite: tagupl-baredebian: New test

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: baredebian: Break up baredebian-test-core

For a baredebian tag-to-upload test, we're going to want to replace
the push.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Dgit.pm: printcmd: Print in one go

Sometimes this message is used to report failure of `git fetch' etc.
But when such operations fail, they may interleave output from the
remote git, which typically prints
  fatal: The remote end hung up unexpectedly
(referring to the local end as the `remote' end, from its point
of view), after the the local end printed its message and failed.

This sets up a race between Dgit.pm's printcmd and the message from
the remote.  If we're really unlucky, the remote's message interrupts
the output from printcmd.

We can avoid the within-line interleaving - at least, the interruption
of our message - by printing the whole message in one go, so do that.
(Strictly, the approach we use here only ensures that the message
appears in a single write(2) call if it fits in the stdio buffer.)

One of our test cases (tagupl) depends on this message not being split
and has a small chance of failing without this fix.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: baredebian: Move the quilt fixup forward

We are going to split this function up so that our caller can do the
push via tag-to-upload.

No overall functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

tag2upload: baredebian: Pass dgit refs/tags/, not the commit hash

This will enable git to report the upstream tag in its messages.

The tag name is safe to pass here since dgit only uses its referent,
unless it is nice enough to print.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Dgit.pm: resolve_upstream_version: Report tag, where sensible

This affect what is show in commit messages (including the parseable
parts) made by dgit and git-debrebase.  Now they prefer to refer to
the tag name if the caller specified refs/tags/ and the tag name is
a nice one.

This is not expected to make much difference for human callers (who
will probably not explicitly qualify their tag refs).  But it will
make a difference for scripts.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

tag2upload: Actually recognise and honour --quilt in tag

Non-default quilt modes were completely broken before.

We only didn't notice because the existing tests worked with the
default.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit-repos-server: parsetag: Print a bit more debug about junk

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit-repos-server: Don't crash on --quilt=baredebian

The die needs to be qualified.  This little bit of fcode has never
worked.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Dgit.pm: reflog_cache_insert: Use ensurepath for the reflog

Otherwise quilt fixup can fail in a too-fresh git tree.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Dgit.pm: Provide `ensurepath'

Like `ensuredir' but also a bit like a more controlled `mkdir -p'.

We use make_path from File::Path.  It dies on failure.

No caller yet.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: tagupl-*: Move t-pushed-good out of t-tagupl-test

We are going to want to do this separately in some cases.

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Start 9.3

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Finalise 9.2 (no source changes)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: start 9.2

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 9.1

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Minor adjustment to git-debpush discussion

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Add a couple of missing entries

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit-repos-server: Remove comment saying it's a prototype

We have done everything to make it safe to use.  Obviously it still
needs some web service queue thing around it to invoke it.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debpush(1): Tweak AUTHOR section

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: Don't pretend to support --baredebian+tarball

Instead of just deleting the option from the argument parser, accept
it, but print a message explicitly stating that the *quilt mode* is
not supported by this workflow (i.e. it's not just git-debpush passing
the quilt mode on to tag2upload that we don't support).

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

changelog: Update

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush(1): Fix POD formatting

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

test suite: unset CDPATH

This can cause tests to pass which should fail (!)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: tagupl: Test git-debpush --upstream=

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debpush: Defend against git-deborig #931509

Without this, when git-deborig fails and #931509 isn't fixed,
git-debpush will silently exit with a nonzero exit status which is
very unfriendly.

With this change it does something like this:
  git-debpush: git-deborig failed; maybe try git-debpush --upstream=TAG
  fatal: Invalid object name 'refs/tags/couldn't find any of the following tags'.
It then exits status 127.  This is obviously daft, but making this
any better is even more complex.

When git-deborig is fixed it will do this:
  couldn't find any of the following tags: 1.0, v1.0, upstream/1.0
  tell me a tag or branch head to make an orig.tar from: git deborig --just-print '--version=1.0-1' COMMITTISH
  git-debpush: git-deborig failed; maybe try git-debpush --upstream=TAG
which is still not brilliant but I guess it will do.

The downside, if it is one, is that we lose git-deborig's original
exit status.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Acked-by: Sean Whitton <spwhitton@spwhitton.name>

test suite: tagupl: Drop an unused variable

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debpush: Support --upstream=TAG (and document it)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debpush: Defend against unexpected git-deborig output

If git-deborig prints something other than a tag name, for example an
error message (see #931509), this approach will bomb out, roughly
appropriately.

This will also become useful when we make upstream_tag overrideable.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Acked-by: Sean Whitton <spwhitton@spwhitton.name>

git-debpush: Change comment heading

IMO "Principles of Operation" would be a comprehensive reference
detailing the model and functionality.  I think "Design Principles"
is the conventional name for semi-self-imposed choices/constraints.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Acked-by: Sean Whitton <spwhitton@spwhitton.name>

test suite: t-tagupl-test: Check that salsa master was pushed

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: tagupl-native: Drop obsolete todo comment

We do now test all these, in tests/tests/tagupl.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: t-tagupl-test: Use "salsa" as the source

This checks that push pushed the tag(s) there.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: t-tagupl-run-drs: New source parameter

Change all callers.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: tagupl*: Drop use of --no-push

Now we do actually push.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: t-tagupl-setting: Provide a "salsa" remote

This will give git-debpush somewhere to push to.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: tagupl*: Run t-tagupl-settings in $p

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

git-debpush: Fixes to help with pushing native packages

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Entry for new "git-push-to-upload" workflow tools

Signed-off-by: Sean Whitton <spwhitton@spwhitton.name>

dgit-repos-server: tag2upload: Clarify key situation comments

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: tagupl: Check Source mismatch

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: tagupl: Test changelog version

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dgit-repos-server: tag2upload: Check changelog info

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: tagupl: Update version when needed

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: tagupl: Test lack of signature

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: tagupl: Test signature verification fail.

This test case is broken because it doesn't call t-tagupl-run-drs

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: tagupl: raw-mangled: Use 1.0-2 tagname in raw-mangled

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: tagupl: raw-mangled: Fix mangled tagger test

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: tagupl: raw-mangled: Break out

No functional change.

This test case is broken because it doesn't call t-tagupl-run-drs

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test suite: tagupl: A few more error case tests

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>