python: Provide feature for argparse --[no-]foo options

This is surprisingly awkward.  StackExchange has one.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Makefile.in: break out PYMODULES

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: output_data: Change recursive variables

Pass a tuple path, of the objects. rather than the string so far and
an indent value.

This will make it easier to more complex things here.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: output_vpnflat: Change recursive variables

Pass a tuple path, of the objects. rather than the string so far and
an indent value.

This will make it easier to more complex things here.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

tests: Rename locations to `in' and `out'

It is better for testing for the location names not to be equal to the
site names.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Fix argument parsing with options

In "make-secnet-sites: Use argparse rather than ad-hoc parser",
we missed a reference to sys.argv.  The effect is that if
make-secnet-sites run in sites file output mode, and provided
with options, the output is written to the wrong place.

The only syntax that worked in 0.4.5 and broke in 0.5.0 is
make-secnet-sites -P.  So here we fix that regression.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Makefile.in: release checklist: use -j8 for sbuild

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Start 0.5.1~

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: finalise 0.5.0

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Document NAK logging changes

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

site: Provide NAK reasons mentioning names and unpick failure

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

util: priomsg: Provide and use priomsg_update_fixed

We are going to want to reuse this.

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

site: Record whynot in default (end of comm_notify) case

send_nak will print the type and the two indices, so we don't.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

site: named_for_us: Reorganise into "if"s

This gives us somewhere to put our whynot updates.

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

site: named_for_us: Introduce NAME_MATCHES

This will make the next changes much smaller.

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

site: Pass whynot through to named_for_us

We only pass this as non-0 when, if named_for_us rejects, we actually
return false from comm_notify.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

udp: Use commcommon->why_unwanted

This is the one call site of comm_notify.  (If we had had any comm
that wasn't based on udp.c, that would have needed updating too.)

No overall functional change: nothing ever sets this yet, so we always
use the default message.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

util: Reorder information in NAK log messages

In particular, put the supplied reason string last so that it will be
unambiguous.  And putting the message type first seems more natural.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

comm: Introduce commcommon->why_unwanted

This will "accumulate" a message.  It will give the information about
why the closest approach to wanting this message ultimately rejected
it.

No overall functional change with current code: nothing ever either
updates this yet, or uses it.  Users will come next.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

comm: comm_notify takes the cc, not just the notify_list

We're going to want this in a moment.

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

util: priomsg: Allow passing priomsg_update_p a null pm

This makes it more convenient for callers who may not have anything to
update.

No functional change with any existing caller, of why there aren't any
anyway.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

util: priomsg: New facility

No callers yet.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

util: truncmsg: New way of manipulating buffers

No callers yet.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

buffer: Abolish unused `flags'

This variable is never examined.

This has the comment `How paranoid should we be?' but in fact the
paranoia level (such as it is) is set by which entrypoint we call.  It
would not be appropriate to make this a buffer property anyway.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

changelog: Document changes since 0.4.5

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Taint the `group' parameter

This comes from the untrusted caller.  It should be tainted before we
use it as a filename.  (Actually in practice it's checked against the
`location' from the header, so this doesn't actually fix a
vulnerability.)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: allow Tainted construction to specify file/line

Unfortunately these default values are evaluated to constants at the
time of class definition, so we need some circumlocution.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Do not write out unchecked output in sites

In principle our downstreams should be able to cope with this.  But
maybe they haven't had the fixes, and dumping strange stuff into the
output file seems unfriendly.

So reimplement copyout as a function which reassembles the output line
from pieces, and checks that each Tainted word it is writing out has
been verified by someone to be OK.

As a side effect we normalise the whitespace including indentation.

We rename the input line variable in pline from `i' to `il', since it
contains possibly dangerous content.  This makes sure we caught all of
the places it is used.

With these changes, it is necessary to add a couple of explicit calls
to Tainted methods for otherwise-unused parameters, notably the group
name in location() definitions in user-provided site fragments, and
the (optional) email address in an ssh1 rsa key.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Introduce copyout() in pline()

These are all the places where we simply copy the input line to our
output.  We are going to do something more complicated in a moment, so
centralising this is useful.

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Check input file word syntax

make-secnet sites sometimes reads untrusted input.  And we copy it to
various output files, including secnet configuration files which have
a different lexical syntax and are particularly vulnerable to a
syntax stuffing/inadequate escaping attack.

In principle we could quote everything appropriately on output but a
actually we probably just want to check it since the syntax of all
these directives and their parameters is quite restricted.

In order to ensure that we catch everything, and that if we missed a
location we get a crash rather than a security vulnerability, we take
the following approach:

Each untrusted input word is wrapped up in a new Tainted object.  The
Tainted object has a number of methods for checking and returning
values which are suitable for various purposes.  But attempts to
simply print it (eg to an output file) are made to fail.

The Tainted object keeps track internally of whether it has been
checked.  This is going to be important in a moment.

Naive call sites use straightforward methods on w[N] to get checked
values for storage in their own data structures.

Knowledgeable use sites may call .raw() to get the unchecked value,
and .raw_mark_ok() if they know that the value is good (or are about
to do something which will definitely crash if not, so that a bad
value cannot escape).

Obviously storing the results of .raw() in a call site's data
structure would escape the taint checking.  So we don't do that unless
we have done the check ourselves.

Within the Tainted implication we really wanted an error monad.  Using
python exceptions for this looked like it was going to be too
abstruse.  So we open-code the monad with a conventional `ok' local
variable.  Each entrypoint returns using ._rtn() which can
double-check that no error has been lost.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Crash if complain() is called too late

Every call to complain() is supposed to occurs before the code in the
main program which checks `complaints'.  But maybe there is an
erroneous late call, or one may be introduced.  In this case it is
important to crash, because otherwise bad data might end up being
written into our output.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Introduce a couple of local variables

We are going to want these in a moment to avoid repeatedly referring
to the same w[] element.

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Use argparse rather than ad-hoc parser

This is much less ridiculous now.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest/t-basic: New test, with expected output

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Switch to python3

ipaddress is in the python3 stdlib.  python-future is not needed
either as it is aliases for things from the python3 stdlib.

We have to explicitly add python3 to Build-Depends now; it was
previously pulled in implicitly.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Makefile.in: clean: Remove __pycache__ too

Python3 genrates this.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Switch to `ipaddress' from `ipaddr'

ipaddress is available in python3 and ipaddr is not.

Code changes:
 - Change the imports and references to the module name
 - IPNetwork & IPAddress functions => ip_address & ip_network
 - There is no IPNetwork superclass so don't mention it in docstrings
 - collapse_address_list => collapse_addresses
 - There is no version parameter to ip_address; we have to
   switch on v ourselves and call IPv6Address or IPv4Address

Administrivia:
 - Update debian/control and INSTALL.
 - Remove references to ipaddr's licence.  ipaddress is under
   the same licence as python so does not need special mention.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Apply list() to keys in delempty

It is not permitted (in Python3) to modify a dictionary like this,
while iterating over keys().  We have to make a list of the keys,
copying them.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites etc.: Use unicode

We are going to want to switch to ipaddress from ipaddr, since
ipaddress is available in python3.  But ipaddress insists on unicode
strings, even in python2.  ipaddr doesn't mind them.

So make everything be unicode.  In particular: all of our literals and
all of our io streams.  We wrap up io.open(), which is a compatibility
thing from python-future.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

ipaddrset: Define __bool__ and make __nonzero__ an alias

Python3 calls __bool_.  Python2 calls __nonzero__.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Set .type in the `level' base class

We have one instance of directly this, the root node.  If it has an
error, the lack of the .type would cause a stack trace while trying to
print the error message.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Replace string.atol with int()

string.atol retured a long, I assume.  In python2, longs and ints are
distinct.  We could use long() here.  But that is not available in
python3.  Instead, write the python3 version already: just int.  We
can get an `int' that always produces longs from the python-future
module.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Fix calls to string.split and sring.join

These go away in python3.  They want us to use this daft objecty
syntax instead.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Abolish use of .has_key

This is deprecated and goes away in python3.  They want us to use this
`in' syntax instead.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Fix python path manipulation

This makes it possible to set PYTHONPATH to prefer the in-tree
modules.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Put parens around print() statements

This is part of the transition to python3.

In actual fact these are all error messages and should go to stderr
but I'm ot fixing that right now.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Move option parser to the front of the file

This means that we will be able to use information from the option
parser when creating our classes etc.  This will be useful as we are
going to support multiple output file versions etc.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Move input file reading further down the file

This separates it from the option parser, which I want to move and
rewrite.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Introduce a notion of listish types.

A property of such a listish type can be assigned multiple times, and
the values accumulate, and get reported as a list in the output
configuration.

Currently none are defined, so you can't see what this does.

Signed-off-by: Mark Wooding <mdw@distorted.org.uk>

make-secnet-sites: Introduce a superclass for the config types.

Somewhere to put common behaviour.  Not that there is any yet, so
there's no functional change.

Signed-off-by: Mark Wooding <mdw@distorted.org.uk>

mtest/t-userv: Check for dangerous parsing of late options

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest/t-userv: Break out `good'

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

tests: Dump logfile(s) of failing test(s)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

tests: Print subdir in summary output too

Now we have multiple subdirs the output might be interleaved.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

tests: Provide `recheck' to rerun fast tests

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Tidy up output from Makefile

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Break out diff-output

No functional change

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Provide run-mss

No caller yet

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest/t-userv: Check the expected output

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Set PYTHONBYTECODEBASE here too

This prevents ad-hoc manual runs from genrating unwanted cache files.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Set PYTHONHASHSEED

This will allow us to avoid test output being reordered due to hash
instability.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Wire up into toplevel Makefile

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Provide a makefile to run the tests

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test-common: Set PYTHONBYTECODEBASE to /dev/null

Python is not entirely reliable at figuring out when its .pyc files
are out of date, especially if you do something like
  git-rebase -i --exec 'make check-mtest' <commitish>

So squash the bytecode cache entirely.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test-common: Rename SECNET_TEST_BUILDDIR variable

No longer just stest.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test-common.make: Add missing dependencies on makefiles

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test-common.make: Fix hardcoded stest references

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

build system: Break out test-common.make

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

build system: make clean calls clean in stest

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

stest: Add missing test-common.tcl to DEPS

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

stest: Break out DEPS

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Honour MTEST_PYTHON

To allow running with different python versions.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Break out mss-run-userv

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: First test case

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test-common: Handle mtest correctly too

The default value for tmp nees to be right for mtest/ too.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

tests: Break out prefix_some_path

This incidentally fixes a bug: previously, we wrote PRELOAD rather
than LD_PRELOAD in one place, which meant that existing LD_PRELOADs
would be overwritten.  Now they no longer are.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

tests: Break out test-common.tcl

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Test files for make-secnet-sites userv mode

No test execution machinery yet.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

stest: Use proper builddir subdir as default tmp

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

.gitignore: ignore config.stamp.in too

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

stest: Use topbuilddir (now in common.make)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

stest: Use common.make and therefore our standard CFLAGS

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

stest/udp-preload: Fix some compiler warnings

These come up with our standard CFLAGS which we are erroneously not
using.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

build system: stest: Fix out-of-tree builds

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

build system: test-example: Fix out-of-tree builds

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

build system: Move srcdir setting out of common.make.in

This varies according to the cwd.  So for common.make.in it is always
the top-level.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

build system: Process test-example with autoconf

This makes configure make the directory during out-of-tree builds.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

stest: Rename from `test'

We want other tests too.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

build system: Rename stamp-h to config.stamp

This makes more sense and gets it out of the way of "st..." tab
completion which we are going to want in a momen.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

ipaddrset-test: Fix network with host bits

2001:23:24:: has 3x16 bits set, ie /48.  This was always wrong.

We need to fix this now because we are going to switch to ipaddress
from ipaddr, which actually checks this.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Add a missing dependency on the sites file

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Rerun tests only when deps changed

By touching the stamp file.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: New t-dyni-kex

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Beak out proc test-kex

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Slurp test-example/sites.conf and paste it in

This will enable us to edit this common config.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Drop redundant headers

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Fix copyright dates and error message

Also upgrade the licence to GPLv3+ like the rest of secnet.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Fix build dependencies so `make check' works in sbuild again

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Disconnect -j for check parallelism

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Wire into "make check"

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>