make-secnet-sites: Introduce copyout() in pline()

These are all the places where we simply copy the input line to our
output.  We are going to do something more complicated in a moment, so
centralising this is useful.

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Check input file word syntax

make-secnet sites sometimes reads untrusted input.  And we copy it to
various output files, including secnet configuration files which have
a different lexical syntax and are particularly vulnerable to a
syntax stuffing/inadequate escaping attack.

In principle we could quote everything appropriately on output but a
actually we probably just want to check it since the syntax of all
these directives and their parameters is quite restricted.

In order to ensure that we catch everything, and that if we missed a
location we get a crash rather than a security vulnerability, we take
the following approach:

Each untrusted input word is wrapped up in a new Tainted object.  The
Tainted object has a number of methods for checking and returning
values which are suitable for various purposes.  But attempts to
simply print it (eg to an output file) are made to fail.

The Tainted object keeps track internally of whether it has been
checked.  This is going to be important in a moment.

Naive call sites use straightforward methods on w[N] to get checked
values for storage in their own data structures.

Knowledgeable use sites may call .raw() to get the unchecked value,
and .raw_mark_ok() if they know that the value is good (or are about
to do something which will definitely crash if not, so that a bad
value cannot escape).

Obviously storing the results of .raw() in a call site's data
structure would escape the taint checking.  So we don't do that unless
we have done the check ourselves.

Within the Tainted implication we really wanted an error monad.  Using
python exceptions for this looked like it was going to be too
abstruse.  So we open-code the monad with a conventional `ok' local
variable.  Each entrypoint returns using ._rtn() which can
double-check that no error has been lost.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Crash if complain() is called too late

Every call to complain() is supposed to occurs before the code in the
main program which checks `complaints'.  But maybe there is an
erroneous late call, or one may be introduced.  In this case it is
important to crash, because otherwise bad data might end up being
written into our output.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Introduce a couple of local variables

We are going to want these in a moment to avoid repeatedly referring
to the same w[] element.

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Use argparse rather than ad-hoc parser

This is much less ridiculous now.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest/t-basic: New test, with expected output

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Switch to python3

ipaddress is in the python3 stdlib.  python-future is not needed
either as it is aliases for things from the python3 stdlib.

We have to explicitly add python3 to Build-Depends now; it was
previously pulled in implicitly.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Makefile.in: clean: Remove __pycache__ too

Python3 genrates this.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Switch to `ipaddress' from `ipaddr'

ipaddress is available in python3 and ipaddr is not.

Code changes:
 - Change the imports and references to the module name
 - IPNetwork & IPAddress functions => ip_address & ip_network
 - There is no IPNetwork superclass so don't mention it in docstrings
 - collapse_address_list => collapse_addresses
 - There is no version parameter to ip_address; we have to
   switch on v ourselves and call IPv6Address or IPv4Address

Administrivia:
 - Update debian/control and INSTALL.
 - Remove references to ipaddr's licence.  ipaddress is under
   the same licence as python so does not need special mention.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Apply list() to keys in delempty

It is not permitted (in Python3) to modify a dictionary like this,
while iterating over keys().  We have to make a list of the keys,
copying them.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites etc.: Use unicode

We are going to want to switch to ipaddress from ipaddr, since
ipaddress is available in python3.  But ipaddress insists on unicode
strings, even in python2.  ipaddr doesn't mind them.

So make everything be unicode.  In particular: all of our literals and
all of our io streams.  We wrap up io.open(), which is a compatibility
thing from python-future.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

ipaddrset: Define __bool__ and make __nonzero__ an alias

Python3 calls __bool_.  Python2 calls __nonzero__.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Set .type in the `level' base class

We have one instance of directly this, the root node.  If it has an
error, the lack of the .type would cause a stack trace while trying to
print the error message.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Replace string.atol with int()

string.atol retured a long, I assume.  In python2, longs and ints are
distinct.  We could use long() here.  But that is not available in
python3.  Instead, write the python3 version already: just int.  We
can get an `int' that always produces longs from the python-future
module.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Fix calls to string.split and sring.join

These go away in python3.  They want us to use this daft objecty
syntax instead.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Abolish use of .has_key

This is deprecated and goes away in python3.  They want us to use this
`in' syntax instead.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Fix python path manipulation

This makes it possible to set PYTHONPATH to prefer the in-tree
modules.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Put parens around print() statements

This is part of the transition to python3.

In actual fact these are all error messages and should go to stderr
but I'm ot fixing that right now.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Move option parser to the front of the file

This means that we will be able to use information from the option
parser when creating our classes etc.  This will be useful as we are
going to support multiple output file versions etc.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Move input file reading further down the file

This separates it from the option parser, which I want to move and
rewrite.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Introduce a notion of listish types.

A property of such a listish type can be assigned multiple times, and
the values accumulate, and get reported as a list in the output
configuration.

Currently none are defined, so you can't see what this does.

Signed-off-by: Mark Wooding <mdw@distorted.org.uk>

make-secnet-sites: Introduce a superclass for the config types.

Somewhere to put common behaviour.  Not that there is any yet, so
there's no functional change.

Signed-off-by: Mark Wooding <mdw@distorted.org.uk>

mtest/t-userv: Check for dangerous parsing of late options

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest/t-userv: Break out `good'

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

tests: Dump logfile(s) of failing test(s)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

tests: Print subdir in summary output too

Now we have multiple subdirs the output might be interleaved.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

tests: Provide `recheck' to rerun fast tests

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Tidy up output from Makefile

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Break out diff-output

No functional change

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Provide run-mss

No caller yet

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest/t-userv: Check the expected output

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Set PYTHONBYTECODEBASE here too

This prevents ad-hoc manual runs from genrating unwanted cache files.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Set PYTHONHASHSEED

This will allow us to avoid test output being reordered due to hash
instability.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Wire up into toplevel Makefile

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Provide a makefile to run the tests

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test-common: Set PYTHONBYTECODEBASE to /dev/null

Python is not entirely reliable at figuring out when its .pyc files
are out of date, especially if you do something like
  git-rebase -i --exec 'make check-mtest' <commitish>

So squash the bytecode cache entirely.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test-common: Rename SECNET_TEST_BUILDDIR variable

No longer just stest.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test-common.make: Add missing dependencies on makefiles

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test-common.make: Fix hardcoded stest references

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

build system: Break out test-common.make

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

build system: make clean calls clean in stest

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

stest: Add missing test-common.tcl to DEPS

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

stest: Break out DEPS

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Honour MTEST_PYTHON

To allow running with different python versions.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Break out mss-run-userv

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: First test case

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test-common: Handle mtest correctly too

The default value for tmp nees to be right for mtest/ too.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

tests: Break out prefix_some_path

This incidentally fixes a bug: previously, we wrote PRELOAD rather
than LD_PRELOAD in one place, which meant that existing LD_PRELOADs
would be overwritten.  Now they no longer are.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

tests: Break out test-common.tcl

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Test files for make-secnet-sites userv mode

No test execution machinery yet.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

stest: Use proper builddir subdir as default tmp

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

.gitignore: ignore config.stamp.in too

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

stest: Use topbuilddir (now in common.make)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

stest: Use common.make and therefore our standard CFLAGS

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

stest/udp-preload: Fix some compiler warnings

These come up with our standard CFLAGS which we are erroneously not
using.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

build system: stest: Fix out-of-tree builds

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

build system: test-example: Fix out-of-tree builds

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

build system: Move srcdir setting out of common.make.in

This varies according to the cwd.  So for common.make.in it is always
the top-level.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

build system: Process test-example with autoconf

This makes configure make the directory during out-of-tree builds.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

stest: Rename from `test'

We want other tests too.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

build system: Rename stamp-h to config.stamp

This makes more sense and gets it out of the way of "st..." tab
completion which we are going to want in a momen.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

ipaddrset-test: Fix network with host bits

2001:23:24:: has 3x16 bits set, ie /48.  This was always wrong.

We need to fix this now because we are going to switch to ipaddress
from ipaddr, which actually checks this.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Add a missing dependency on the sites file

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Rerun tests only when deps changed

By touching the stamp file.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: New t-dyni-kex

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Beak out proc test-kex

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Slurp test-example/sites.conf and paste it in

This will enable us to edit this common config.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Drop redundant headers

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Fix copyright dates and error message

Also upgrade the licence to GPLv3+ like the rest of secnet.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Fix build dependencies so `make check' works in sbuild again

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Disconnect -j for check parallelism

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Wire into "make check"

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Use test/d-* instead of test/tmp for everything

Now it is actually ok to run multiple tests in parallel.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Makefile rune for `check'

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Move sockets in a subdirectory

They need to be not world-accessible and this is the easiest way.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Specify the LD_PRELOAD etc.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Split "invoke" up

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Use $(CC) for link, provide clean target

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Build system

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Makefiles: Break some settings out into common.make

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Provide recvfrom

Now we can do a key exchange!

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Fix inet_ntop calling convention

inet_ntop has a weird error return protocol.  And our code for calling
it never worked properly because we didn't strip the leading directory
names from the bound socket name.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Proxy udp packets

We must change the config to specify localhost addrs explicitly,
because we don't implement any special logic for IN[6]ADDR_ANY.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Cope with -ve fds

Eg, Tcl passes -1 to close (!)

 #0  0x00007f62949883ca in close (fd=-1) at udp-preload.c:207
 #1  0x00007f6294719362 in Tcl_FinalizeNotifier () from /usr/lib/x86_64-linux-gnu/libtcl8.6.so

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Run secnet under strace

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Provide sendto

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Prepare for wrapping fns that don't return int

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Introduce sun_prep

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Consolidate program name in argl

This avoids pratting about with the weird way execl takes its
arguments.  No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Provide close

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Provide getsockname

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Provide setsockopt

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

iaddr_to_string: Do not falsely claim bad addrs are scoped IPv6

In particular, if the AF is neither INET nor INET6, adns_addr2text
quite rightly fails with EAFNOSUPPORT.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Remove now-obsolete `bound'

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Fix binding, unlink

Avoids EADDRINUSE from the real bind(2).

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Fix binding

inet_ntop needs just the addr field.  How "convenient".

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: More actual implementation

Now needs to be invoked like this

  UDP_PRELOAD_DIR=test/tmp LD_PRELOAD=test/udp-preload.so test/invoke

It binds to test/tmp/...

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Beginning of actual implementation

  gcc -Wall -D_REENTRANT -fPIC -c udp-preload.c && \
  ld -shared -soname foo.so.1 udp-preload.o -o udp-preload.so -ldl -lc

produces a library with which

  LD_PRELOAD=test/udp-preload.so test/invoke

produces various complaints like

  udp (test/tmp/outside.conf:19): setsockopt(,IPV6_V6ONLY,&1,): Operation not supported
  udp (test/tmp/inside.conf:19): socket [::]:16913 experiencing some trouble transmitting IPv6 (to [::1]:16900): Bad file descriptor

This is progress.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Proof of concept wrapping (2)

  gcc -D_REENTRANT -fPIC -c udp-preload.c && \
  ld -shared -soname foo.so.1 udp-preload.o -o udp-preload.so -ldl -lc

produces a library with which

  LD_PRELOAD=test/udp-preload.so test/invoke

still works.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Proof of concept wrapping

  gcc -D_REENTRANT -fPIC -c udp-preload.c -ldl -lc && \
  ld -shared -soname foo.so.1 udp-preload.o -o udp-preload.so

produces a library which makes secnet go

  secnet fatal error: Failed to initialise ADNS: Message too long

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>