Makefile.in: clean: Remove __pycache__ too

Python3 genrates this.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Switch to `ipaddress' from `ipaddr'

ipaddress is available in python3 and ipaddr is not.

Code changes:
 - Change the imports and references to the module name
 - IPNetwork & IPAddress functions => ip_address & ip_network
 - There is no IPNetwork superclass so don't mention it in docstrings
 - collapse_address_list => collapse_addresses
 - There is no version parameter to ip_address; we have to
   switch on v ourselves and call IPv6Address or IPv4Address

Administrivia:
 - Update debian/control and INSTALL.
 - Remove references to ipaddr's licence.  ipaddress is under
   the same licence as python so does not need special mention.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Apply list() to keys in delempty

It is not permitted (in Python3) to modify a dictionary like this,
while iterating over keys().  We have to make a list of the keys,
copying them.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites etc.: Use unicode

We are going to want to switch to ipaddress from ipaddr, since
ipaddress is available in python3.  But ipaddress insists on unicode
strings, even in python2.  ipaddr doesn't mind them.

So make everything be unicode.  In particular: all of our literals and
all of our io streams.  We wrap up io.open(), which is a compatibility
thing from python-future.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

ipaddrset: Define __bool__ and make __nonzero__ an alias

Python3 calls __bool_.  Python2 calls __nonzero__.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Set .type in the `level' base class

We have one instance of directly this, the root node.  If it has an
error, the lack of the .type would cause a stack trace while trying to
print the error message.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Replace string.atol with int()

string.atol retured a long, I assume.  In python2, longs and ints are
distinct.  We could use long() here.  But that is not available in
python3.  Instead, write the python3 version already: just int.  We
can get an `int' that always produces longs from the python-future
module.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Fix calls to string.split and sring.join

These go away in python3.  They want us to use this daft objecty
syntax instead.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Abolish use of .has_key

This is deprecated and goes away in python3.  They want us to use this
`in' syntax instead.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Fix python path manipulation

This makes it possible to set PYTHONPATH to prefer the in-tree
modules.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Put parens around print() statements

This is part of the transition to python3.

In actual fact these are all error messages and should go to stderr
but I'm ot fixing that right now.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Move option parser to the front of the file

This means that we will be able to use information from the option
parser when creating our classes etc.  This will be useful as we are
going to support multiple output file versions etc.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Move input file reading further down the file

This separates it from the option parser, which I want to move and
rewrite.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

make-secnet-sites: Introduce a notion of listish types.

A property of such a listish type can be assigned multiple times, and
the values accumulate, and get reported as a list in the output
configuration.

Currently none are defined, so you can't see what this does.

Signed-off-by: Mark Wooding <mdw@distorted.org.uk>

make-secnet-sites: Introduce a superclass for the config types.

Somewhere to put common behaviour.  Not that there is any yet, so
there's no functional change.

Signed-off-by: Mark Wooding <mdw@distorted.org.uk>

mtest/t-userv: Check for dangerous parsing of late options

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest/t-userv: Break out `good'

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

tests: Dump logfile(s) of failing test(s)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

tests: Print subdir in summary output too

Now we have multiple subdirs the output might be interleaved.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

tests: Provide `recheck' to rerun fast tests

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Tidy up output from Makefile

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Break out diff-output

No functional change

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Provide run-mss

No caller yet

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest/t-userv: Check the expected output

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Set PYTHONBYTECODEBASE here too

This prevents ad-hoc manual runs from genrating unwanted cache files.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Set PYTHONHASHSEED

This will allow us to avoid test output being reordered due to hash
instability.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Wire up into toplevel Makefile

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Provide a makefile to run the tests

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test-common: Set PYTHONBYTECODEBASE to /dev/null

Python is not entirely reliable at figuring out when its .pyc files
are out of date, especially if you do something like
  git-rebase -i --exec 'make check-mtest' <commitish>

So squash the bytecode cache entirely.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test-common: Rename SECNET_TEST_BUILDDIR variable

No longer just stest.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test-common.make: Add missing dependencies on makefiles

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test-common.make: Fix hardcoded stest references

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

build system: Break out test-common.make

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

build system: make clean calls clean in stest

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

stest: Add missing test-common.tcl to DEPS

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

stest: Break out DEPS

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Honour MTEST_PYTHON

To allow running with different python versions.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Break out mss-run-userv

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: First test case

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test-common: Handle mtest correctly too

The default value for tmp nees to be right for mtest/ too.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

tests: Break out prefix_some_path

This incidentally fixes a bug: previously, we wrote PRELOAD rather
than LD_PRELOAD in one place, which meant that existing LD_PRELOADs
would be overwritten.  Now they no longer are.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

tests: Break out test-common.tcl

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtest: Test files for make-secnet-sites userv mode

No test execution machinery yet.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

stest: Use proper builddir subdir as default tmp

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

.gitignore: ignore config.stamp.in too

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

stest: Use topbuilddir (now in common.make)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

stest: Use common.make and therefore our standard CFLAGS

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

stest/udp-preload: Fix some compiler warnings

These come up with our standard CFLAGS which we are erroneously not
using.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

build system: stest: Fix out-of-tree builds

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

build system: test-example: Fix out-of-tree builds

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

build system: Move srcdir setting out of common.make.in

This varies according to the cwd.  So for common.make.in it is always
the top-level.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

build system: Process test-example with autoconf

This makes configure make the directory during out-of-tree builds.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

stest: Rename from `test'

We want other tests too.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

build system: Rename stamp-h to config.stamp

This makes more sense and gets it out of the way of "st..." tab
completion which we are going to want in a momen.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

ipaddrset-test: Fix network with host bits

2001:23:24:: has 3x16 bits set, ie /48.  This was always wrong.

We need to fix this now because we are going to switch to ipaddress
from ipaddr, which actually checks this.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Add a missing dependency on the sites file

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Rerun tests only when deps changed

By touching the stamp file.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: New t-dyni-kex

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Beak out proc test-kex

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Slurp test-example/sites.conf and paste it in

This will enable us to edit this common config.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Drop redundant headers

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Fix copyright dates and error message

Also upgrade the licence to GPLv3+ like the rest of secnet.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Fix build dependencies so `make check' works in sbuild again

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Disconnect -j for check parallelism

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Wire into "make check"

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Use test/d-* instead of test/tmp for everything

Now it is actually ok to run multiple tests in parallel.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Makefile rune for `check'

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Move sockets in a subdirectory

They need to be not world-accessible and this is the easiest way.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Specify the LD_PRELOAD etc.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Split "invoke" up

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Use $(CC) for link, provide clean target

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Build system

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Makefiles: Break some settings out into common.make

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Provide recvfrom

Now we can do a key exchange!

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Fix inet_ntop calling convention

inet_ntop has a weird error return protocol.  And our code for calling
it never worked properly because we didn't strip the leading directory
names from the bound socket name.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Proxy udp packets

We must change the config to specify localhost addrs explicitly,
because we don't implement any special logic for IN[6]ADDR_ANY.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Cope with -ve fds

Eg, Tcl passes -1 to close (!)

 #0  0x00007f62949883ca in close (fd=-1) at udp-preload.c:207
 #1  0x00007f6294719362 in Tcl_FinalizeNotifier () from /usr/lib/x86_64-linux-gnu/libtcl8.6.so

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Run secnet under strace

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Provide sendto

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Prepare for wrapping fns that don't return int

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Introduce sun_prep

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Consolidate program name in argl

This avoids pratting about with the weird way execl takes its
arguments.  No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Provide close

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Provide getsockname

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Provide setsockopt

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

iaddr_to_string: Do not falsely claim bad addrs are scoped IPv6

In particular, if the AF is neither INET nor INET6, adns_addr2text
quite rightly fails with EAFNOSUPPORT.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Remove now-obsolete `bound'

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Fix binding, unlink

Avoids EADDRINUSE from the real bind(2).

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Fix binding

inet_ntop needs just the addr field.  How "convenient".

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: More actual implementation

Now needs to be invoked like this

  UDP_PRELOAD_DIR=test/tmp LD_PRELOAD=test/udp-preload.so test/invoke

It binds to test/tmp/...

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Beginning of actual implementation

  gcc -Wall -D_REENTRANT -fPIC -c udp-preload.c && \
  ld -shared -soname foo.so.1 udp-preload.o -o udp-preload.so -ldl -lc

produces a library with which

  LD_PRELOAD=test/udp-preload.so test/invoke

produces various complaints like

  udp (test/tmp/outside.conf:19): setsockopt(,IPV6_V6ONLY,&1,): Operation not supported
  udp (test/tmp/inside.conf:19): socket [::]:16913 experiencing some trouble transmitting IPv6 (to [::1]:16900): Bad file descriptor

This is progress.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Proof of concept wrapping (2)

  gcc -D_REENTRANT -fPIC -c udp-preload.c && \
  ld -shared -soname foo.so.1 udp-preload.o -o udp-preload.so -ldl -lc

produces a library with which

  LD_PRELOAD=test/udp-preload.so test/invoke

still works.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Proof of concept wrapping

  gcc -D_REENTRANT -fPIC -c udp-preload.c -ldl -lc && \
  ld -shared -soname foo.so.1 udp-preload.o -o udp-preload.so

produces a library which makes secnet go

  secnet fatal error: Failed to initialise ADNS: Message too long

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: udp-preload: Import libauthbind.c

This is from authbind 2.1.2 64b7841344fcc3cc.  It is GPLv2+ and
my own copyright so no licence problem.  I'm going to hack it up into
what is needed here.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Use dedicated tmp directory in variable

We are going to want to run multiple tests at once, so we'll need
this.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: if things aren't working time out rather than waiting forever

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: expect initial ping packet to come back on inside, for now

We are pinging the remote secnet, which responds without sending the
packet to its host (ie, "outside.r" in our terms).

We can improve this later if we care.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: send an initial ping packet

The hex data came from
  ping -I secnet-test-i 172.18.232.1
  tcpdump -wt -isecnet-test-i
  tcpdump -rt -x

We send it twice because the first causes a key exchange and then gets
lost.

The result is that the script crashes with
  inside rx'd!

This is because we are pinging the remote secnet, which responds
without sending the packet to its host (ie, "outside.r" in our terms).

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: use chiark-tcl-hbytes rather than ad-hoc \x quoting

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

test: Fix fd plumbing

In a background process, stdin defaults to /dev/null, so we need to do
a little dance.  <&0 looks like it ought to work according to the bash
manual and does indeed work in bash, but not in dash.  This
construction seems to work in both.

Tested with:
  date | sh -exc 'cat <&0 >t &'

SuS says:
  If job control is disabled (see set, -m), the standard input for an
  asynchronous list, before any explicit redirections are performed,
  shall be considered to be assigned to a file that has the same
  properties as /dev/null. This shall not happen if job control is
  enabled. In all cases, explicit redirection of standard input shall
  override this activity.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>