[SECNET REVIEW 000/227] New signature key arrangements

[Ian Jackson]

Someone very generously expressed an interest in reading this.
I think it is now ready for review.  You will be pleased to learn that
no-one asked for this as an emailed patchbomb.

The branch is at
  chiark:/u/ian/things/secnet
  git://git.chiark.greenend.org.uk/~ian/secnet.git
in
  refs/heads/wip.pubkeys.v1
and the commit range is
  4ab0e4ad32e169bfd9e3af465e5850ccb8520890
to
  2defa2bf837c1e8e00b6eff767fc3e6821c05da0

There is a fair amount of stuff here.  It comes in phases.
I have tried to describe the structure of the branch, below.

Caveats:

 - In general I have avoided squashing together work done at
   different times even if it is thematically related, or even
   if a later commit fixes a bug in an earlier one.  In the
   most blatant cases, the earlier commit will have a note.

 - I have often refrained from transposing things to organise them
   more correctly, so as to avoid having to squash or resolve
   conflicts.  Similar work may be quite far apart in the series.

 - There are two commits which ought to be inbound subtree merges but
   aren't.  I will turn them into proper git-subtree merges before
   pushing.  Additionally, there are two commits to subdirmk which I
   have not yet exported there.

 - There is not yet any code (nor a defined protocol) to allow a
   secnet to dynamically update its peer's idea of its keys.

The changes come in 19 phases:


BOOK I - PREPARATION

1. Build system fixes and improvements.

  subdirmk: break out subdirmk_target in generated Makefile
  subdirmk: Suppress MAKEFILE_TEMPLATES when running `make clean'
  build system: Cope if HEAD refers to a packed ref
  stest/udp-preload.c: Fix error handling of sun_prep
  stest/udp-preload: Be more relaxed about paths
  comprehensive-test: new script
  pretest-to-tested: Script for helping run comprehensive-test
  comprehensive-test: Test the stest/ and mtest/ directories
  build system, tests: Marginally speed some things up with -f
  stest: Fix breakage if nproc is not installed
  build system: Switch to using subdirmk's clean and cdeps
  build system: Add some missing clean and cdeps hooks
  comprehensive-test: Test that clean target works
  test-example: Replace a literal with $@
  test-example: Fix a dependency on Subdir.mk

2. Fixes and improvements to testing (or to support testing)

  secnet: Make stderr line buffered
  logfile: Log to stderr by default (`filename' key now optional)
  logfile: Remove redundant check for st->f
  logfile: Do not log differently with --nodetach
  logfile: New `prefix' option.
  stest: Use stderr, not tty, for logging
  stest: Use new `prefix' option.
  stest: Add debug logs
  dir-locals: Provide python-indent-offset too
  tests, mtest: Provide and use `prexec' for helping with debugging
  stest: Print the secnet runes we run
  stest: Print the special env vars we pass to secnet
  stest: Don't prefix ./ with another ./, in socktmp
  stest: Support not running secnet ourselves
  stest: Replace the call to `exit 1' with a new proc `finish'
  stest: Record the pids of of the secnets we spawn
  stest: Kill our child secnets when we call finish
  stest: Use `finish' for success exits
  stest: If one of our secnets dies, call the test a failure
  stest: Decode the slip packets that come via fake netlink
  stest: Check that received packet is as expected

3. Build system and tests performance improvements

  configure.ac: Drop checks for systems lacking stdint.h
  configure.ac: Drop AC_STDC_HEADERS
  configure.ac: Do not check for lack of standard headers
  autogen.sh: Write a comment about need for autoheader
  stest: Drop a redundant test

4. Bugfixes and preparatory cleanups

  make-secnet-sites: Tainted: Fix a lot of bad return values
  conffile: find_cl_if: fix fail_if_invalid==False
  rsa: emsa_pkcs1: Fix a message
  mtest/t-prefix: Drop redundant setting of seddery


BOOK II - NEW PUBLIC KEY HANDLING, NEW PROTOCOL

5. New scheme preliminaries: documentation, headers, types

  pubkey handling: Document key sets, id, etc. plan
  sigkey handling: Introduce sigkeyid type
  sigkey handling: define MAX_SIG_KEYS
  sigkey handling: Introduce serialt type

6. Adjust the signature scheme interface slightly

  sigpubkey/sigprivkey: Provide a hash_if
  sigpubkey/sigprivkey: Provide a dispose() method
  sigscheme: Interface for signature schemes
  util: pathprefix_template: New facility

7. Introduce new private key loading machinery in secnet

  privcache: New closure for signature key handling
  privcache: Use new pathprefix_template facility
  privcache: Better debug
  privcache: Remove some spurious \n in messages

8. Introduce base91s

  README: Introduce spec for "base91s" encoding
  B91 C DUMMY SYMLINK - WILL BE SUBTREE MERGE
  base91: CREDITS: Add information about C library
  base91: Build the C basE91 code with our own renaming
  base91: Patch the C version for our charset change
  base91: Link it into secnet
  BASE91 PYTHON DUMMY SYMLINK - WILL BE SUBTREE MERGE
  base91: CREDITS: Add information about Python library
  base91: Python: make `import' work

9. Public key file (sites file subset) parser in secnet

  Provide osdep.[ch], currently empty
  portability: Provide implementation of fmemopen
  pubkeys: Provide parser (and spec) for peer pubkeys files
  pubkeys: Provide ability to add extra action to KEYWORD
  pubkeys: Provide `fallback_skip' variable
  pubkeys: Introduce `pkgf' fallback groups
  pubkeys: Print file path in some log messages
  pubkeys: Move path and lno into a struct cloc
  pubkeys: Pass a cloc to loadpub
  pubkeys: Improve debug message for expected ENOENT
  pubkeys: Better debug logging for syntax errors
  pubkeys: Do not mind missing serial

10. In secnet, adjust rsa1 to provide the new interfaces

  rsa1: rsapriv_apply: Introduce macros for cfgfatal* and free
  rsa1: rsapriv_apply: Initialise local mpz's early
  rsa1: rsapriv_apply: Initialise st mpz's early
  rsa1: Break rsa_loadpriv_core out of rsapriv_apply
  rsa1: Provide dispose functions
  rsa1: Move "rsa-private" closure string into LDFATAL etc. macros
  rsa1: rsa_loadpriv_core: Make assume_valid do the frees
  rsa1: rsa_loadpriv_core: Always free b, and zero things we free
  rsa1: rsa_loadpriv_core: Introduce rsapriv_load_ctx
  rsa1: Wrap calls to keyfile_get* in a macro
  rsa1: Indirect postreadcheck via the context
  rsa1: Drop filename from two error messages
  rsa1: Provide rsa1_loadpriv function
  rsa: Do not do validity checks at runtime (in privcache load)
  rsa: sharing load_ctx: Code motion
  rsa: sharing load_ctx: Rename load_ctx
  rsa: sharing load_ctx: Move loc into common part
  privcache: Pass a cloc to loadpriv and use it
  rsa: sharing load_ctx: Introduce what
  rsa: Move FREE macro up
  rsa: Introduce RSAPUB_BNS and RSAPUB_APPLY_GETBN
  rsa: Break out rsa_loadpub_core
  rsa: Introduce LDPUBFATAL
  rsa: Provide RSAPUB_INIT_ST_BN and _CLEAR_
  rsa: Prepare rsa_loadpub_core for failure.
  rsa: Pass an additional maybe_loc to load_err
  rsa: Move load_err earlier
  rsa: De-indirect a lot of `loc's
  rsa: rsapub_dispose: Make passing 0 OK
  rsa: rsapub_dispose: Use RSAPUB_CLEAR_ST_BN
  rsa: rsa_loadpub_core: Make it take a load_ctx and use load_err
  rsa: LDPUBFATAL: Move & before lc into callers
  rsa: loadpriv: Use FREE
  rsa: Provide rsa1_loadpub and therefore rsa1 sigscheme
  rsa: Strip a couple of spurious newlines from pub key messages

11. Leftover cleanups from secnet sig key innards changes

  priv-cache etc.: private key algorithm is specified in key id
  rsa: Drop `unsup' parameter to load_err, verror, etc.
  privcache: Reorganise private key loading

12. Make secnet `site' use new sig key interfaces

  site: New SIGKEYS log class, at level INFO, enabled by default
  site: Introduce SETUP_SETHASH
  site: Move declaration/initialisation of `hash'
  NOTES: Add protocol elements for public key negotiation
  site: generate_msg: Introduce `privkey' variable
  site: Introduce `pubkey' variable (twice)
  site: Be able to use multiple private keys
  site: Read public peer keys from key file
  site: Update peer keys from ~update file
  site: Write an argument for the soundness of key file update
  site: key update soundness argument: deal with concurrency
  site: Reserve <peer-keys>~tmp for make-secnet-sites
  site: Remove a spurious \n in a message
  README: Document new semantics for `hash'
  site: Move defhash from setup to st variable
  pubkey handling: Call sethash when needed
  docs: Fix rsapriv/rsapub closure names
  docs: Move peer-keys documentation into a README file
  site: Better logging of key ids etc.
  site: Include key indices in log output

13. Finishing touches to secnet public key handling

  pubkeys: Introduce `fill' variable to make code prettier
  secnet: loadpub interface: Return a closure_t too
  secnet: Provide `make-public' verb
  vcfgfatal_maybefile: New suffix argument
  log: Provide struct cfgfile_log
  make-public: Use cfgfile_log, not system_log

14. Teach make-secnet-sites to write pubkeys files etc.

  make-secnet-sites: Deduplicate `complain'
  make-secnet-sites: Tolerate late `complain'
  make-secnet-sites: Write output to ...~tmp~ and rename
  make-secnet-sites: Tainted.name: take `what'
  make-secnet-sites: Tainted: Provide hexid and base91
  make-secnet-sites: set_property: Break out kw
  make-secnet-sites: set_property: Break out propname
  make-secnet-sites: set_property: Support "aliases"
  make-secnet-sites: pubkeys: Allow multiple
  make-secnet-sites: Provide --debug and debugrepr
  make-secnet-sites: Tainted: Provide some commented-out debug
  make-secnet-sites: Introduce FilterState
  make-secnet-sites: pline: Break up `copyout'
  make-secnet-sites: Provide base91s_encode and base91s_decode
  make-secnet-sites: Provide --output-version option
  make-secnet-sites: Allow properties to control output to sites
  make-secnet-sites: Provide ArgActionLambda
  make-secnet-sites: Provide `serial' property class
  make-secnet-sites: Prepare for multiple public key types
  make-secnet-sites: Support new `pub' directive
  make-secnet-sites: Make the `pub' key name primary
  make-secnet-sites: Do not write `pub' entries in v1 output
  make-secnet-sites: Do not use `copy' in pubkey forsites
  make-secnet-sites: Write rsa1 keys as `pub rsa1 ...' in >=v2
  make-secnet-sites: Handle `pub rsa1' properties specially
  make-secnet-sites: Add new pubkeys-dir option, and pubkey paths
  make-secnet-sites: New --pubkeys-install option
  make-secnet-sites: Without --pubkeys-install, write appropriate key
  make-secnet-sites: Support `serial'
  make-secnet-sites: Support `pkg' and `pkgf'
  make-secnet-sites: In v1 output, ignore keys for non-0 groups
  make-secnet-sites: Rework with new `inpub' class
  make-secnet-sites: Rework --pubkeys-MODE, provide --pubkeys-single
  make-secnet-sites: New --pubkeys-elide mode
  make-secnet-sites: Declare `pubkey' directive old

15. Tests for the work so far

  test-example: Rules for making a privkeys directory
  test-example: Make privkeys by default
  test-example: Add the inside.key to the privcache too
  test-example: Add a 2nd rsa1 key
  stest: Transfer pubkeys from test-example
  stest: Provide a global `hash' key
  stest: Honour new privkey() hash
  tests: Pass --output-version=1 to make-secnet-sites
  test-example: Provide a new-style public key set for site outside
  test-example: Make sites.conf variable, in principle
  test-example: Generate new style sites.conf
  test-example: Test new long-term key arrangements
  test-example: all-privkeys: Define, and indirect through, a variable
  stest: Fix an entry in &DEPS
  mtest: Use --pubkeys-elide instead of --output-version
  stest: Honour new sitesconf_hook
  stest/t-nonnego-oo: Use `make-public' verb


BOOK 3 - REMAINING PIECES

16. Provide make-secnet-sites --filter and test it

  make-secnet-sites: Introduce new OpMod classes
  make-secnet-sites: Refactor operational code into OpModes
  make-secnet-sites: Move sites file writing into OpBase
  make-secnet-sites: OpBase.read_in: save read lines
  make-secnet-sites: Provide explicit --conf mode option
  make-secnet-sites: Abolish now-obsolete `service' variable
  make-secnet-sites: OpBase.write_out: Tolerate sitesfile is None
  make-secnet-sites: OpBase.write_out: Move heading back into OpUserv
  make-secnet-sites: OpUserv: Rename sitesfile to outputfile
  make-secnet-sites: OpConf: Move positional_args to OpBase
  README.make-secnet-sites: Slightly restructure mode docs
  make-secnet-sites: Provide --filter mode
  mtest/t-filter: New test for --filter mode

17. Leftover cleanups

  test-common: Use $(wildcard ) rather than $(shell echo )
  test-common: Provide for compatibility tests

18. Test compatibility against old secnet

  test-common: Use $(wildcard ) rather than $(shell echo )
  test-common: Framework for compatibility tests
  stest: Machinery for runing an old secnet if a test wants
  sest/t-Cnonnego-on: New test
  stest: Make it possible to have kex initiated by `outside'
  stest/t-Cnonnego-onr: New test

19. Provide `load-private', new-style fixed private key loading

  privcache: uncached_get: Introduce `path' local
  privcache: uncached_get: Introduce `defhash' local
  privcache: uncached_get: Introduce `databuf' local
  privcache: Transpose scheme lookup and file opening
  privcache: Break out uncached_load_file
  CL_GET_STR_ARG: Move into util.h
  privcache; uncached_load_file: Introduce error_out
  privcache: Regularise error handling in missing defhash case
  privcache: Minor style fixes
  secnet: loadpriv interface: Return a closure_t too
  privcache: Break out DEFAULT_MAXPRIV_BYTES
  secnet: Provide `load-private' verb
  load-private: Use cfgfile_log, not system_log
  stest: Add test for load-private

-- 
Ian Jackson <ijackson at chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.