config, wip macro, generete fields with SKLs

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

config, wip macro, generetes some code

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

config, wip macro

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

config, wip macro

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

config, wip macro

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

config, wip macro

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

config, break out var section list

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

config, wip before break out var section list

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

macros wip

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

macros wip

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

macros skeleton

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

wip resolve, comment out for macro work, for revert

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

wip resolve

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

wip resolve

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

wip resolve

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

wip resolve

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

wip resolve

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

document and parse limit section headings

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

wip parsing, tolerate hyphens in config

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

wip parsing, record loc

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

wip parsing, record loc

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

wip parsing

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

wip parsing

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

wip parsing

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

wip parsing

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

wip parsing, shuffle

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

wip parsing

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

wip parsing

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

README.config: Further restrict server name

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

README.config: fix to match Python implementation

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

wip impl config reading, runs

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

wip impl config reading, compiles

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

wip impl config reading

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

README.config: clarifications

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

wip optiona nnd config parser

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Import docs from Python program

We are aiming to be compatible with this.

Copied these files from
  d5100ea6b9bb1d0d858c5475469f9cc4e3200b3a
verbatim.  We'll free to hack them here, though.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

dependencies

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

initial commit, cargo init + edits + licence

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Add missing cpio dependency

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Merge pull request #1 from mavuramd/master

Implement clean in make

Fix the clean target

Implement clean in make

mtu: Discard over-mtu packets

There is no authentication of packet data.  Bad captive portals
etc. can send garbage bad enough to overflow the mtu and cause the
client to crash.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

mtu: Add mtu parameter to SlipStreamDecoder and start_ipif

These don't take a config, so we must pass this as a parameter.
Change all call sites.  Not used yet so no functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

config: Make c.mtu always be an integer, not a string

We are going to want to use this other than just to feed to ipif.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Create LICENSE

Initial commit

hippotat: Convert an explicitly configured URL to ASCII.

The conversion is rather annoying, and not especially efficient, but it
does the job.  In particular, it works better than leaving the URL as
`str', because that results in a `TypeError' later.

Signed-off-by: Mark Wooding <mdw@distorted.org.uk>

hippotat: Don't set an explicit `Content-Length'.

It turns out (a) that the existing Twisted machinery already sets a
`Content-Length' header on `POST' submissions, (b) that it's too stupid
to notice that the caller's already set one, and (c) that nginx rejects
`POST' requests with duplicate `Content-Length' headers with `400 Bad
Request', which turns out to be surprisingly hard to debug.

Signed-off-by: Mark Wooding <mdw@distorted.org.uk>

hippotatlib/__init__.py: Make `vaddr' be an actual address object.

Otherwise `route' won't actually match it correctly.

Signed-off-by: Mark Wooding <mdw@distorted.org.uk>

ResponseConsumer: tolerate daft exception PotentialDataLoss

The docs for this say:

  PotentialDataLoss may be raised by a transfer encoding decoder's
  noMoreData method to indicate that it cannot be determined if the
  entire response body has been delivered.  This only occurs when
  making requests to HTTP servers which do not set Content-Length or a
  Transfer-Encoding in the response because in this case the end of
  the response is indicated by the connection being closed, an event
  which may also be due to a transient network problem or other error.

[1] https://twistedmatrix.com/documents/16.6.0/api/twisted.web.http.PotentialDataLoss.html retrieved 22.8.2017

If the origin server (or proxy) chooses to use `Connection: close' and
not to provide a Content-Length, then this exception will always be
raised.  hippotatd uses Twisted's http server so does not trigger
this, but there might be an (intercepting) proxy.  If so, then
hippotat will simply not work.

Additionally, the statement in the Twisted documentation that
connection close might be due to a "transient problem or other error"
is not correct.  TCP is able to distinguish a graceful shutdown from
an ungraceful close, and origin servers ought to arrange that they do
not inappropriately perform a graceful close in error situations.

Of course a badly-implemented proxy might get this wrong, but in that
case a general web client would have no way to detect this situation
if it wants to function at all.  So certainly turning this situation
into an exception by default seems wrong.

For hippotat, ignoring the potential problem hippotat just means
potentially processing a truncated response, which might mean lost
packets (which should be retransmitted) or truncated packets (which
will be detected by the IP header length field, and retransmitted).

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

ResponseConsumer: break out connectionLostOK into superclass

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

ResponseConsumer: move _resp initialisation to superclass constructor (nfc)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

ResponseConsumer: pass resp to superclass constructor (nfc)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

ResponseConsumer: pass resp to constructor (nfc)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

do not crash on unexpected connectionLost

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

ensure mtu is in the ipif substitution set

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

report success after failure

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

replace plaintext secret transmission with time-limited hmac-based bearer tokens

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

rename "password" to "secret" everywhere

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

document that nonce-based authentication is for the future

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

better document routing assistance

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

hippotatd: do not crash on some unexpected requests

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

process putatives: Introduce dbg()

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

process putatives: Properly report regexp mismatch ignores

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

process putatives: Fix processing of [<servername> <client>] sections

Convert only the client part to an ip address.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

process putatives: Add many log_debug_config calls

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

process putatives: break out log_ignore

No significant functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

do actually process server-specific [<server> LIMIT] sections

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

better message from client on http errors

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

daemon: record the deferred for a request in the queue, too

So we can check nf.called, rather than request.finished.  The latter
can be false even after the request has failed and is unfinishable.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

config: Abolish use of ConfigParser's DEFAULT feature

ConfigParser provides no way to find out which sections a key is
_really_ in: in all of its views, keys in DEFAULT appear aliased in
all sections.

So instead we use a COMMON section, and implement the defaulting
ourselves.  This involves:

* Changing the cfg.get[int] calls to use cfg_search (these are
  the things which implictly use DEFAULT and now need to be
  explicit).
* Changing all our explicit references to and definitions of
  DEFAULT options to COMMON.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

config: replace open-coded cfg.get[int]

We are going to want to do something more sophisticated.
Centralise things.  No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

move cfg_get_raw into hippotatlib

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

rename cfg_process_common to cfg_process_general

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

config reading: log correct filenames

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

hippotatd: catch errors from request.finish in _req_cancel

It is not completely clear when it is necessary to call
request.finish.  Doing so when it is not necessary has a tendency to
throw an exception.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

logging: if logevent_is_boringtwisted fails, print why when we print the exception

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

syslog handling: print tracebacks when logging failure objects

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

fix crash on discarding packets

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

fix typo

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

ipif: set interface names

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

debian: Recommend userv-utils, now that it has sane ipif

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

config: fix client config (no SERVER)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

config: a bit more debugging

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

cope with jesse's logger, which lacks --id=VALUE

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

fix pidfile handling

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

suppress boring twisted messages to syslog, too

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

fix exit status

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

fix cfg_process_saddrs ServerAddr

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

logevent_is_boringtwisted: break out as a function from LogNotBoringTwisted (nfc)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

cope with jesse's signal module (which lacks sig.name)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

init script fixes

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

config dir reading fix

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

packaging tidy

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

packaging fixes

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

packaging fixes - really now it installs on xenophobe

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

packaging fixes - now it installs on xenophobe

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

config: better examples/docs (update to how things are now)

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>

config: sort out directory handling

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>