.SH DESCRIPTION
.B innduct
implements NNTP peer-to-peer news transmission including the streaming
-extensions, for sending news articles to a remote site.
+extensions, for sending news articles to a remote site. It is
+intended as a replacement for
+.B innfeed
+or
+.BR nntpsend
+and
+.BR innxmit .
You need to run one instance of innduct for each peer site. innduct
-manages its interaction with innd, including flushing the feed as
-appropriate, etc., so that articles are transmitted quickly, and
-manages the retransmission of its own backlog. innduct includes the
-locking necessary to avoid multiple simutaneous invocations.
+manages its interaction with
+.BR innd ,
+including flushing the feed as appropriate, etc., so that articles are
+transmitted quickly, and manages the retransmission of its own
+backlog. innduct includes the locking necessary to avoid multiple
+simutaneous invocations.
By default, innduct reads the default feedfile corresponding to
the site
.I site
-(is
+(ie
.IR pathoutgoing / site )
and feeds it via NNTP, streaming if possible, to the host
.IR fqdn .
-
If
.I fqdn
is not specified, it defaults to
error messages) are sent to syslog (facility
.BR news ).
-The best way to run innduct is probably to periodically invoke innduct
-for each feed (e.g. from cron), passing innduct it the
+The best way to run innduct is probably to periodically invoke it
+for each feed (e.g. from cron), passing the
.B \-q
-option to arrange that it silently exits if an innduct is already
-running for that site.
-.SH INNDUCT VS INNFEED/NNTPSEND/INNXMIT
-.TP
-.B innfeed
-does roughly the same thing as innduct. However, the way it receives
-information from innd can result in articles being lost (not offered
-to peers) if innfeed crashes for any reason. This is an inherent
-defect in the innd channel feed protocol. innduct uses a file feed,
-constantly "tailing" the feed file, and where implemented uses
-.BR inotify (2)
-to reduce the latency which would come from having to constantly poll
-the feed file. innfeed is capable of feeding multiple peers from a
-single innfeed instance, whereas each innduct process handles exactly
-one peer. innduct is much smaller and simpler, at <4kloc to innfeed's
-25kloc. innfeed needs a separate wrapper script or similar
-infrastructure (of which there is an example in its manpage), whereas
-innduct can be run directly and doesn't need help from shell scripts.
-.TP
-.B nntpsend
-processes feed files in batch mode. That is, you have to periodically
-invoke nntpsend, and when you do, the feed is flushed and articles
-which arrived before the flush are sent to the peer. This introduces
-a batching delay, and also means that the NNTP connection to the peer
-needs to be remade at each batch. nntpsend (which uses innxmit)
-cannot make use of multiple connections to a single peer site.
-However, nntpsend can be left to find automatically which sites need
-feeding by looking in
-.IR pathoutgoing .
-.TP
-.B innxmit
-is the actual NNTP feeder program used by nntpsend.
+option to arrange that innduct silently exits if an instance is
+already running for that site.
.SH GENERAL OPTIONS
.TP
-.BR \-f | \-\-feedfile= \fIfeedfile\fR
-Specifies
-.IR feedfile .
-If the specified value ends in a
-.B /
-it is taken as a directory to use as if it were
-.I pathoutgoing
-and the actual feed file used is
-.IR specified_feedfile / site .
+.BR \-f | \-\-feedfile= \fIDIR\fR / |\fIPATH\fR
+Specifies the
+.I feedfile
+to read, and indirectly specifies the paths to
+be used for various associated files (see FILES, below).
+If specified as
+.IB DIR /
+it is taken as a directory to use, and the actual feed file used is
+.IR path / site .
+If
+.I PATH
+or
+.I DIR
+does not start with a
+.BR / ,
+it is taken to be relative to
+.IR pathoutgoing
+from inn.conf.
+The default is
+.IR site .
.TP
.BR \-q | \-\-quiet-multiple
Makes innduct silently exit (with status 0) if another innduct holds
.TP
.BR \-\-no-daemon
Do not daemonise. innduct runs in the foreground and all messages
-(including all debug messages) are written to stderr.
+(including all debug messages) are written to stderr rather than
+syslog. A control command line is also available on stdin/stdout.
.TP
.BI \-\-no-streaming
Do not try to use the streaming extensions to NNTP (for use eg if the
instead of the default
.BR inn.conf .
.TP
-.BR \-\-cli= \fIDIR\fR / |\fIPATH\fR
-Bind the control command line socket to
-.IB DIR / site _cli
-(if the value ends with a
-.BR /)
-or
-.I PATH
-(if it doesn't). See CONTROLLING INNDUCT, below.
-This option may be essential, if the
-path to
-.I feedfile
-is too long, as there is a fairly short limit on the paths to AF_UNIX
-sockets. The default is
-.IR feedfile \fB_cli\fR.
-.TP
.BI \-\-port= PORT
Connect to port
.I PORT
at the remote site rather than to the NNTP port (119).
.TP
+.BI \-\-chdir= PATHRUN
+Change directory to
+.IR pathrun
+at startup. The default is
+.I pathrun
+from inn.conf.
+.TP
+.BR \-\-cli= \fICLI-DIR\fR / |\fICLI-PATH\fR| none
+Listen for control command line connections on
+.IB CLI-DIR / site
+(if the value ends with a
+.BR /)
+or
+.I CLI-PATH
+(if it doesn't). See CONTROLLING INNDUCT, below.
+Note that there is a fairly short limit on the lengths of AF_UNIX
+socket pathnames. If specified as
+.IR CLI-DIR \fB/\fR,
+the directory will be created with mode 700 if necessary.
+The default is
+.B innduct/
+which means to create that directory in
+.I PATHRUN
+and listen on
+.RB \fIPATHRUN\fR /innduct/ \fIsite\fR.
+.TP
.BI \-\-help
Just print a brief usage message and list of the options to stdout.
+.LP
+See TUNING OPTIONS below for more options.
+.SH CONTROLLING INNDUCT
+If you tell innd to drop the feed, innduct will (when it notices,
+which will normally be the next time it decides to flush) finish up the
+articles it has in hand now, and then exit. It is harmless to cause
+innd to flush the feed (but innduct won't notice and flushing won't
+start a new feedfile; you have to leave that to innduct).
+.LP
+If you want to stop innduct you can send it SIGTERM or SIGINT, or the
+.B stop
+control command, in which case it will report statistics so far and
+quickly exit. If innduct receives SIGKILL nothing will be broken or
+corrupted; you just won't see some of the article stats.
+.LP
+innduct listens on an AF_UNIX socket (by default,
+.IR pathrun \fB/innduct/\fR site ),
+and provides a command-line interface which can be used to trigger
+various events and for debugging. When a connection arrives, innduct
+writes a prompt, reads commands a line at a time, and writes any
+output back to the caller. (Everything uses unix line endings.) The
+cli can most easily be accessed with a program like
+.I netcat-openbsd
+(eg
+.B nc.openbsd -U
+.RI \fB/var/run/news/innduct/\fR site )
+or
+.IR socat .
+The prompt is
+.IR site \fB|\fR.
+.LP
+The following control commands are supported:
+.TP
+.B h
+Print a list of all the commands understood. This list includes
+undocumented commands which mess with innduct's internal state and
+should only be used by a developer in conjuction with the innduct
+source code.
+.TP
+.B flush
+Start a new feed file and trigger a flush of the feed. (Or, cause
+the
+.I FLUSH-FINISH-PERIOD
+to expire early, forcibly completing a previously started flush.)
+.TP
+.B stop
+Log statistics and exit. (Same effect as SIGTERM or SIGINT.)
+.TP
+.BR "dump q" | a
+Writes information about innduct's state to a plain text file
+.IR feedfile \fB_dump\fR.
+This overwrites any previous dump. innduct does not ever delete these
+dump files.
+.B "dump q"
+gives a summary including general state and a list of connections;
+.B "dump a"
+also includes information about each article innduct is dealing with.
+.TP
+.B next blscan
+Requests that innduct rescan for new backlog files at the next
+.I PERIOD
+poll. Normally innduct assumes that any backlog files dropped in by
+the administrator are not urgent, and it may not get around to
+noticing them for
+.IR BACKLOG-SCAN-PERIOD .
+.TP
+.B next conn
+Resets the connection startup delay counter so that innduct may
+consider making a new connection to the peer right away, regardless
+of the setting of
+.IR RECONNECT-PERIOD .
+A connection attempt will still only be made if innduct feels that it
+needs one, and innduct may wait up to
+.I PERIOD
+before actually starting the attempt.
.SH TUNING OPTIONS
You should not normally need to adjust these. Time intervals may
specified in seconds, or as a number followed by one of the following
.BR "das hs ks Ms" .
.TP
.BI \-\-max-connections= max
-Restricts the maximum number of simultaneous NNTP connections used by
-for each site to
+Restricts the maximum number of simultaneous NNTP connections
+per peer to
.IR max .
+There is no global limit on the number of connections used by all
+innducts, as the instances for different sites are entirely
+independent.
The default is
.BR 10 .
-There is no global limit on the number of connections.
.TP
.BI \-\-max-queue-per-conn= per-conn-max
Restricts the maximum number of outstanding articles queued on any
Restricts the maximum number articles read into core from any one
input file to
.IR max .
-The default is twice the value of per-conn-max.
+The default is
+.B twice
+.IR per-conn-max .
.TP
.BI \-\-feedfile-flush-size= bytes
Specifies that innduct should flush the feed and start a new feedfile
when the existing feedfile size exceeds
.IR bytes ;
the effect is that the innduct will try to avoid the various
-batchfiles growing much beyond this size while the link to the peer is
-working. The default is
+batchfiles growing much beyond this size. The default is
.BR 100000 .
.TP
.BI \-\-period-interval= PERIOD-INTERVAL
we wait at least
.IR PERIOD .
before offering the article again. The default is
-.BR 50s .
+.BR 100s .
.TP
.BI \-\-backlog-rescan-interval= BACKLOG-SCAN-PERIOD
We scan the directory containing
in case the administrator has manually dropped in a file there for
processing.
The default is
-.TP
-.BI \-\-max-flush-interval= PERIOD
-We flush the feed at least every
-.IR PERIOD
-even if the current instance of the feedfile has not reached the size
-threshold.
-The default is
-.BR 100000s .
+.BR 300s .
.TP
.BI \-\-max-flush-interval= PERIOD
We flush the feed and start a new feedfile at least every
.IR FLUSH-FINISH-PERIOD
ago, and are still trying to finish processing articles that were
written to the old feed file, we forcibly and violently make sure that
-we do by abandoning and deferring all the work (which includes
-unceremoniously dropping any connections on which we've sent some of
-those articles but not yet had replies, as they're probably stuck
-somehow).
-The default is
+we can finish the old feed file: we abandon and defer all the work,
+which includes unceremoniously dropping any connections on which
+we've sent some of those articles but not yet had replies, as they're
+probably stuck somehow. The default is
.BR 2000s .
.TP
.BI \-\-idle-timeout= PERIOD
The default is
.BR 1000s .
.TP
+.BI \-\-low-volume-thresh= "WIN-THRESH " \-\-low-volume-window= "PERIOD "
+If innduct has only one connection to the peer, and has processed
+fewer than
+.I WIN-THRESH
+articles in the last
+.I PERIOD
+and also no articles in the last
+.IR PERIOD-INTERVAL
+it will close the connection quickly. That is, innduct switches to a
+mode where it opens a connection for each article (or, perhaps, each
+handful of articles arriving together).
+The default is to close if fewer than
+.BR 3
+articles in the last
+.BR 1000s .
+.TP
.BI \-\-max-bad-input-data-ratio= PERCENT
We tolerate up to this proportion of badly-formatted lines in the
feedfile and other input files. Every badly-formatted line is logged,
on-disk data is too severe, and crash; to successfully restart,
administrator intervention will be required. This avoids flooding the
logs with warnings and also arranges to abort earlyish if an attempt
-is made to process a file in the wrong format.
-The default is
+is made to process a file in the wrong format. We need to tolerate a
+small proportion of broken lines, if for no other reason than that a
+crash might leave a half-blanked-out entry. The default is
.BR 1
(ie, 1%).
.TP
filled with random data, or one corrupt 1024-byte disk block filled
with an inappropriate text file with a mean line length of at least
35).
-.SH CONTROLLING INNDUCT
-If you tell innd to drop the feed, innduct will (when it notices,
-which will normally be the next time it decides flushes) finish up the
-articles it has in hand now, and then exit. It is harmless to cause
-innd to flush the feed (but innduct won't notice and flushing won't
-start a new feedfile; you have to leave that to innduct).
-.LP
-If you want to stop innduct you can send it SIGTERM or SIGINT, or the
-.B stop
-control command, in which case it will report statistics so far and
-quickly exit. If innduct receives SIGKILL nothing will be broken or
-corrupted; you just won't see some of the article stats.
-.LP
-innduct listens on an AF_UNIX socket, and provides a command-line
-interface which can be used to trigger various events and for
-debugging. The socket is called
-.IB feedfile _cli
-and when connected reads and writes lines (with unix line endings).
-It can most easily be accessed with a program like
-.I netcat-openbsd
-(eg
-.B nc.openbsd -U
-.IR feedfile \fB_cli\fR)
-or
-.IR socat .
-The prompt is
-.IR site \fB|\fR.
-.LP
-The following control commands are supported:
-.TP
-.B h
-Print a list of all the commands understood. This list includes
-undocumented commands which mess with innduct's internal state and
-should only be used by a developer in conjuction with the innduct
-source code.
-.TP
-.B flush
-Start a new feed file and trigger a flush of the feed. (Or, cause
-the
-.I FLUSH-FINISH-PERIOD
-to expire early, forcibly completing a previously started flush.)
-.TP
-.B stop
-Log statistics and exit. (Same effect as SIGTERM or SIGINT.)
+.SH INNDUCT VS INNFEED/NNTPSEND/INNXMIT
.TP
-.BR "dump q" | a
-Writes information about innduct's state to a plain text file
-.IR feedfile \fB_dump\fR.
-This overwrites any previous dump.
-.B "dump q"
-is a summary including general state and a list of connections;
-.B "dump a"
-also includes information about each article innduct is dealing with.
+.B innfeed
+does roughly the same thing as innduct. However, the way it receives
+information from innd can result in articles being lost (not offered
+to peers) if innfeed crashes for any reason. This is an inherent
+defect in the innd channel feed protocol. innduct uses a file feed,
+constantly "tailing" the feed file, and where implemented uses
+.BR inotify (2)
+to reduce the latency which would come from having to constantly poll
+the feed file. innduct is much smaller and simpler, at <4kloc to
+innfeed's 25kloc. innfeed needs a separate helper script or similar
+infrastructure (of which there is an example in its manpage), whereas
+innduct can be run directly and doesn't need help from shell scripts.
+However, innfeed is capable of feeding multiple peers from a single
+innfeed instance, whereas each innduct process handles exactly one
+peer.
.TP
-.B next blscan
-Requests that innduct rescan for new backlog files at the next
-.I PERIOD
-poll. Normally innduct assumes that any backlog files dropped in by
-the administrator are not urgent and may not get around to noticing
-them for
-.IR BACKLOG-SCAN-PERIOD .
+.B nntpsend
+processes feed files in batch mode. That is, you have to periodically
+invoke nntpsend, and when you do, the feed is flushed and articles
+which arrived before the flush are sent to the peer. This introduces
+a batching delay, and also means that the NNTP connection to the peer
+needs to be remade at each batch. nntpsend (which uses innxmit)
+cannot make use of multiple connections to a single peer site.
+However, nntpsend automatically find which sites need feeding by
+looking in
+.IR pathoutgoing ,
+whereas the administrator needs to arrange to invoke innduct
+separately for each peer.
.TP
-.B next conn
-Resets the connection startup delay counter so that innduct may
-consider making a new connection to the peer right away, regardless
-of the setting of
-.IR RECONNECT-PERIOD .
-A connection attempt will still only be made if innduct feels that it
-needs one, and innduct may wait up to
-.I PERIOD
-before actually starting the attempt.
-.IR BACKLOG-SCAN-PERIOD .
+.B innxmit
+is the actual NNTP feeder program used by nntpsend.
.SH EXIT STATUS
.TP
.B 0
not mess with the feedfile and other associated files, other than as
explained here:
.IX Header "FILES"
+.IP \fIpathrun\fR
+.IX Item "default directory"
+Default current working directory for innduct, and also default
+grandparent directory for the command line socket.
.IP \fIpathoutgoing\fR/\fIsite\fR
.IX Item "default feedfile"
Default
where \fItoken\fR is the inn storage API token. Such lines can be
written by \fBTf,Wnm\fR in a \fInewsfeeds\fR(5) entry. During
processing, innduct overwrites lines in the batch files which
-correspond to articles it has processed: the line is replaced with
-one containing only spaces. Only innd should create this file, and
-only innduct should remove it.
+correspond to articles it has processed: each such line is replaced
+with one containing only spaces. Only innd should create
+.IR feedfile ,
+and only innduct should remove it.
.IP \fIfeedfile\fR_lock
.IX Item "lock file"
Lockfile, preventing multiple innduct invocations for the same
.IP \fIfeedfile\fR_flushing
.IX Item "flushing file"
Batch file: the main feedfile is renamed to this filename by innduct
-before it asks inn to flush the feed. Only innduct should create or
-remove this file.
+before it asks inn to flush the feed. Only innduct should create,
+modify or remove this file.
.IP \fIfeedfile\fR_defer
.IX Item "flushing file"
-Batch file containing details of articles whose transmission has
+Batch file containing details of articles whose transmission has very
recently been deferred at the request of the recipient site. Created,
-written, read and removed by innduct.
+written, read and removed (only) by innduct.
.IP \fIfeedfile\fR_backlog.\fItime_t\fR.\fIinum\fR
.IX Item "backlog file"
Batch file containing details of articles whose transmission has less
recently been deferred at the request of the recipient site. Created
-by innduct, and will also be read and removed by innduct. However you
-(the administrator) may also safely remove backlog files.
+by innduct, and will also be read, updated and removed by innduct.
+However you (the administrator) may also safely remove backlog files.
.IP \fIfeedfile\fR_backlog\fIsomething\fR
.IX Item "manual backlog file"
Batch file manually provided by the administrator. The file should be
complete and ready to process at the time it is renamed or hardlinked
to this name. innduct will then automatically find and read and
-process it and eventually remove it. The administrator may also
-safely remove backlog files. \fIsomething\fR may not contain \fB#\fR
-\fB~\fR or \fB/\fR. Be sure to have finished writing the file before
-you rename it to match the pattern \fIfeedfile\fR\fB_backlog\fR*, as
-otherwise innduct may find and process the file and read it to EOF
-before you have finished creating it.
-.IP \fIfeedfile\fR_cli
+process it (including blanking out entries for processed articles) and
+eventually remove it. \fIsomething\fR may not contain \fB#\fR \fB~\fR
+or \fB/\fR.
+.IP
+Be sure to have finished writing the file before you rename it to
+match the pattern \fIfeedfile\fR\fB_backlog\fR*, as otherwise innduct
+may find and process the file, and even think it has finished it,
+before you have written the complete file. You may also safely remove
+backlog files.
+.IP \fIpathrun\fR\fB/innduct/\fB\fIsite\fR
.IX Item "control command line socket"
Default AF_UNIX listening socket for the control command line. See
CONTROLLING INNDUCT, above.
state to this text file. This is mostly useful for debugging.
.IP /etc/news/inn.conf
.IX Item inn.conf
-Used to find
+Used for
.IR pathoutgoing
-if none is specified, for finding how to communicate with innd,
-and also for
+(to compute default
+.IR feedfile
+and associated paths),
+.IR pathrun
+(to compute default
+.IR PATHRUN
+and hence effective default
+.IR CLI-DIR
+and
+.IR CLI-PATH ),
+for finding how to communicate with innd, and also for
.IR sourceaddress
and/or
.IR sourceaddress6 .
Written by Ian Jackson <ijackson@chiark.greenend.org.uk>
.SH "SEE ALSO"
inn.conf(5),
+innd(8),
newsfeeds(5)
~ian / inn-innduct.git / blobdiff