3 innduct \- quickly and reliably stream Usenet articles to remote site
11 implements NNTP peer-to-peer news transmission including the streaming
12 extensions, for sending news articles to a remote site. It is
13 intended as a replacement for
20 You need to run one instance of innduct for each peer site. innduct
21 manages its interaction with
23 including flushing the feed as appropriate, etc., so that articles are
24 transmitted quickly, and manages the retransmission of its own
25 backlog. innduct includes the locking necessary to avoid multiple
26 simutaneous invocations.
28 By default, innduct reads the default feedfile corresponding to
32 .IR pathoutgoing / site )
33 and feeds it via NNTP, streaming if possible, to the host
37 is not specified, it defaults to
40 innduct daemonises after argument parsing, and all logging (including
41 error messages) are sent to syslog (facility
44 The best way to run innduct is probably to periodically invoke it
45 for each feed (e.g. from cron), passing the
47 option to arrange that innduct silently exits if an instance is
48 already running for that site.
51 .BR \-f | \-\-feedfile= \fIDIR\fR / |\fIPATH\fR
54 to read, and indirectly specifies the paths to
55 be used for various associated files (see FILES, below).
58 it is taken as a directory to use, and the actual feed file used is
66 it is taken to be relative to
72 .BR \-q | \-\-quiet-multiple
73 Makes innduct silently exit (with status 0) if another innduct holds
74 the lock for the site. Without \fB-q\fR, this causes a fatal error to
75 be logged and a nonzero exit.
78 Do not daemonise. innduct runs in the foreground and all messages
79 (including all debug messages) are written to stderr rather than
80 syslog. A control command line is also available on stdin/stdout.
83 Do not try to use the streaming extensions to NNTP (for use eg if the
84 peer can't cope when we send MODE STREAM).
87 Do not try to use the file change monitoring support to watch for
88 writes by innd to the feed file; poll it instead. (If file monitoring
89 is not compiled in, this option just downgrades the log message which
90 warns about this situation.)
92 .BR \-C | \-\-inndconf= \fIFILE\fR
95 instead of the default
101 at the remote site rather than to the NNTP port (119).
103 .BI \-\-chdir= PATHRUN
106 at startup. The default is
110 .BR \-\-cli= \fICLI-DIR\fR / |\fICLI-PATH\fR| none
111 Listen for control command line connections on
113 (if the value ends with a
117 (if it doesn't). See CONTROLLING INNDUCT, below.
118 Note that there is a fairly short limit on the lengths of AF_UNIX
119 socket pathnames. If specified as
121 the directory will be created with mode 700 if necessary.
124 which means to create that directory in
127 .RB \fIPATHRUN\fR /innduct/ \fIsite\fR.
130 Just print a brief usage message and list of the options to stdout.
132 See TUNING OPTIONS below for more options.
133 .SH CONTROLLING INNDUCT
134 If you tell innd to drop the feed, innduct will (when it notices,
135 which will normally be the next time it decides to flush) finish up the
136 articles it has in hand now, and then exit. It is harmless to cause
137 innd to flush the feed (but innduct won't notice and flushing won't
138 start a new feedfile; you have to leave that to innduct).
140 If you want to stop innduct you can send it SIGTERM or SIGINT, or the
142 control command, in which case it will report statistics so far and
143 quickly exit. If innduct receives SIGKILL nothing will be broken or
144 corrupted; you just won't see some of the article stats.
146 innduct listens on an AF_UNIX socket (by default,
147 .IR pathrun \fB/innduct/\fR site ),
148 and provides a command-line interface which can be used to trigger
149 various events and for debugging. When a connection arrives, innduct
150 writes a prompt, reads commands a line at a time, and writes any
151 output back to the caller. (Everything uses unix line endings.) The
152 cli can most easily be accessed with a program like
156 .RI \fB/var/run/news/innduct/\fR site )
162 The following control commands are supported:
165 Print a list of all the commands understood. This list includes
166 undocumented commands which mess with innduct's internal state and
167 should only be used by a developer in conjuction with the innduct
171 Start a new feed file and trigger a flush of the feed. (Or, cause
173 .I FLUSH-FINISH-PERIOD
174 to expire early, forcibly completing a previously started flush.)
177 Log statistics and exit. (Same effect as SIGTERM or SIGINT.)
180 Writes information about innduct's state to a plain text file
181 .IR feedfile \fB_dump\fR.
182 This overwrites any previous dump. innduct does not ever delete these
185 gives a summary including general state and a list of connections;
187 also includes information about each article innduct is dealing with.
190 Requests that innduct rescan for new backlog files at the next
192 poll. Normally innduct assumes that any backlog files dropped in by
193 the administrator are not urgent, and it may not get around to
195 .IR BACKLOG-SCAN-PERIOD .
198 Resets the connection startup delay counter so that innduct may
199 consider making a new connection to the peer right away, regardless
201 .IR RECONNECT-PERIOD .
202 A connection attempt will still only be made if innduct feels that it
203 needs one, and innduct may wait up to
205 before actually starting the attempt.
207 You should not normally need to adjust these. Time intervals may
208 specified in seconds, or as a number followed by one of the following
211 .BR "sec min hour day" ,
214 .BI \-\-max-connections= max
215 Restricts the maximum number of simultaneous NNTP connections
218 There is no global limit on the number of connections used by all
219 innducts, as the instances for different sites are entirely
224 .BI \-\-max-queue-per-conn= per-conn-max
225 Restricts the maximum number of outstanding articles queued on any
226 particular connection to
228 (Non-streaming connections can only handle one article at a time.)
232 .BI \-\-max-queue-per-file= max
233 Restricts the maximum number articles read into core from any one
240 .BI \-\-feedfile-flush-size= bytes
241 Specifies that innduct should flush the feed and start a new feedfile
242 when the existing feedfile size exceeds
244 the effect is that the innduct will try to avoid the various
245 batchfiles growing much beyond this size. The default is
248 .BI \-\-period-interval= PERIOD-INTERVAL
249 Specifies wakup interval and period granularity.
250 innduct wakes up every
252 to do various housekeeping checks. Also, many of the timeout and
253 rescan intervals (those specified in this manual as
255 are rounded up to the next multiple of
256 .IR PERIOD-INTERVAL .
258 .BI \-\-connection-timeout= TIME
259 How long to allow for a connection setup attempt before giving up.
263 .BI \-\-stuck-flush-timeout= TIME
264 How long to wait for innd to respond to a flush request before giving
268 .BI \-\-feedfile-poll= TIME
269 How often to poll the feedfile for new articles written by innd
272 or equivalent) is not available. (When file monitoring is available,
273 there is no need for periodic checks and we wake immediately up
274 whenever the feedfile changes.)
278 .BI \-\-no-check-proportion= PERCENT
279 If the moving average of the proportion of articles being accepted
280 (rather than declined) by the peer exceeds this value, innduct uses
281 "no check mode" - ie it just sends the peer the articles with TAKETHIS
282 rather than checking first with CHECK whether the article is wanted.
283 This only affects streaming connections. The default is
287 .BI \-\-no-check-response-time= ARTICLES
288 The moving average mentioned above is an alpha-smoothed value with a
294 .BI \-\-reconnect-interval= RECONNECT-PERIOD
295 Limits initiation of new connections to one each
296 .IR RECONNECT-PERIOD .
297 This applies to reconnections if the peer has been down, and also to
298 ramping up the number of connections we are using after startup or in
299 response to an article flood. The default is
302 .BI \-\-flush-retry-interval= PERIOD
303 If our attempt to flush the feed failed (usually this will be because
304 innd is not running), try again after
309 .BI \-\-earliest-deferred-retry= PERIOD
310 When the peer responds to our offer of an article with a 431 or 436
311 NNTP response code, indicating that the article has already been
312 offered to it by another of its peers, and that we should try again,
315 before offering the article again. The default is
318 .BI \-\-backlog-rescan-interval= BACKLOG-SCAN-PERIOD
319 We scan the directory containing
321 for backlog files at least every
322 .IR BACKLOG-SCAN-PERIOD ,
323 in case the administrator has manually dropped in a file there for
328 .BI \-\-max-flush-interval= PERIOD
329 We flush the feed and start a new feedfile at least every
331 even if the current instance of the feedfile has not reached the size
336 .BI \-\-flush-finish-timeout= FLUSH-FINISH-PERIOD
338 .IR FLUSH-FINISH-PERIOD
339 ago, and are still trying to finish processing articles that were
340 written to the old feed file, we forcibly and violently make sure that
341 we can finish the old feed file: we abandon and defer all the work,
342 which includes unceremoniously dropping any connections on which
343 we've sent some of those articles but not yet had replies, as they're
344 probably stuck somehow. The default is
347 .BI \-\-idle-timeout= PERIOD
348 Connections which have had no activity for
350 will be closed. This includes connections where we have sent commands
351 or articles but have not yet had the responses, so this same value
352 doubles as the timeout after which we conclude that the peer is
353 unresponsive or the connection has become broken.
357 .BI \-\-max-bad-input-data-ratio= PERCENT
358 We tolerate up to this proportion of badly-formatted lines in the
359 feedfile and other input files. Every badly-formatted line is logged,
360 but if there are too many we conclude that the corruption to our
361 on-disk data is too severe, and crash; to successfully restart,
362 administrator intervention will be required. This avoids flooding the
363 logs with warnings and also arranges to abort earlyish if an attempt
364 is made to process a file in the wrong format. We need to tolerate a
365 small proportion of broken lines, if for no other reason than that a
366 crash might leave a half-blanked-out entry. The default is
370 .BI \-\-max-bad-input-data-init= LINES
371 Additionally, we tolerate this number of additional badly-formatted
372 lines, so that if the badly-formatted lines are a few but at the start
373 of the file, we don't crash immediately.
376 (which would suffice to ignore one whole corrupt 4096-byte disk block
377 filled with random data, or one corrupt 1024-byte disk block filled
378 with an inappropriate text file with a mean line length of at least
380 .SH INNDUCT VS INNFEED/NNTPSEND/INNXMIT
383 does roughly the same thing as innduct. However, the way it receives
384 information from innd can result in articles being lost (not offered
385 to peers) if innfeed crashes for any reason. This is an inherent
386 defect in the innd channel feed protocol. innduct uses a file feed,
387 constantly "tailing" the feed file, and where implemented uses
389 to reduce the latency which would come from having to constantly poll
390 the feed file. innduct is much smaller and simpler, at <4kloc to
391 innfeed's 25kloc. innfeed needs a separate helper script or similar
392 infrastructure (of which there is an example in its manpage), whereas
393 innduct can be run directly and doesn't need help from shell scripts.
394 However, innfeed is capable of feeding multiple peers from a single
395 innfeed instance, whereas each innduct process handles exactly one
399 processes feed files in batch mode. That is, you have to periodically
400 invoke nntpsend, and when you do, the feed is flushed and articles
401 which arrived before the flush are sent to the peer. This introduces
402 a batching delay, and also means that the NNTP connection to the peer
403 needs to be remade at each batch. nntpsend (which uses innxmit)
404 cannot make use of multiple connections to a single peer site.
405 However, nntpsend automatically find which sites need feeding by
408 whereas the administrator needs to arrange to invoke innduct
409 separately for each peer.
412 is the actual NNTP feeder program used by nntpsend.
416 An instance of innduct is already running for this
423 The feed has been dropped by innd, and we (or previous innducts) have
424 successfully offered all the old articles to the peer site. Our work
428 innduct was invoked with bad options or command line arguments. The
429 error message will be printed to stderr, and also (if any options or
430 arguments were passed at all) to syslog with severity
434 Things are going wrong, hopefully shortage of memory, system file
435 table entries; disk IO problems; disk full; etc. The specifics of the
436 error will be logged to syslog with severity
438 (if syslog is working!)
441 Things are going badly wrong in an unexpected way: system calls which
442 are not expected to fail are doing so, or the protocol for
443 communicating with innd is being violated, or some such. Details will
444 be logged with severity
446 (if syslog is working!)
449 These exit statuses are used by children forked by innduct to
450 communicate to the parent. You should not see them. If you do, it is
453 innduct dances a somewhat complicated dance with innd to make sure
454 that everything goes smoothly and that there are no races. (See the
455 two ascii-art diagrams in innduct.c for details of the protocol.) Do
456 not mess with the feedfile and other associated files, other than as
460 .IX Item "default directory"
461 Default current working directory for innduct, and also default
462 grandparent directory for the command line socket.
463 .IP \fIpathoutgoing\fR/\fIsite\fR
464 .IX Item "default feedfile"
469 Main feed file as specified in
471 This and other batchfiles used by innduct contains lines each of which
473 \& \fItoken\fR \fImessageid\fR
474 where \fItoken\fR is the inn storage API token. Such lines can be
475 written by \fBTf,Wnm\fR in a \fInewsfeeds\fR(5) entry. During
476 processing, innduct overwrites lines in the batch files which
477 correspond to articles it has processed: each such line is replaced
478 with one containing only spaces. Only innd should create
480 and only innduct should remove it.
481 .IP \fIfeedfile\fR_lock
483 Lockfile, preventing multiple innduct invocations for the same
484 feed. A process holds this lock after it has opened the lockfile,
485 made an fcntl F_SETLK call, and then checked with stat and fstat that
486 the file it now has open and has locked still has the name
487 \fIfeedfile\fR_lock. (Only) the lockholder may delete the lockfile.
488 For your convenience, after the lockfile is locked,
495 are all written to the lockfile. NB that stale lockfiles may contain
496 stale data so this information should not be relied on other than for
498 .IP \fIfeedfile\fR_flushing
499 .IX Item "flushing file"
500 Batch file: the main feedfile is renamed to this filename by innduct
501 before it asks inn to flush the feed. Only innduct should create,
502 modify or remove this file.
503 .IP \fIfeedfile\fR_defer
504 .IX Item "flushing file"
505 Batch file containing details of articles whose transmission has very
506 recently been deferred at the request of the recipient site. Created,
507 written, read and removed (only) by innduct.
508 .IP \fIfeedfile\fR_backlog.\fItime_t\fR.\fIinum\fR
509 .IX Item "backlog file"
510 Batch file containing details of articles whose transmission has less
511 recently been deferred at the request of the recipient site. Created
512 by innduct, and will also be read, updated and removed by innduct.
513 However you (the administrator) may also safely remove backlog files.
514 .IP \fIfeedfile\fR_backlog\fIsomething\fR
515 .IX Item "manual backlog file"
516 Batch file manually provided by the administrator. The file should be
517 complete and ready to process at the time it is renamed or hardlinked
518 to this name. innduct will then automatically find and read and
519 process it (including blanking out entries for processed articles) and
520 eventually remove it. \fIsomething\fR may not contain \fB#\fR \fB~\fR
523 Be sure to have finished writing the file before you rename it to
524 match the pattern \fIfeedfile\fR\fB_backlog\fR*, as otherwise innduct
525 may find and process the file, and even think it has finished it,
526 before you have written the complete file. You may also safely remove
528 .IP \fIpathrun\fR\fB/innduct/\fB\fIsite\fR
529 .IX Item "control command line socket"
530 Default AF_UNIX listening socket for the control command line. See
531 CONTROLLING INNDUCT, above.
532 .IP \fIfeedfile\fR_dump
533 .IX Item "debug dump file"
534 On request via a control connection innduct dumps a summary of its
535 state to this text file. This is mostly useful for debugging.
536 .IP /etc/news/inn.conf
542 and associated paths),
546 and hence effective default
550 for finding how to communicate with innd, and also for
555 Written by Ian Jackson <ijackson@chiark.greenend.org.uk>
~ian / innduct.git / blob