3 innduct \- quickly and reliably stream Usenet articles to remote site
11 implements NNTP peer-to-peer news transmission including the streaming
12 extensions, for sending news articles to a remote site. It is
13 intended as a replacement for
20 You need to run one instance of innduct for each peer site. innduct
21 manages its interaction with innd, including flushing the feed as
22 appropriate, etc., so that articles are transmitted quickly, and
23 manages the retransmission of its own backlog. innduct includes the
24 locking necessary to avoid multiple simutaneous invocations.
26 By default, innduct reads the default feedfile corresponding to
30 .IR pathoutgoing / site )
31 and feeds it via NNTP, streaming if possible, to the host
35 is not specified, it defaults to
38 innduct daemonises after argument parsing, and all logging (including
39 error messages) are sent to syslog (facility
42 The best way to run innduct is probably to periodically invoke innduct
43 for each feed (e.g. from cron), passing the
45 option to arrange that innduct silently exits if an instance is
46 already running for that site.
47 .SH INNDUCT VS INNFEED/NNTPSEND/INNXMIT
50 does roughly the same thing as innduct. However, the way it receives
51 information from innd can result in articles being lost (not offered
52 to peers) if innfeed crashes for any reason. This is an inherent
53 defect in the innd channel feed protocol. innduct uses a file feed,
54 constantly "tailing" the feed file, and where implemented uses
56 to reduce the latency which would come from having to constantly poll
57 the feed file. innfeed is capable of feeding multiple peers from a
58 single innfeed instance, whereas each innduct process handles exactly
59 one peer. innduct is much smaller and simpler, at <4kloc to innfeed's
60 25kloc. innfeed needs a separate wrapper script or similar
61 infrastructure (of which there is an example in its manpage), whereas
62 innduct can be run directly and doesn't need help from shell scripts.
65 processes feed files in batch mode. That is, you have to periodically
66 invoke nntpsend, and when you do, the feed is flushed and articles
67 which arrived before the flush are sent to the peer. This introduces
68 a batching delay, and also means that the NNTP connection to the peer
69 needs to be remade at each batch. nntpsend (which uses innxmit)
70 cannot make use of multiple connections to a single peer site.
71 However, nntpsend can be left to find automatically which sites need
76 is the actual NNTP feeder program used by nntpsend.
79 .BR \-f | \-\-feedfile= \fIfeedfile\fR
82 If the specified value ends in a
84 it is taken as a directory to use as if it were
86 and the actual feed file used is
87 .IR specified_feedfile / site .
89 .BR \-q | \-\-quiet-multiple
90 Makes innduct silently exit (with status 0) if another innduct holds
91 the lock for the site. Without \fB-q\fR, this causes a fatal error to
92 be logged and a nonzero exit.
95 Do not daemonise. innduct runs in the foreground and all messages
96 (including all debug messages) are written to stderr. A control
97 command line is also available on stdin/stdout.
100 Do not try to use the streaming extensions to NNTP (for use eg if the
101 peer can't cope when we send MODE STREAM).
104 Do not try to use the file change monitoring support to watch for
105 writes by innd to the feed file; poll it instead. (If file monitoring
106 is not compiled in, this option just downgrades the log message which
107 warns about this situation.)
109 .BR \-C | \-\-inndconf= \fIFILE\fR
112 instead of the default
118 at the remote site rather than to the NNTP port (119).
120 .BI \-\-chdir= pathrun
123 at startup. The default is
127 .BR \-\-cli= \fICLI-DIR\fR / |\fICLI-PATH\fR| none
128 Listen for control command line connections on
130 (if the value ends with a
134 (if it doesn't). See CONTROLLING INNDUCT, below.
135 Note that there is a fairly short limit on the paths to AF_UNIX
136 sockets. If specified as
138 the directory will be created with mode 700 if necessary.
141 which means to create that directory in
144 .RB \fIpathrun\fR /innduct/ \fIsite\fR.
147 Just print a brief usage message and list of the options to stdout.
149 You should not normally need to adjust these. Time intervals may
150 specified in seconds, or as a number followed by one of the following
153 .BR "sec min hour day" ,
156 .BI \-\-max-connections= max
157 Restricts the maximum number of simultaneous NNTP connections used by
162 There is no global limit on the number of connections.
164 .BI \-\-max-queue-per-conn= per-conn-max
165 Restricts the maximum number of outstanding articles queued on any
166 particular connection to
168 (Non-streaming connections can only handle one article at a time.)
172 .BI \-\-max-queue-per-file= max
173 Restricts the maximum number articles read into core from any one
176 The default is twice the value of per-conn-max.
178 .BI \-\-feedfile-flush-size= bytes
179 Specifies that innduct should flush the feed and start a new feedfile
180 when the existing feedfile size exceeds
182 the effect is that the innduct will try to avoid the various
183 batchfiles growing much beyond this size while the link to the peer is
184 working. The default is
187 .BI \-\-period-interval= PERIOD-INTERVAL
188 Specifies wakup interval and period granularity.
189 innduct wakes up every
191 to do various housekeeping checks. Also, many of the timeout and
192 rescan intervals (those specified in this manual as
194 are rounded up to the next multiple of
195 .IR PERIOD-INTERVAL .
197 .BI \-\-connection-timeout= TIME
198 How long to allow for a connection setup attempt before giving up.
202 .BI \-\-stuck-flush-timeout= TIME
203 How long to wait for innd to respond to a flush request before giving
207 .BI \-\-feedfile-poll= TIME
208 How often to poll the feedfile for new articles written by innd
211 or equivalent) is not available. (When file monitoring is available,
212 there is no need for periodic checks and we wake immediately up
213 whenever the feedfile changes.)
217 .BI \-\-no-check-proportion= PERCENT
218 If the moving average of the proportion of articles being accepted
219 (rather than declined) by the peer exceeds this value, innduct uses
220 "no check mode" - ie it just sends the peer the articles with TAKETHIS
221 rather than checking first with CHECK whether the article is wanted.
222 This only affects streaming connections. The default is
226 .BI \-\-no-check-response-time= ARTICLES
227 The moving average mentioned above is an alpha-smoothed value with a
233 .BI \-\-reconnect-interval= RECONNECT-PERIOD
234 Limits initiation of new connections to one each
235 .IR RECONNECT-PERIOD .
236 This applies to reconnections if the peer has been down, and also to
237 ramping up the number of connections we are using after startup or in
238 response to an article flood. The default is
241 .BI \-\-flush-retry-interval= PERIOD
242 If our attempt to flush the feed failed (usually this will be because
243 innd is not running), try again after
248 .BI \-\-earliest-deferred-retry= PERIOD
249 When the peer responds to our offer of an article with a 431 or 436
250 NNTP response code, indicating that the article has already been
251 offered to it by another of its peers, and that we should try again,
254 before offering the article again. The default is
257 .BI \-\-backlog-rescan-interval= BACKLOG-SCAN-PERIOD
258 We scan the directory containing
260 for backlog files at least every
261 .IR BACKLOG-SCAN-PERIOD ,
262 in case the administrator has manually dropped in a file there for
266 .BI \-\-max-flush-interval= PERIOD
267 We flush the feed at least every
269 even if the current instance of the feedfile has not reached the size
274 .BI \-\-max-flush-interval= PERIOD
275 We flush the feed and start a new feedfile at least every
277 even if the current instance of the feedfile has not reached the size
282 .BI \-\-flush-finish-timeout= FLUSH-FINISH-PERIOD
284 .IR FLUSH-FINISH-PERIOD
285 ago, and are still trying to finish processing articles that were
286 written to the old feed file, we forcibly and violently make sure that
287 we do by abandoning and deferring all the work (which includes
288 unceremoniously dropping any connections on which we've sent some of
289 those articles but not yet had replies, as they're probably stuck
294 .BI \-\-idle-timeout= PERIOD
295 Connections which have had no activity for
297 will be closed. This includes connections where we have sent commands
298 or articles but have not yet had the responses, so this same value
299 doubles as the timeout after which we conclude that the peer is
300 unresponsive or the connection has become broken.
304 .BI \-\-max-bad-input-data-ratio= PERCENT
305 We tolerate up to this proportion of badly-formatted lines in the
306 feedfile and other input files. Every badly-formatted line is logged,
307 but if there are too many we conclude that the corruption to our
308 on-disk data is too severe, and crash; to successfully restart,
309 administrator intervention will be required. This avoids flooding the
310 logs with warnings and also arranges to abort earlyish if an attempt
311 is made to process a file in the wrong format.
316 .BI \-\-max-bad-input-data-init= LINES
317 Additionally, we tolerate this number of additional badly-formatted
318 lines, so that if the badly-formatted lines are a few but at the start
319 of the file, we don't crash immediately.
322 (which would suffice to ignore one whole corrupt 4096-byte disk block
323 filled with random data, or one corrupt 1024-byte disk block filled
324 with an inappropriate text file with a mean line length of at least
326 .SH CONTROLLING INNDUCT
327 If you tell innd to drop the feed, innduct will (when it notices,
328 which will normally be the next time it decides flushes) finish up the
329 articles it has in hand now, and then exit. It is harmless to cause
330 innd to flush the feed (but innduct won't notice and flushing won't
331 start a new feedfile; you have to leave that to innduct).
333 If you want to stop innduct you can send it SIGTERM or SIGINT, or the
335 control command, in which case it will report statistics so far and
336 quickly exit. If innduct receives SIGKILL nothing will be broken or
337 corrupted; you just won't see some of the article stats.
339 innduct listens on an AF_UNIX socket, and provides a command-line
340 interface which can be used to trigger various events and for
341 debugging. innduct listens (by default on
342 .IR pathrun \fB/innduct/\fR site )
343 and when connected reads and writes lines (with unix line endings).
344 The cli can most easily be accessed with a program like
347 .B nc.openbsd -U /var/run/news/innduct/
354 The following control commands are supported:
357 Print a list of all the commands understood. This list includes
358 undocumented commands which mess with innduct's internal state and
359 should only be used by a developer in conjuction with the innduct
363 Start a new feed file and trigger a flush of the feed. (Or, cause
365 .I FLUSH-FINISH-PERIOD
366 to expire early, forcibly completing a previously started flush.)
369 Log statistics and exit. (Same effect as SIGTERM or SIGINT.)
372 Writes information about innduct's state to a plain text file
373 .IR feedfile \fB_dump\fR.
374 This overwrites any previous dump.
376 is a summary including general state and a list of connections;
378 also includes information about each article innduct is dealing with.
381 Requests that innduct rescan for new backlog files at the next
383 poll. Normally innduct assumes that any backlog files dropped in by
384 the administrator are not urgent and may not get around to noticing
386 .IR BACKLOG-SCAN-PERIOD .
389 Resets the connection startup delay counter so that innduct may
390 consider making a new connection to the peer right away, regardless
392 .IR RECONNECT-PERIOD .
393 A connection attempt will still only be made if innduct feels that it
394 needs one, and innduct may wait up to
396 before actually starting the attempt.
397 .IR BACKLOG-SCAN-PERIOD .
401 An instance of innduct is already running for this
408 The feed has been dropped by innd, and we (or previous innducts) have
409 successfully offered all the old articles to the peer site. Our work
413 innduct was invoked with bad options or command line arguments. The
414 error message will be printed to stderr, and also (if any options or
415 arguments were passed at all) to syslog with severity
419 Things are going wrong, hopefully shortage of memory, system file
420 table entries; disk IO problems; disk full; etc. The specifics of the
421 error will be logged to syslog with severity
423 (if syslog is working!)
426 Things are going badly wrong in an unexpected way: system calls which
427 are not expected to fail are doing so, or the protocol for
428 communicating with innd is being violated, or some such. Details will
429 be logged with severity
431 (if syslog is working!)
434 These exit statuses are used by children forked by innduct to
435 communicate to the parent. You should not see them. If you do, it is
438 innduct dances a somewhat complicated dance with innd to make sure
439 that everything goes smoothly and that there are no races. (See the
440 two ascii-art diagrams in innduct.c for details of the protocol.) Do
441 not mess with the feedfile and other associated files, other than as
445 .IX Item "default directory"
446 Default current working directory for innduct, and also default
447 parent directory for the command line socket.
448 .IP \fIpathoutgoing\fR/\fIsite\fR
449 .IX Item "default feedfile"
454 Main feed file as specified in
456 This and other batchfiles used by innduct contains lines each of which
458 \& \fItoken\fR \fImessageid\fR
459 where \fItoken\fR is the inn storage API token. Such lines can be
460 written by \fBTf,Wnm\fR in a \fInewsfeeds\fR(5) entry. During
461 processing, innduct overwrites lines in the batch files which
462 correspond to articles it has processed: the line is replaced with
463 one containing only spaces. Only innd should create this file, and
464 only innduct should remove it.
465 .IP \fIfeedfile\fR_lock
467 Lockfile, preventing multiple innduct invocations for the same
468 feed. A process holds this lock after it has opened the lockfile,
469 made an fcntl F_SETLK call, and then checked with stat and fstat that
470 the file it now has open and has locked still has the name
471 \fIfeedfile\fR_lock. (Only) the lockholder may delete the lockfile.
472 For your convenience, after the lockfile is locked,
479 are all written to the lockfile. NB that stale lockfiles may contain
480 stale data so this information should not be relied on other than for
482 .IP \fIfeedfile\fR_flushing
483 .IX Item "flushing file"
484 Batch file: the main feedfile is renamed to this filename by innduct
485 before it asks inn to flush the feed. Only innduct should create or
487 .IP \fIfeedfile\fR_defer
488 .IX Item "flushing file"
489 Batch file containing details of articles whose transmission has
490 recently been deferred at the request of the recipient site. Created,
491 written, read and removed by innduct.
492 .IP \fIfeedfile\fR_backlog.\fItime_t\fR.\fIinum\fR
493 .IX Item "backlog file"
494 Batch file containing details of articles whose transmission has less
495 recently been deferred at the request of the recipient site. Created
496 by innduct, and will also be read and removed by innduct. However you
497 (the administrator) may also safely remove backlog files.
498 .IP \fIfeedfile\fR_backlog\fIsomething\fR
499 .IX Item "manual backlog file"
500 Batch file manually provided by the administrator. The file should be
501 complete and ready to process at the time it is renamed or hardlinked
502 to this name. innduct will then automatically find and read and
503 process it and eventually remove it. The administrator may also
504 safely remove backlog files. \fIsomething\fR may not contain \fB#\fR
505 \fB~\fR or \fB/\fR. Be sure to have finished writing the file before
506 you rename it to match the pattern \fIfeedfile\fR\fB_backlog\fR*, as
507 otherwise innduct may find and process the file and read it to EOF
508 before you have finished creating it.
509 .IP \fIpathrun\fR\fB/innduct/\fB\fIsite\fR
510 .IX Item "control command line socket"
511 Default AF_UNIX listening socket for the control command line. See
512 CONTROLLING INNDUCT, above.
513 .IP \fIfeedfile\fR_dump
514 .IX Item "debug dump file"
515 On request via a control connection innduct dumps a summary of its
516 state to this text file. This is mostly useful for debugging.
517 .IP /etc/news/inn.conf
523 and associated paths),
529 for finding how to communicate with innd, and also for
534 Written by Ian Jackson <ijackson@chiark.greenend.org.uk>
~ian / innduct.git / blob