3 innduct \- quickly and reliably stream Usenet articles to remote site
11 implements NNTP peer-to-peer news transmission including the streaming
12 extensions, for sending news articles to a remote site.
14 You need to run one instance of innduct for each peer site. innduct
15 manages its interaction with innd, including flushing the feed as
16 appropriate, etc., so that articles are transmitted quickly, and
17 manages the retransmission of its own backlog. innduct includes the
18 locking necessary to avoid multiple simutaneous invocations.
20 By default, innduct reads the default feedfile corresponding to
24 .IR pathoutgoing / site )
25 and feeds it via NNTP, streaming if possible, to the host
30 is not specified, it defaults to
33 innduct daemonises after argument parsing, and all logging (including
34 error messages) are sent to syslog (facility
37 The best way to run innduct is probably to periodically invoke innduct
38 for each feed (e.g. from cron), passing innduct it the
40 option to arrange that it silently exits if an innduct is already
41 running for that site.
42 .SH INNDUCT VS INNFEED/NNTPSEND/INNXMIT
45 does roughly the same thing as innduct. However, the way it receives
46 information from innd can result in articles being lost (not offered
47 to peers) if innfeed crashes for any reason. This is an inherent
48 defect in the innd channel feed protocol. innduct uses a file feed,
49 constantly "tailing" the feed file, and where implemented uses
51 to reduce the latency which would come from having to constantly poll
52 the feed file. innfeed is capable of feeding multiple peers from a
53 single innfeed instance, whereas each innduct process handles exactly
54 one peer. innduct is much smaller and simpler, at <4kloc to innfeed's
55 25kloc. innfeed needs a separate wrapper script or similar
56 infrastructure (of which there is an example in its manpage), whereas
57 innduct can be run directly and doesn't need help from shell scripts.
60 processes feed files in batch mode. That is, you have to periodically
61 invoke nntpsend, and when you do, the feed is flushed and articles
62 which arrived before the flush are sent to the peer. This introduces
63 a batching delay, and also means that the NNTP connection to the peer
64 needs to be remade at each batch. nntpsend (which uses innxmit)
65 cannot make use of multiple connections to a single peer site.
66 However, nntpsend can be left to find automatically which sites need
71 is the actual NNTP feeder program used by nntpsend.
74 .BR \-f | \-\-feedfile= \fIfeedfile\fR
77 If the specified value ends in a
79 it is taken as a directory to use as if it were
81 and the actual feed file used is
82 .IR specified_feedfile / site .
84 .BR \-q | \-\-quiet-multiple
85 Makes innduct silently exit (with status 0) if another innduct holds
86 the lock for the site. Without \fB-q\fR, this causes a fatal error to
87 be logged and a nonzero exit.
90 Do not daemonise. innduct runs in the foreground and all messages
91 (including all debug messages) are written to stderr.
94 Do not try to use the streaming extensions to NNTP (for use eg if the
95 peer can't cope when we send MODE STREAM).
98 Do not try to use the file change monitoring support to watch for
99 writes by innd to the feed file; poll it instead. (If file monitoring
100 is not compiled in, this option just downgrades the log message which
101 warns about this situation.)
103 .BR \-C | \-\-inndconf= \fIFILE\fR
106 instead of the default
109 .BI \-\-ctrl-sock-dir= CTRL-SOCK-DIR
112 as the directory to contain the actual control socket. See
113 CONTROLLING INNDUCT, below.
116 does not exist it will be created with mode 0700; if it does
117 exist it must not be a symlink and must be owned by the user running
118 innduct and have no access for "other". If the control socket cannot
119 be set up (for this or any other reason), a warning is logged, but
120 such situations are not fatal for innduct's startup. The default is
121 .BR /tmp/innduct.control .
123 .BI \-\-no-ctrl-sock-dir-perms-check
126 even if it has unexpected permissions and ownership. Note that this
127 is not safe with the default value of
133 at the remote site rather than to the NNTP port (119).
136 Just print a brief usage message and list of the options to stdout.
138 You should not normally need to adjust these. Time intervals may
139 specified in seconds, or as a number followed by one of the following
142 .BR "sec min hour day" ,
145 .BI \-\-max-connections= max
146 Restricts the maximum number of simultaneous NNTP connections used by
151 There is no global limit on the number of connections.
153 .BI \-\-max-queue-per-conn= per-conn-max
154 Restricts the maximum number of outstanding articles queued on any
155 particular connection to
157 (Non-streaming connections can only handle one article at a time.)
161 .BI \-\-max-queue-per-file= max
162 Restricts the maximum number articles read into core from any one
165 The default is twice the value of per-conn-max.
167 .BI \-\-feedfile-flush-size= bytes
168 Specifies that innduct should flush the feed and start a new feedfile
169 when the existing feedfile size exceeds
171 the effect is that the innduct will try to avoid the various
172 batchfiles growing much beyond this size while the link to the peer is
173 working. The default is
176 .BI \-\-period-interval= PERIOD-INTERVAL
177 Specifies wakup interval and period granularity.
178 innduct wakes up every
180 to do various housekeeping checks. Also, many of the timeout and
181 rescan intervals (those specified in this manual as
183 are rounded up to the next multiple of
184 .IR PERIOD-INTERVAL .
186 .BI \-\-connection-timeout= TIME
187 How long to allow for a connection setup attempt before giving up.
191 .BI \-\-stuck-flush-timeout= TIME
192 How long to wait for innd to respond to a flush request before giving
196 .BI \-\-feedfile-poll= TIME
197 How often to poll the feedfile for new articles written by innd
200 or equivalent) is not available. (When file monitoring is available,
201 there is no need for periodic checks and we wake immediately up
202 whenever the feedfile changes.)
206 .BI \-\-no-check-proportion= PERCENT
207 If the moving average of the proportion of articles being accepted
208 (rather than declined) by the peer exceeds this value, innduct uses
209 "no check mode" - ie it just sends the peer the articles with TAKETHIS
210 rather than checking first with CHECK whether the article is wanted.
211 This only affects streaming connections. The default is
215 .BI \-\-no-check-response-time= ARTICLES
216 The moving average mentioned above is an alpha-smoothed value with a
222 .BI \-\-reconnect-interval= RECONNECT-PERIOD
223 Limits initiation of new connections to one each
224 .IR RECONNECT-PERIOD .
225 This applies to reconnections if the peer has been down, and also to
226 ramping up the number of connections we are using after startup or in
227 response to an article flood. The default is
230 .BI \-\-flush-retry-interval= PERIOD
231 If our attempt to flush the feed failed (usually this will be because
232 innd is not running), try again after
237 .BI \-\-earliest-deferred-retry= PERIOD
238 When the peer responds to our offer of an article with a 431 or 436
239 NNTP response code, indicating that the article has already been
240 offered to it by another of its peers, and that we should try again,
243 before offering the article again. The default is
246 .BI \-\-backlog-rescan-interval= BACKLOG-SCAN-PERIOD
247 We scan the directory containing
249 for backlog files at least every
250 .IR BACKLOG-SCAN-PERIOD ,
251 in case the administrator has manually dropped in a file there for
255 .BI \-\-max-flush-interval= PERIOD
256 We flush the feed at least every
258 even if the current instance of the feedfile has not reached the size
263 .BI \-\-max-flush-interval= PERIOD
264 We flush the feed and start a new feedfile at least every
266 even if the current instance of the feedfile has not reached the size
271 .BI \-\-flush-finish-timeout= FLUSH-FINISH-PERIOD
273 .IR FLUSH-FINISH-PERIOD
274 ago, and are still trying to finish processing articles that were
275 written to the old feed file, we forcibly and violently make sure that
276 we do by abandoning and deferring all the work (which includes
277 unceremoniously dropping any connections on which we've sent some of
278 those articles but not yet had replies, as they're probably stuck
283 .BI \-\-idle-timeout= PERIOD
284 Connections which have had no activity for
286 will be closed. This includes connections where we have sent commands
287 or articles but have not yet had the responses, so this same value
288 doubles as the timeout after which we conclude that the peer is
289 unresponsive or the connection has become broken.
293 .BI \-\-max-bad-input-data-ratio= PERCENT
294 We tolerate up to this proportion of badly-formatted lines in the
295 feedfile and other input files. Every badly-formatted line is logged,
296 but if there are too many we conclude that the corruption to our
297 on-disk data is too severe, and crash; to successfully restart,
298 administrator intervention will be required. This avoids flooding the
299 logs with warnings and also arranges to abort earlyish if an attempt
300 is made to process a file in the wrong format.
305 .BI \-\-max-bad-input-data-init= LINES
306 Additionally, we tolerate this number of additional badly-formatted
307 lines, so that if the badly-formatted lines are a few but at the start
308 of the file, we don't crash immediately.
311 (which would suffice to ignore one whole corrupt 4096-byte disk block
312 filled with random data, or one corrupt 1024-byte disk block filled
313 with an inappropriate text file with a mean line length of at least
315 .SH CONTROLLING INNDUCT
316 If you tell innd to drop the feed, innduct will (when it notices,
317 which will normally be the next time it decides flushes) finish up the
318 articles it has in hand now, and then exit. It is harmless to cause
319 innd to flush the feed (but innduct won't notice and flushing won't
320 start a new feedfile; you have to leave that to innduct).
322 If you want to stop innduct you can send it SIGTERM or SIGINT, or the
324 control command, in which case it will report statistics so far and
325 quickly exit. If innduct receives SIGKILL nothing will be broken or
326 corrupted; you just won't see some of the article stats.
327 innduct listens on an AF_UNIX socket, and provides a command-line
328 interface which can be used to trigger various events and for
332 .IB feedfile _control
333 and when connected reads and writes lines (with unix line endings).
334 It can most easily be accessed with a program like
338 .IR feedfile \fB_control\fR)
341 The CLI has as a prompt
344 Because there is a fairly short limit on the length of pathnames used
345 for AF_UNIX sockets, innduct actually creates the socket with a unique
346 name in a different dedicated subdirectory of /tmp, and makes
347 .IB feedfile _control
348 a symlink to the real socket.
350 The following control commands are supported:
353 Print a list of all the commands understood. This list includes
354 undocumented commands which mess with innduct's internal state and
355 should only be used by a developer in conjuction with the innduct
359 Start a new feed file and trigger a flush of the feed. (Or, cause
361 .I FLUSH-FINISH-PERIOD
362 to expire early, forcibly completing a previously started flush.)
365 Log statistics and exit. (Same effect as SIGTERM or SIGINT.)
368 Writes information about innduct's state to a plain text file
369 .IR feedfile \fB_dump\fR.
370 This overwrites any previous dump.
372 is a summary including general state and a list of connections;
374 also includes information about each article innduct is dealing with.
377 Requests that innduct rescan for new backlog files at the next
379 poll. Normally innduct assumes that any backlog files dropped in by
380 the administrator are not urgent and may not get around to noticing
382 .IR BACKLOG-SCAN-PERIOD .
385 Resets the connection startup delay counter so that innduct may
386 consider making a new connection to the peer right away, regardless
388 .IR RECONNECT-PERIOD .
389 A connection attempt will still only be made if innduct feels that it
390 needs one, and innduct may wait up to
392 before actually starting the attempt.
393 .IR BACKLOG-SCAN-PERIOD .
397 An instance of innduct is already running for this
404 The feed has been dropped by innd, and we (or previous innducts) have
405 successfully offered all the old articles to the peer site. Our work
409 innduct was invoked with bad options or command line arguments. The
410 error message will be printed to stderr, and also (if any options or
411 arguments were passed at all) to syslog with severity
415 Things are going wrong, hopefully shortage of memory, system file
416 table entries; disk IO problems; disk full; etc. The specifics of the
417 error will be logged to syslog with severity
419 (if syslog is working!)
422 Things are going badly wrong in an unexpected way: system calls which
423 are not expected to fail are doing so, or the protocol for
424 communicating with innd is being violated, or some such. Details will
425 be logged with severity
427 (if syslog is working!)
430 These exit statuses are used by children forked by innduct to
431 communicate to the parent. You should not see them. If you do, it is
434 innduct dances a somewhat complicated dance with innd to make sure
435 that everything goes smoothly and that there are no races. (See the
436 two ascii-art diagrams in innduct.c for details of the protocol.) Do
437 not mess with the feedfile and other associated files, other than as
440 .IP \fIpathoutgoing\fR/\fIsite\fR
441 .IX Item "default feedfile"
446 Main feed file as specified in
448 This and other batchfiles used by innduct contains lines each of which
450 \& \fItoken\fR \fImessageid\fR
451 where \fItoken\fR is the inn storage API token. Such lines can be
452 written by \fBTf,Wnm\fR in a \fInewsfeeds\fR(5) entry. During
453 processing, innduct overwrites lines in the batch files which
454 correspond to articles it has processed: the line is replaced with
455 one containing only spaces. Only innd should create this file, and
456 only innduct should remove it.
457 .IP \fIfeedfile\fR_lock
459 Lockfile, preventing multiple innduct invocations for the same
460 feed. A process holds this lock after it has opened the lockfile,
461 made an fcntl F_SETLK call, and then checked with stat and fstat that
462 the file it now has open and has locked still has the name
463 \fIfeedfile\fR_lock. (Only) the lockholder may delete the lockfile.
464 For your convenience, after the lockfile is locked,
471 are all written to the lockfile. NB that stale lockfiles may contain
472 stale data so this information should not be relied on other than for
474 .IP \fIfeedfile\fR_flushing
475 .IX Item "flushing file"
476 Batch file: the main feedfile is renamed to this filename by innduct
477 before it asks inn to flush the feed. Only innduct should create or
479 .IP \fIfeedfile\fR_defer
480 .IX Item "flushing file"
481 Batch file containing details of articles whose transmission has
482 recently been deferred at the request of the recipient site. Created,
483 written, read and removed by innduct.
484 .IP \fIfeedfile\fR_backlog.\fItime_t\fR.\fIinum\fR
485 .IX Item "backlog file"
486 Batch file containing details of articles whose transmission has less
487 recently been deferred at the request of the recipient site. Created
488 by innduct, and will also be read and removed by innduct. However you
489 (the administrator) may also safely remove backlog files.
490 .IP \fIfeedfile\fR_backlog\fIsomething\fR
491 .IX Item "manual backlog file"
492 Batch file manually provided by the administrator. The file should be
493 complete and ready to process at the time it is renamed or hardlinked
494 to this name. innduct will then automatically find and read and
495 process it and eventually remove it. The administrator may also
496 safely remove backlog files. \fIsomething\fR may not contain \fB#\fR
497 \fB~\fR or \fB/\fR. Be sure to have finished writing the file before
498 you rename it to match the pattern \fIfeedfile\fR\fB_backlog\fR*, as
499 otherwise innduct may find and process the file and read it to EOF
500 before you have finished creating it.
501 .IP \fIfeedfile\fR_dump
502 .IX Item "debug dump file"
503 On request via a control connection innduct dumps a summary of its
504 state to this text file. This is mostly useful for debugging.
505 .IP /etc/news/inn.conf
509 if none is specified, for finding how to communicate with innd,
515 Written by Ian Jackson <ijackson@chiark.greenend.org.uk>
~ian / inn-innduct.git / blob