3 .\" Author: James A. Brister <brister@vix.com> -- berkeley-unix --
4 .\" Start Date: Sat, 20 Jan 1996 15:50:56 +1100
5 .\" Project: INN -- innfeed
7 .\" RCSId: $Id: innfeed.1 7798 2008-04-26 08:47:01Z iulius $
8 .\" Description: Man page for innfeed(1)
17 innfeed \- multi-host, multi-connection, streaming NNTP feeder.
80 implements the NNTP protocol for transferring news between computers. It
81 handles the standard IHAVE protocol as well as the CHECK/TAKETHIS
82 streaming extension. Innfeed can feed any number of remote hosts at once
83 and will open multiple connections to each host if configured to do so. The
84 only limitations are the process limits for open file descriptors and memory.
86 As an alternative to using NNTP, INN may also be fed to an IMAP
87 server. This is done by using an executable called imapfeed, which is
88 identical to innfeed except for the delivery process. The new version
89 has two types of connections: an LMTP connection to deliver regular
90 messages and an IMAP connection to handle control messages. The
91 startinnfeed process can then be told to start imapfeed instead of innfeed.
92 (See the INSTALL file for how to do this.)
101 has three modes of operation: channel, funnel-file and batch.
103 Channel mode is used when no filename is given on the command line,
104 the ``input-file'' keyword is \fInot\fP given in the config file, \fIand\fP
105 the ``\fI\-x\fP'' option is \fInot\fP given.
106 In channel mode innfeed runs with stdin connected via a pipe to
107 innd. Whenever innd closes this pipe (and it has several reasons during
108 normal processing to do so), innfeed will exit. It first will try to
109 finish sending all articles it was in the middle of transmitting, before
110 issuing a QUIT command. This means innfeed may take a while to exit
111 depending on how slow your peers are. It never (well, almost never) just
112 drops the connection.
114 The recommended way to restart innfeed when run in channel mode is
115 therefore to tell innd to close the pipe and spawn a new innfeed process.
116 This can be done with ``ctlinnd flush <feed>'' where <feed> is the name of
117 the innfeed channel feed in ``\fInewsfeeds\fP''.
119 Funnel-file mode is used when a filename is given as an argument
120 or the ``input-file'' keyword is given in the config file.
121 In funnel file mode it reads the specified file for the same formatted
122 information as innd would give in channel mode. It is expected that innd is
123 continually writing to this file, so when innfeed reaches the end of the file
124 it will check periodically for new information. To prevent the funnel file
125 from growing without bounds, you will need to periodically move the file to
126 the side (or simply remove it) and have innd flush the file. Then, after
127 the file is flushed by innd, you can send innfeed a SIGALRM, and it too
128 will close the file and open the new file created by innd. Something like:
132 innfeed -p /var/run/news/innfeed.pid my-funnel-file &
136 ctlinnd flush funnel-file-site
137 kill -ALRM `cat /var/run/news/innfeed.pid`
142 Batch mode is used when the ``\fI\-x\fP'' flag is used.
143 In batch mode innfeed will ignore stdin, and will simply process any
144 backlog created by a previously running innfeed. This mode is not normally
145 needed as innfeed will take care of backlog processing.
152 Innfeed expects a couple of things to be able to run correctly: a directory
153 where it can store backlog files and a configuration file to describe which
154 peers it should handle.
156 The configuration file is described in
157 .IR innfeed.conf (5).
159 option can be used to specify a different file.
161 For each peer (say, ``\fIfoo\fP''), innfeed manages up to 4 files in the
162 backlog directory: a ``\fIfoo.lock\fP'' file, which prevents other
163 instances of innfeed from interfering with this one; a ``\fIfoo.input\fP''
164 file which has old article information innfeed is reading for
165 re-processing; a ``\fIfoo.output\fP'' file where innfeed is writing
166 information on articles that couldn't be processed (normally due to a slow
167 or blocked peer); and a ``\fIfoo\fP'' file.
169 This last file (``\fIfoo\fP'') is never created by innfeed, but if innfeed
170 notices it, it will rename it to ``\fIfoo.input\fP'' at the next
171 opportunity and will start reading from it. This lets you create a batch
172 file and put it in a place where innfeed will find it. You should never
173 alter the .input or .output files of a running innfeed.
175 The format of these last three files is one of the following:
179 /path/to/article <message-id>
184 This is the same as the first two fields of the lines innd feeds to
185 innfeed, and the same as the first two fields of the lines of the batch
186 file innd will write if innfeed is unavailable for some reason. When
187 innfeed processes its own batch files it ignores everything after the first
188 two whitespace separated fields, so moving the innd-created batch file to
189 the appropriate spot will work, even though the lines have extra fields.
191 The first field can also be a storage API token. The two types of lines can
192 be intermingled; innfeed will use the storage manager if appropriate and
193 otherwise treat the first field as a filename to read directly.
195 Innfeed writes its current status to the file ``\fIinnfeed.status\fP'' (or
196 the file given by the ``\fI-S\fP'' option). This file contains details on
197 the process as a whole, and on each peer this instance of innfeed is
200 If innfeed is told to send an article to a host it is not managing, then
201 the article information will be put into a file matching the pattern
202 ``\fIinnfeed-dropped.*\fP'', with part of the file name matching the pid of
203 the innfeed process that is writing to it. Innfeed will not process this
204 file except to write to it. If nothing is written to the file then it will
205 be removed if innfeed exits normally.
213 Upon receipt of a SIGALRM innfeed will close the funnel-file specified on
214 the command line, and will reopen it (see funnel file description above).
216 Innfeed with catch SIGINT and will write a large debugging snapshot of the
217 state of the running system.
219 Innfeed will catch SIGHUP and will reload the config file.
224 Innfeed will catch SIGCHLD and will close and reopen all backlog files.
226 Innfeed will catch SIGTERM and will do an orderly shutdown.
228 Upon receipt of a SIGUSR1 innfeed will increment the debugging level by
229 one; receipt of a SIGUSR2 will decrement it by one. The debugging level
230 starts at zero (unless the ``-d'' option it used), in which case no debugging
231 information is emitted. A larger value for the level means more debugging
232 information. Numbers up to 5 are currently useful.
240 There are 3 different categories of syslog entries for statistics: Host,
241 Connection and Global.
243 The Host statistics are generated for a given peer at regular intervals
244 after the first connection is made (or, if the remote is unreachable, after
245 spooling starts). The Host statistics give totals over all Connections that
246 have been active during the given time frame. For example (broken here to
247 fit the page, with ``vixie'' being the peer):
250 May 23 12:49:08 data innfeed[16015]: vixie checkpoint
251 seconds 1381 offered 2744 accepted 1286
252 refused 1021 rejected 437 missing 0 spooled 990
253 on_close 0 unspooled 240 deferred 10 requeued 25
254 queue 42.1/100:14,35,13,4,24,10
257 These meanings of these fields are:
258 .nr XX \w'unspooled '
261 The time since innfeed connected to the host or since the statistics
262 were reset by a ``final'' log entry.
265 The number of IHAVE commands sent to the host if it is not in streaming mode.
266 The sum of the number of TAKETHIS commands sent when no-CHECK mode
267 is in effect plus the number CHECK commands sent in streaming mode (when
268 no-CHECK mode is not in effect).
271 The number of articles which were sent to the remote host and accepted
275 The number of articles offered to the host that it it indicated it
276 didn't want because it had already seen the Message-ID. The remote
277 host indicates this by sending a 435 response to an IHAVE command or
278 a 438 response to a CHECK command.
281 The number of articles transferred to the host that it did not accept
282 because it determined either that it already had the article or it did
283 not want it because of the article's Newsgroups: or Distribution: headers,
284 etc. The remote host indicates that it is rejecting the article by
285 sending a 437 or 439 response after innfeed sent the entire article.
288 The number of articles which innfeed was told to offer to the host but
289 which were not present in the article spool. These articles were probably
290 cancelled or expired before innfeed was able to offer them to the host.
293 The number of article entries that were written to the .output backlog file
294 because the articles could not either be sent to the host or be refused
295 by it. Articles are generally spooled either because new articles are
296 arriving more quickly than they can be offered to the host, or because
297 innfeed closed all the connections to the host and pushed all the
298 articles currently in progress to the .output backlog file.
301 The number of articles that were spooled when innfeed closed all the
302 connections to the host.
305 The number of article entries that were read from the .input backlog
309 The number of articles that the host told innfeed to retry later by
310 sending a 431 or 436 response. Innfeed immediately puts these articles
311 back on the tail of the queue.
314 The number of articles that were in progress on connections when innfeed
315 dropped those connections and put the articles back on the queue. These
316 connections may have been broken by a network problem or became unresponsive
317 causing innfeed to time them out.
320 The first number is the average (mean) queue size during the previous logging
321 interval. The second number is the maximum allowable queue size.
322 The third number is the percentage of the time that the queue
323 was empty. The fourth through seventh numbers are the percentages of the
324 time that the queue was >0% to 25% full, 25% to 50% full, 50% to 75%
325 full, and 75% to <100% full. The last number is the percentage of the
326 time that the queue was totally full.
328 If the ``\fI\-z\fP'' option is used (see below), then when the peer stats are
329 generated, each Connection will log its stats too. For example, for
330 connection number zero (from a set of five):
333 May 23 12:49:08 data innfeed[16015]: vixie:0 checkpoint
334 seconds 1381 offered 596 accepted 274
335 refused 225 rejected 97
338 If you only open a maximum of one Connection to a remote, then there will
339 be a close correlation between Connection numbers and Host numbers, but in
340 general you can't tie the two sets of number together in any easy or very
341 meaningful way. When a Connection closes it will always log its stats.
343 If all Connections for a Host get closed together, then the Host logs its
344 stats as ``final'' and resets its counters. If the feed is so busy that
345 there's always at least one Connection open and running, then after some
346 amount of time (set via the config file), the Host stats are logged as
347 final and reset. This is to make generating higher level stats from log
348 files, by other programs, easier.
350 There is one log entry that is emitted for a Host just after its last
351 Connection closes and innfeed is preparing to exit. This entry contains
352 counts over the entire life of the process. The ``seconds'' field is from the
353 first time a Connection was successfully built, or the first time spooling
354 started. If a Host has been completely idle, it will have no such log entry.
357 May 23 12:49:08 data innfeed[16015]: decwrl global
358 seconds 1381 offered 34 accepted 22
359 refused 3 rejected 7 missing 0
362 The final log entry is emitted immediately before exiting. It contains a
363 summary of the statistics over the entire life of the process.
366 Feb 13 14:43:41 data innfeed-0.9.4[22344]: ME global
367 seconds 15742 offered 273441 accepted 45750
368 refused 222008 rejected 3334 missing 217
379 The ``\fI\-a\fP'' flag is used to specify the top of the article spool
380 tree. Innfeed does a chdir(2) to this directory, so it should probably be
381 an absolute path. The default is <patharticles\ in\ inn.conf>.
384 The ``\fI\-b\fP'' flag may be used to specify a different directory for backlog
385 file storage and retrieval. If the path is relative then it is relative
386 to <pathspool\ in\ inn.conf>. The default is ``\fIinnfeed\fP''.
389 The ``\fI\-c\fP'' flag may be used to specify a different config file from the
390 default value. If the path is relative then it is relative to
391 <pathetc\ in\ inn.conf>. The default is ``\fIinnfeed.conf\fP''.
394 The ``\fI\-C\fP'' flag is used to have innfeed simply check the config
395 file, report on any errors and then exit.
398 The ``\fI\-d\fP'' flag may be used to specify the initial logging level. All
399 debugging messages go to stderr (which may not be what you want, see the
400 ``\fI\-l\fP'' flag below).
403 The ``\fI\-e\fP'' flag may be used to specify the size limit (in bytes) for the
404 \fI\%.output\fP backlog files innfeed creates. If the output file gets bigger
405 than 10% more than the given number, innfeed will replace the output file
406 with the tail of the original version. The default value is 0, which means
410 Use the ``\fI\-h\fP'' flag to print the usage message.
413 The ``\fI\-l\fP'' flag may be used to specify a different log file from
414 stderr. As innd starts innfeed with stderr attached to /dev/null, using this
415 option can be useful in catching any abnormal error messages, or any
416 debugging messages (all ``normal'' errors messages go to syslog).
419 If innfeed has been built with mmap support, then the ``\fI\-M\fP'' flag
420 turns OFF the use of mmap(); otherwise it has no effect.
423 The ``\fI\-m\fP'' flag is used to turn on logging of all missing
424 articles. Normally if an article is missing, innfeed keeps a count, but
425 logs no further information. When this flag is used, details about
426 message-id and expected pathname are logged.
429 The ``\fI\-o\fP'' flag sets a value of the maximum number of bytes of article
430 data innfeed is supposed to keep in memory. This doesn't work properly yet.
433 The ``\fI\-p\fP'' flag is used to specify the filename to write the pid of the
434 process into. A relative path is relative to <pathrun\ in\ inn.conf>. The
435 default is ``\fIinnfeed.pid\fP''.
438 The ``\fI\-S\fP'' flag specifies the name of the file to write the periodic
439 staus to. If the path is relative it is considered relative to
440 <pathlog\ in\ inn.conf>. The default is ``\fIinnfeed.status\fP''.
443 When the ``\fI\-v\fP'' flag is given, version information is printed to stderr
444 and then innfeed exits.
447 The ``\fI\-x\fP'' flag is used to tell innfeed not to expect any article
448 information from innd but just to process any backlog files that exist and
452 The ``\fI\-y\fP'' flag is used to allow dynamic peer binding. If this flag is
453 used and article information is received from innd that specifies an
454 unknown peer, then the peer name is taken to be the IP name too, and an
455 association with it is created. Using this it is possible to only
456 have the global defaults in the
458 file, provided the peername as used by innd is the same as the ip name.
461 with ``\fI\-y\fP'' and no peer in
463 would cause a problem that
465 drops the first article.
468 The ``\fI\-z\fP'' flag is used to cause each connection, in a parallel feed
469 configuration, to report statistics when the controller for the connections
470 prints its statistics.
479 When using the ``-x'' option, the config file entry's
480 ``initial-connections'' field will be the total number of connections
481 created and used, no matter how many big the batch file, and no
482 matter how big the ``max-connectiond'' field specifies. Thus a value
483 of 0 for ``initial-connections'' means nothing will happen in ``-x''
486 Innfeed does not automatically grab the file out of out.going--this needs
487 to be prepared for it by external means.
489 Probably too many other bugs to count.
496 innfeed.conf config file.
498 innfeed directory for backlog files.
505 Written by James Brister <brister@vix.com> for InterNetNews.
507 This is revision \\$3, dated \\$4.
509 .R$ $Id: innfeed.1 7798 2008-04-26 08:47:01Z iulius $
~ian / inn-innduct.git / blob