3 .\" Author: James A. Brister <brister@vix.com> -- berkeley-unix --
4 .\" Start Date: Sun, 21 Jan 1996 00:47:37 +1100
5 .\" Project: INN -- innfeed
6 .\" File: innfeed.conf.5
7 .\" RCSId: $Id: innfeed.conf.5 7778 2008-04-17 21:27:22Z iulius $
8 .\" Description: Man page for innfeed.conf(5)
12 innfeed.conf \- configuration file for innfeed
15 This man page describes the configuration file for version 1.0 of
16 innfeed. This format has changed dramatically since version 0.9.3.
20 is used to control the innfeed(1) program. It is a fairly free-format file
21 that consists of three types of entries: \fIkey/value\fP, \fIpeer\fP and
23 Comments are from the hash character ``#'' to the end of the line.
25 \fIKey/value\fP entries are a keyword and a value separated by a colon
26 (which can itself be surrounded by whitespace). For example:
35 key starts with a letter and contains only letters, digits, and ``_'',
38 There are 5 different type of values: integers, floating-point numbers,
39 characters, booleans, and strings. Integer and floating point numbers are
40 as to be expected except that exponents in floating point numbers are not
41 supported. A boolean value is either ``true'' or ``false'' (case is not
42 significant). A character value is a single-quoted character as defined by
43 the C-language. A string value is any other sequence of characters. If the
44 string needs to contain whitespace, then it must be quoted with double
45 quotes, and uses the same format for embedding non-printing characters as
46 normal C-language string.
48 Peer entries look like:
58 The word ``peer'' is required. The ``<name>'' is the same as the site name
59 in INN's newsfeeds file. The body of a peer entry contains some number
60 (possibly zero) of key/value entries.
62 Group entries look like:
72 The word ``group'' is required. The ``<name>'' is any string valid as a
73 key. The body of a group entry contains any number of the three types of
74 entries. So key/value pairs can be defined inside a group, and peers can be
75 nested inside a group, and other groups can be nested inside a group.
77 Key/value entries that are defined outside of all peer and group entries
78 are said to be at ``global scope''. There are global key/value entries that
79 apply to the process as a whole (for example the location of the backlog
80 file directory), and there are global key/value entries that act as
81 defaults for peers. When innfeed looks for a specific value in a peer entry
82 (for example, the maximum number of connections to set up), if the value is
83 not defined in the peer entry, then the enclosing groups are examined for
84 the entry (starting at the closest enclosing group). If there are no
85 enclosing groups, or the enclosing groups don't define the key/value, then
86 the value at global scope is used.
88 A small example could be:
92 # Global value applied to all peers that have
93 # no value of their own.
96 # A peer definition. ``uunet'' is the name used by innd in
99 ip-name: usenet1.uu.net
103 ip-name: gw.home.vix.com
104 max-connections: 10 # override global value.
107 # A group of two peers who can handle more connections
112 # Another peer. The ``max-connections'' value from the
113 # ``fast-sites'' group scope is used. The ``ip-name'' value
114 # defaults to the peer's name.
115 peer data.ramona.vix.com {
118 peer bb.home.vix.com {
119 max-connections: 20 # he can really cook.
125 Given the above configuration file, the defined peers would have the
126 following values for the ``max-connections'' key.
132 data.ramona.vix.com 15
137 Innfeed ignores key/value pairs it is not interested in. Some config file
138 values can be set via a command line option, in which case that setting
139 overrides the settings in the file.
141 Config files can be included in other config files via the syntax:
149 There is a maximum nesting depth of 10.
151 For a fuller example config file, see the supplied \fIinnfeed.conf\fP.
154 The following listing show all the keys that apply to the process as
155 whole. These are not required (compiled-in defaults are used where needed).
158 This key requires a pathname value. It specifies where the top of the
159 article spool is. This corresponds to the ``\fI\-a\fP'' command-line
163 This key requires a pathname value. It specifies the pathname (relative to
164 the \fBbacklog-directory\fP) that should be read in funnel-file mode. This
165 corresponds to giving a filename as an argument on the command-line (i.e.
166 its presence also implies that funnel-file mode should be used).
169 This key requires a pathname value. It specifies the pathname (relative to
170 the \fBbacklog-directory\fP) where the pid of the innfeed process should be
171 stored. This corresponds to the ``\fI\-p\fP'' command-line option.
174 This key defines the debug level for the process. A non-zero number
175 generates a lot of messages to stderr, or to the config-defined ``log-file''.
176 This corresponds to the ``\fI\-d\fP'' command-line option.
179 This key requires a boolean value. It specifies whether mmaping should be
180 used if innfeed has been built with mmap support. If article data on disk
181 is not in NNTP-ready format (CR/LF at the end of each line), then after
182 mmaping the article is read into memory and fixed up, so mmaping has no
183 positive effect (and possibly some negative effect depending on your
184 system), and so in such a case this value should be \fIfalse\fP. This
185 corresponds to the ``\fI\-M\fP'' command-line option.
188 This key requires a pathname value. It specifies where any logging messages
189 that couldn't be sent via syslog(3) should go (such as those generated when
190 a positive value for ``\fBdebug-value\fP'', is used). This corresponds to
191 the ``\fI\-l\fP'' command-line option. A relative pathname is relative to
192 the ``\fBbacklog-directory\fP'' value.
195 .\" This key requires a positive integer value. It specifies how many seconds
196 .\" innfeed should sleep at startup before attempting to take out its locks. On
197 .\" fast machines and with innfeed handling many connections, it can take too
198 .\" long for innfeed to recognise that its input has been closed, and that it
199 .\" should release any locks it holds.
200 .\"..................................................
203 This key requires a pathname value. It specifies where the current innfeed
204 process should store backlog files. This corresponds to the ``\fI\-b\fP''
208 This key requires a positive integer value. It specifies how many articles
209 should be kept on the backlog file queue before starting to write new
212 .B backlog-ckpt-period
213 This key requires a positive integer value. It specifies how many seconds
214 between checkpoints of the input backlog file. Too small a number will mean
215 frequent disk accesses, too large a number will mean after a crash innfeed
216 will re-offer more already-processed articles than necessary.
218 .B backlog-newfile-period
219 This key requires a positive integer value. It specifies how many seconds
220 before each checks for externally generated backlog files that are to be
221 picked up and processed.
223 .B backlog-rotate-period
224 This key requires a positive integer value. It specifies how many seconds
227 checks for a manually created backlog file and moves the output backlog
228 file to the input backlog file.
229 .\"..................................................
232 This key requires a positive integer value. It defines the number of seconds
233 between attempts to re-lookup host information that previous failed to be
237 This key requires a positive integer value. It defines the number of seconds
238 between refreshes of name to address DNS translation. This is so long-running
239 processes don't get stuck with stale data, should peer ip addresses change.
242 This key requires a positive integer value. It is the maximum number of
243 seconds a connection should be kept open. Some NNTP servers don't deal well
244 with connections being held open for long periods.
247 This key requires a boolean value. It specifies whether the
248 \fBstatus-file\fP should be HTML-ified.
251 This key requires a pathname value. It specifies the pathname (relative to
252 the \fBbacklog-directory\fP) where the periodic status of the innfeed
253 process should be stored. This corresponds to the ``\fI\-S\fP''
257 This key requires a boolean value. If the value is true, then whenever the
258 transmission statistics for a peer are logged, then each active connection
259 logs its own statistics. This corresponds to the ``\fI\-z\fP''
262 .B host-queue-highwater
263 This key requires a positive integer value. It defines how many articles
264 will be held internally for a peer before new arrivals cause article
265 information to be spooled to the backlog file.
268 This key requires a positive integer value. It defines how many seconds
269 innfeed waits between generating statistics on transfer rates.
272 This key requires a positive integer value. It defines how many seconds
273 innfeed waits before resetting all internal transfer counters back to zero
274 (after logging one final time). This is so a innfeed-process running more
275 than a day will generate ``final'' stats that will be picked up by logfile
277 .\"..................................................
279 .B initial-reconnect-time
280 This key requires a positive integer value. It defines how many seconds to
281 first wait before retrying to reconnect after a connection failure. If the
282 next attempt fails too, then the reconnect time is approximately doubled
283 until the connection succeeds, or \fBmax-reconnection-time\fP is reached.
285 .B max-reconnect-time
286 This key requires an integer value. It defines the maximum number of
287 seconds to wait between attempt to reconnect to a peer. The initial value
288 for reconnection attempts is defined by \fBinitial-reconnect-time\fP, and
289 it is doubled after each failure, up to this value.
292 This key requires a non-negative integer value. If the value is greater
293 than zero, then whenever a network socket file descriptor is created and
294 it has a value \fIless than\fP this, the file descriptor will be dup'ed to
295 bring the value up greater than this. This is to leave lower numbered file
296 descriptors free for stdio. Certain systems, Sun's in particular, require
297 this. SunOS 4.1.x usually requires a value of 128 and Solaris requires a
298 value of 256. The default if this is not specified, is 0.
299 .\"..................................................
300 .SH "GLOBAL PEER DEFAULTS"
302 All the key/value pairs mentioned in this section must be specified at
303 global scope. They may also be specified inside a group or peer
304 definition. Note that when peers are added dynamically (i.e. when
305 innfeed receives an article for an unspecified peer), it will add
306 the peer site using the parameters specified at global scope.
309 This key requires a non-negative integer value. If no articles need to be
310 sent to the peer for this many seconds, then the peer is considered idle
311 and all its active connections are torn down.
314 This key requires a non-negative integer value. It defines the maximum
315 amount of time to wait for a response from the peer after issuing a
318 .B initial-connections
319 This key requires a non-negative integer value. It defines the number of
320 connections to be opened immediately when setting up a peer binding. A
321 value of 0 means no connections will be created until an article needs to
325 This key requires positive integer value. It defines the maximum number of
326 connections to run in parallel to the peer. A value of zero specifies an
327 unlimited number of maximum connections. In general use of an unlimited
328 number of maximum connections is not recommended. Do not ever set
329 \fBmax-connections\fP to zero with \fBdynamic-method\fP 0 set, as this will
330 saturate peer hosts with connections. [ Note that in previous versions
331 of innfeed, a value of 1 had a special meaning. This is no longer the case,
332 1 means a maximum of 1 connection ].
335 This key requires an integer value between 0 and 3. It controls how connections
336 (up to max-connections) are opened, up to the maximum specified by
337 \fBmax-connections\fP. In general (and specifically, with \fBdynamic-method\fP
338 0), a new connection is opened when the current number of connections is
339 below \fBmax-connections\fP, and an article is to be sent while no current
340 connections are idle. Without further restraint (i.e. using
341 \fBdynamic-method\fP 0), in practice this means that \fBmax-connections\fP
342 connections are established while articles are being sent. Use of other
343 \fBdynamic-method\fP settings imposes a further limit on the amount of
344 connections opened below that specified by \fBmax-connections\fP. This
345 limit is calculated in different ways, depending of the value of
346 \fBdynamic-method\fP.
347 Users should note that adding additional connections is not always
348 productive - just because opening twice as many connections results
349 in a small percentage increase of articles accepted by the remote peer,
350 this may be at considerable resource cost both locally and at the remote
351 site, whereas the remote site might well have received the extra articles
352 sent from another peer a fraction of a second later. Opening large
353 numbers of connections is considered antisocial.
354 The meanings of the various settings are:
358 Increase of connections up to \fBmax-connections\fP is unrestrained.
360 .B 1 maximize articles per second
361 Connections are increased (up to \fBmax-connections\fP) and decreased so as
362 to maximize the number of articles per second sent, while using the fewest
363 connections to do this.
365 .B 2 set target queue length
366 Connections are increased (up to \fBmax-connections\fP) and decreased so as
367 to keep the queue of articles to be sent within the bounds set by
368 \fBdynamic-backlog-low\fP and \fBdynamic-backlog-high\fP,
369 while using the minimum resources possible.
370 As the queue will tend to fill if the site is not keeping up, this method
371 ensures that the maximum number of articles are offered to the peer
372 while using the minimum number of connections to achieve this.
375 This method uses a combination of methods 1 and 2 above. For sites
376 accepting a large percentage of articles, method 2 will be used to
377 ensure these sites are offered as complete a feed as possible. For sites
378 accepting a small percentage of articles, method 1 is used, to minimize
379 remote resource usage. For intermediate sites, an appropriate combination
383 .B dynamic-backlog-low
384 This key requires an integer value between 0 and 100. It represents (as a
385 percentage) the low water mark for the host queue. If the host queue falls
386 below this level while using \fBdynamic-method\fP 2 or 3, and if 2 or more
387 connections are open, innfeed will attempt to drop connections to the host.
388 An IIR filter is applied to the value to prevent connection flap (see
389 \fBdynamic-filter\fP). A value of 25.0 is recommended. This value
390 must be smaller than \fBdynamic-backlog-high\fP.
392 .B dynamic-backlog-high
393 This key requries an integer value between 0 and 100. It represents (as a
394 percentage) the high water mark for the host queue. If the host queue rises
395 above this level while using \fBdynamic-method\fP 2 or 3, and if less than
396 \fBmax-connections\fP are open to the host, innfeed will attempt
397 to open further connections to the host. An IIR filter is applied to the value
398 to prevent connection flap (see \fBdynamic-filter\fP). A value of 50.0
399 is recommended. This value must be larger than \fBdynamic-backlog-low\fP.
401 .B dynamic-backlog-filter
402 This key requires a floating-point value between 0 and 1. It represents the
403 filter coefficient used by the IIR filter used to implement
404 \fBdynamic-method\fP 2 and 3.
405 The recommended value of this filter is 0.7, giving a time
406 constant of 1/(1-0.7) articles. Higher values will result in slower
407 response to queue fullness changes, lower values in faster response.
410 This key requires a positive integer value. It defines the maximum number
411 of articles to process at one time when using streaming to transmit to a
412 peer. Larger numbers mean more memory consumed as articles usually get
413 pulled into memory (see the description of \fBuse-mmap\fP).
416 This key requires a boolean value. It defines whether streaming commands
417 are used to transmit articles to the peers.
420 This key requires a floating-point number which must be in the range [0.0,
421 100.0]. When running transmitting with the streaming commands, innfeed
422 attempts an optimization called ``no-CHECK'' mode. This involves \fInot\fP
423 asking the peer if it wants the article, but just sending it. This
424 optimization occurs when the percentage of the articles the peer has
425 accepted gets larger than this number. If this value is set to 100.0, then
426 this effectively turns off no-CHECK mode, as the percentage can never get
427 above 100.0. If this value is too small, then the number of articles the
428 peer rejects will get bigger (and your bandwidth will be wasted). A value
429 of 95.0 usually works pretty well. NOTE: In innfeed 0.9.3 and earlier this
430 value was in the range [0.0, 9.0].
433 This key requires a floating-point number which must be in the range [0.0,
434 100.0), and it must be smaller that the value for \fBno-check-high\fP. When
435 running in no-CHECK mode, as described above, if the percentage of articles
436 the remote accepts drops below this number, then the no-CHECK optimization
437 is turned off until the percentage gets above the \fBno-check-high\fP value
438 again. If there is small difference between this and the
439 \fBno-check-high\fP value (less than about 5.0), then innfeed may
440 frequently go in and out of no-CHECK mode. If the difference is too big,
441 then it will make it harder to get out of no-CHECK mode when necessary
442 (wasting bandwidth). Keeping this to between 5.0 and 10.0 less than
443 \fBno-check-high\fP usually works pretty well.
446 This is a floating point value representing the time constant, in articles,
447 over which the CHECK / no-CHECK calculations are done. The recommended
448 value is 50.0 which will implement an IIR filter of time constant 50. This
449 roughly equates to making a decision about the mode over the previous
450 50 articles. A higher number will result in a slower response to changing
451 percentages of articles accepted; a lower number will result in a faster
455 This key requires a string value. It specifies which outgoing IPv4 address
456 innfeed should bind the local end of its connection to.
457 Must be an IPv4 address in dotted-quad format (nnn.nnn.nnn.nnn), "any",
458 or "none". If not set or set to "any", innfeed defaults
459 to letting the kernel choose this address.
460 If set to "none", innfeed will not use IPv4 for outgoing connections
461 to peers in this scope (i.e. it forces IPv6).
462 The default value is unset.
465 This key requires a string value. It behaves like \fBbindaddress\fP except
466 for outgoing IPv6 connections. Must be in numeric IPv6 format, "any",
467 or "none". If set to "none", innfeed will not use IPv6 for outgoing
468 connections to peers in this scope. A value containing colons must be
469 enclosed in double quotes.
472 This key requires a positive integer value. It defines the tcp/ip port
473 number to use when connecting to the remote.
476 This key requires a boolean value. By default it is set to false.
477 Setting it to true is the same as setting "bindaddress6: none"
478 and removing "bindaddress: none" if it was set.
481 This key requires a boolean value. By default it is set to false. When
482 set to true, and a peer replies with code 431 or 436 (try again later) just
483 drop the article and don't try to re-send it. This is useful for some
484 peers that keep on deferring articles for a long time to prevent innfeed
485 from trying to offer the same article over and over again.
487 .B min-queue-connection
488 This key requires a boolean value. By default it is set to false. When
489 set to true, innfeed will attempt to use a connection with the least queue
490 size (or the first empty connection). If this key is set to true, it is
491 recommended that \fBdynamic-method\fP be set to 0. This allows for article
492 propagation with the least delay.
495 This key requires a boolean value. It specifies whether spooling should
496 be enabled (false, the default) or disabled (true). Note that when no-backlog
497 is set, articles reported as "spooled" are actually silently discarded.
500 This key requires a non-negative integer value. If the number is 0 then
501 backlog files are allowed to grown without bound when the peer is unable to
502 keep up with the article flow. If this number is greater than 0 then it
503 specifies the size (in bytes) the backlog file should get truncated to when
504 the backlog file reaches a certain limit. The limit depends on whether
505 \fBbacklog-factor\fP or \fBbacklog-limit-highwater\fP is used.
508 This key requires a floating point value, which must be larger than 1.0. It
509 is used in conjunction with the peer key \fBbacklog-limit\fP. If
510 \fBbacklog-limit\fP has a value greater than zero, then when the backlog
511 file gets larger than the value \fBbacklog-limit * backlog-factor\fP, then
512 the backlog file will be truncated to the size \fBbacklog-limit\fP. For
513 example if \fBbacklog-limit\fP has a value of 1000000, and
514 \fBbacklog-factor\fP has a value of 2.0, then when the backlogfile gets to
515 be larger than 2000000 bytes in size, it will be truncated to 1000000 bytes.
517 portion of the file is removed, and the trimming happens on line boundaries,
518 so the final size may be a bit less than this number. If
519 \fBbacklog-limit-highwater\fP is defined too, then \fBbacklog-factor\fP
522 .B backlog-limit-highwater
523 This key requires a positive integer value that must be larger than the
524 value for \fBbacklog-limit\fP. If the size of the backlog file gets larger
525 than this value (in bytes), then the backlog file will be shrunk down to
526 the size of \fBbacklog-limit\fP. If both \fBbacklog-factor\fP and
527 \fBbacklog-limit-highwater\fP are defined, then the value of
528 \fBbacklog-factor\fP is used.
530 .B backlog-feed-first
531 This key requires a boolean value. By default it is set to false. When set
532 to true, the backlog is fed before new files. This is intended to enforce
533 in-order delivery, so setting this to true when initial-connections or
534 max-connections is more than 1 is inconsistent.
537 This key requires a string value. If the value is defined, then innfeed
538 tries to authenticate by ``AUTHINFO USER'' and this value used for user name.
539 \fBpassword\fP must also be defined, if this key is defined.
542 This key requires a string value. The value is the password
543 used for ``AUTHINFO PASS''.
544 \fBusername\fP must also be defined, if this key is defined.
547 This key is used with imapfeed to authenticate to a remote host. It is optional.
548 There are several parameters that must be included with deliver:
552 The authname is who you want to authenticate as.
555 This is the appropriate password for authname.
558 The username is who you want to "act" as, that is, who is actually
559 going to be using the server.
562 In this case, the "realm" is the realm in which the specified authname
563 is valid. Currently this is only needed by the DIGEST-MD5 SASL
567 A printf-style format string for creating the envelope recipient address.
568 The pattern MUST include a single string specifier which will be
569 replaced with the newgroup (e.g "bb+%s"). The default is "+%s".
572 An optional printf-style format string for creating a To: header to be
573 prepended to the article. The pattern MUST include a single string
574 specifier which will be replaced with the newgroup (e.g
575 "post+%s@domain"). If not specified, the To: header will not be
578 .\"..................................................
580 As previously explained, the peer definitions can contain redefinitions of
581 any of the key/value pairs described in the \fBGLOBAL PEER DEFAULTS\fP
582 section above. There is one key/value pair that is specific to a peer
586 This key requires a word value. The word is the host's FQDN, or the dotted
587 quad ip-address. If this value is not specified then the name of the peer
588 is taken to also be its ip-name. See the entry for
589 data.ramona.vix.com in the example below.
590 .\"..................................................
593 If innfeed gets a SIGHUP signal, then it will reread the config file. All
594 values at global scope except for ``\fBbacklog-directory\fP'' can be
595 changed (although note that ``\fBbindaddress\fP'' and
596 ``\fBbindaddress6\fP'' changes will only affect new connections). Any new
597 peers are added and any missing peers have their connections closed.
598 .\"..................................................
601 Below is the sample innfeed.conf file.
605 # innfeed.conf file. See the comment block at the
606 # end for a fuller description.
610 ## Global values. Not specific to any peer. These
611 ## are optional, but if used will override the
612 ## compiled in values. Command-line options used
613 ## will override these values.
616 pid-file: innfeed.pid
619 log-file: innfeed.log
622 backlog-directory: innfeed
623 backlog-rotate-period: 60
624 backlog-ckpt-period: 30
625 backlog-newfile-period: 600
631 status-file: innfeed.status
632 connection-stats: false
633 host-queue-highwater: 200
637 max-reconnect-time: 3600
638 initial-reconnect-time: 30
642 ## Defaults for all peers. These must all exist at
643 ## global scope. Any of them can be redefined
644 ## inside a peer or group definition.
648 response-timeout: 300
649 initial-connections: 1
655 no-check-filter: 50.0
659 backlog-limit-highwater:0
661 dynamic-backlog-filter: 0.7
662 dynamic-backlog-low: 25.0
663 dynamic-backlog-high: 50.0
670 ip-name: news1.pa.dec.com
674 ip-name: news.uunet.uu.net
678 peer data.ramona.vix.com {
679 # ip-name defaults to data.ramona.vix.com
683 peer bb.home.vix.com {
689 # Blank lines are ignored. Everything after a '#'
695 # See innfeed.conf(5) for a description of
696 # necessary & useful keys. Unknown keys and their
697 # values are ignored.
699 # Values may be a integer, floating-point, c-style
700 # single-quoted characters, boolean, and strings.
702 # If a string value contains whitespace, or
703 # embedded quotes, or the comment character
704 # (``#''), then the whole string must be quoted
705 # with double quotes. Inside the quotes, you may
706 # use the standard c-escape sequence
707 # (\\t,\\n,\\r,\\f,\\v,\\",\\').
710 # eg-string: "New\\tConfig\\tfile\\n"
711 # eg-long-string: "A long string that goes
712 # over multiple lines. The
713 # newline is kept in the
714 # string except when quoted
715 # with a backslash \\
717 # eg-simple-string: A-no-quote-string
726 Written by James Brister <brister@vix.com> for InterNetNews.
728 This is revision \\$3, dated \\$4.
730 .R$ $Id: innfeed.conf.5 7778 2008-04-17 21:27:22Z iulius $
732 innfeed(1), newsfeeds(5)
~ian / inn-innduct.git / blob