.\" -*- nroff -*- .\" .\" Author: James A. Brister -- berkeley-unix -- .\" Start Date: Sun, 21 Jan 1996 00:47:37 +1100 .\" Project: INN -- innfeed .\" File: innfeed.conf.5 .\" RCSId: $Id: innfeed.conf.5 7778 2008-04-17 21:27:22Z iulius $ .\" Description: Man page for innfeed.conf(5) .\" .TH innfeed.conf 5 .SH NAME innfeed.conf \- configuration file for innfeed .SH DESCRIPTION .PP This man page describes the configuration file for version 1.0 of innfeed. This format has changed dramatically since version 0.9.3. .PP The file .B innfeed.conf is used to control the innfeed(1) program. It is a fairly free-format file that consists of three types of entries: \fIkey/value\fP, \fIpeer\fP and \fIgroup\fP. Comments are from the hash character ``#'' to the end of the line. .PP \fIKey/value\fP entries are a keyword and a value separated by a colon (which can itself be surrounded by whitespace). For example: .PP .RS .nf max-connections: 10 .fi .RE .PP A legal key starts with a letter and contains only letters, digits, and ``_'', ``-''. .LP There are 5 different type of values: integers, floating-point numbers, characters, booleans, and strings. Integer and floating point numbers are as to be expected except that exponents in floating point numbers are not supported. A boolean value is either ``true'' or ``false'' (case is not significant). A character value is a single-quoted character as defined by the C-language. A string value is any other sequence of characters. If the string needs to contain whitespace, then it must be quoted with double quotes, and uses the same format for embedding non-printing characters as normal C-language string. .PP Peer entries look like: .PP .RS .nf peer { # body ... } .fi .RE .PP The word ``peer'' is required. The ``'' is the same as the site name in INN's newsfeeds file. The body of a peer entry contains some number (possibly zero) of key/value entries. .PP Group entries look like: .PP .RS .nf group { # body } .fi .RE .PP The word ``group'' is required. The ``'' is any string valid as a key. The body of a group entry contains any number of the three types of entries. So key/value pairs can be defined inside a group, and peers can be nested inside a group, and other groups can be nested inside a group. .PP Key/value entries that are defined outside of all peer and group entries are said to be at ``global scope''. There are global key/value entries that apply to the process as a whole (for example the location of the backlog file directory), and there are global key/value entries that act as defaults for peers. When innfeed looks for a specific value in a peer entry (for example, the maximum number of connections to set up), if the value is not defined in the peer entry, then the enclosing groups are examined for the entry (starting at the closest enclosing group). If there are no enclosing groups, or the enclosing groups don't define the key/value, then the value at global scope is used. .PP A small example could be: .PP .RS .nf # Global value applied to all peers that have # no value of their own. max-connections: 5 # A peer definition. ``uunet'' is the name used by innd in # the newsfeeds file. peer uunet { ip-name: usenet1.uu.net } peer vixie { ip-name: gw.home.vix.com max-connections: 10 # override global value. } # A group of two peers who can handle more connections # than normal group fast-sites { max-connections: 15 # Another peer. The ``max-connections'' value from the # ``fast-sites'' group scope is used. The ``ip-name'' value # defaults to the peer's name. peer data.ramona.vix.com { } peer bb.home.vix.com { max-connections: 20 # he can really cook. } } .fi .RE .PP Given the above configuration file, the defined peers would have the following values for the ``max-connections'' key. .PP .RS .nf uunet 5 vixie 10 data.ramona.vix.com 15 bb.home.vix.com 20 .fi .RE .PP Innfeed ignores key/value pairs it is not interested in. Some config file values can be set via a command line option, in which case that setting overrides the settings in the file. .PP Config files can be included in other config files via the syntax: .sp 1 .nf .RS $INCLUDE filename .RE .fi .sp 1 There is a maximum nesting depth of 10. .PP For a fuller example config file, see the supplied \fIinnfeed.conf\fP. .SH "GLOBAL VALUES" .PP The following listing show all the keys that apply to the process as whole. These are not required (compiled-in defaults are used where needed). .TP .B news-spool This key requires a pathname value. It specifies where the top of the article spool is. This corresponds to the ``\fI\-a\fP'' command-line option. .TP .B input-file This key requires a pathname value. It specifies the pathname (relative to the \fBbacklog-directory\fP) that should be read in funnel-file mode. This corresponds to giving a filename as an argument on the command-line (i.e. its presence also implies that funnel-file mode should be used). .TP .B pid-file This key requires a pathname value. It specifies the pathname (relative to the \fBbacklog-directory\fP) where the pid of the innfeed process should be stored. This corresponds to the ``\fI\-p\fP'' command-line option. .TP .B debug-level This key defines the debug level for the process. A non-zero number generates a lot of messages to stderr, or to the config-defined ``log-file''. This corresponds to the ``\fI\-d\fP'' command-line option. .TP .B use-mmap This key requires a boolean value. It specifies whether mmaping should be used if innfeed has been built with mmap support. If article data on disk is not in NNTP-ready format (CR/LF at the end of each line), then after mmaping the article is read into memory and fixed up, so mmaping has no positive effect (and possibly some negative effect depending on your system), and so in such a case this value should be \fIfalse\fP. This corresponds to the ``\fI\-M\fP'' command-line option. .TP .B log-file This key requires a pathname value. It specifies where any logging messages that couldn't be sent via syslog(3) should go (such as those generated when a positive value for ``\fBdebug-value\fP'', is used). This corresponds to the ``\fI\-l\fP'' command-line option. A relative pathname is relative to the ``\fBbacklog-directory\fP'' value. .\" .TP .\" .B initial-sleep .\" This key requires a positive integer value. It specifies how many seconds .\" innfeed should sleep at startup before attempting to take out its locks. On .\" fast machines and with innfeed handling many connections, it can take too .\" long for innfeed to recognise that its input has been closed, and that it .\" should release any locks it holds. .\".................................................. .TP .B backlog-directory This key requires a pathname value. It specifies where the current innfeed process should store backlog files. This corresponds to the ``\fI\-b\fP'' command-line option. .TP .B backlog-highwater This key requires a positive integer value. It specifies how many articles should be kept on the backlog file queue before starting to write new entries to disk. .TP .B backlog-ckpt-period This key requires a positive integer value. It specifies how many seconds between checkpoints of the input backlog file. Too small a number will mean frequent disk accesses, too large a number will mean after a crash innfeed will re-offer more already-processed articles than necessary. .TP .B backlog-newfile-period This key requires a positive integer value. It specifies how many seconds before each checks for externally generated backlog files that are to be picked up and processed. .TP .B backlog-rotate-period This key requires a positive integer value. It specifies how many seconds elapse before .B innfeed checks for a manually created backlog file and moves the output backlog file to the input backlog file. .\".................................................. .TP .B dns-retry This key requires a positive integer value. It defines the number of seconds between attempts to re-lookup host information that previous failed to be resolved. .TP .B dns-expire This key requires a positive integer value. It defines the number of seconds between refreshes of name to address DNS translation. This is so long-running processes don't get stuck with stale data, should peer ip addresses change. .TP .B close-period This key requires a positive integer value. It is the maximum number of seconds a connection should be kept open. Some NNTP servers don't deal well with connections being held open for long periods. .TP .B gen-html This key requires a boolean value. It specifies whether the \fBstatus-file\fP should be HTML-ified. .TP .B status-file This key requires a pathname value. It specifies the pathname (relative to the \fBbacklog-directory\fP) where the periodic status of the innfeed process should be stored. This corresponds to the ``\fI\-S\fP'' command-line option. .TP .B connection-stats This key requires a boolean value. If the value is true, then whenever the transmission statistics for a peer are logged, then each active connection logs its own statistics. This corresponds to the ``\fI\-z\fP'' command-line option. .TP .B host-queue-highwater This key requires a positive integer value. It defines how many articles will be held internally for a peer before new arrivals cause article information to be spooled to the backlog file. .TP .B stats-period This key requires a positive integer value. It defines how many seconds innfeed waits between generating statistics on transfer rates. .TP .B stats-reset This key requires a positive integer value. It defines how many seconds innfeed waits before resetting all internal transfer counters back to zero (after logging one final time). This is so a innfeed-process running more than a day will generate ``final'' stats that will be picked up by logfile processing scripts. .\".................................................. .TP .B initial-reconnect-time This key requires a positive integer value. It defines how many seconds to first wait before retrying to reconnect after a connection failure. If the next attempt fails too, then the reconnect time is approximately doubled until the connection succeeds, or \fBmax-reconnection-time\fP is reached. .TP .B max-reconnect-time This key requires an integer value. It defines the maximum number of seconds to wait between attempt to reconnect to a peer. The initial value for reconnection attempts is defined by \fBinitial-reconnect-time\fP, and it is doubled after each failure, up to this value. .TP .B stdio-fdmax This key requires a non-negative integer value. If the value is greater than zero, then whenever a network socket file descriptor is created and it has a value \fIless than\fP this, the file descriptor will be dup'ed to bring the value up greater than this. This is to leave lower numbered file descriptors free for stdio. Certain systems, Sun's in particular, require this. SunOS 4.1.x usually requires a value of 128 and Solaris requires a value of 256. The default if this is not specified, is 0. .\".................................................. .SH "GLOBAL PEER DEFAULTS" .PP All the key/value pairs mentioned in this section must be specified at global scope. They may also be specified inside a group or peer definition. Note that when peers are added dynamically (i.e. when innfeed receives an article for an unspecified peer), it will add the peer site using the parameters specified at global scope. .TP .B article-timeout This key requires a non-negative integer value. If no articles need to be sent to the peer for this many seconds, then the peer is considered idle and all its active connections are torn down. .TP .B response-timeout This key requires a non-negative integer value. It defines the maximum amount of time to wait for a response from the peer after issuing a command. .TP .B initial-connections This key requires a non-negative integer value. It defines the number of connections to be opened immediately when setting up a peer binding. A value of 0 means no connections will be created until an article needs to be sent. .TP .B max-connections This key requires positive integer value. It defines the maximum number of connections to run in parallel to the peer. A value of zero specifies an unlimited number of maximum connections. In general use of an unlimited number of maximum connections is not recommended. Do not ever set \fBmax-connections\fP to zero with \fBdynamic-method\fP 0 set, as this will saturate peer hosts with connections. [ Note that in previous versions of innfeed, a value of 1 had a special meaning. This is no longer the case, 1 means a maximum of 1 connection ]. .TP .B dynamic-method This key requires an integer value between 0 and 3. It controls how connections (up to max-connections) are opened, up to the maximum specified by \fBmax-connections\fP. In general (and specifically, with \fBdynamic-method\fP 0), a new connection is opened when the current number of connections is below \fBmax-connections\fP, and an article is to be sent while no current connections are idle. Without further restraint (i.e. using \fBdynamic-method\fP 0), in practice this means that \fBmax-connections\fP connections are established while articles are being sent. Use of other \fBdynamic-method\fP settings imposes a further limit on the amount of connections opened below that specified by \fBmax-connections\fP. This limit is calculated in different ways, depending of the value of \fBdynamic-method\fP. Users should note that adding additional connections is not always productive - just because opening twice as many connections results in a small percentage increase of articles accepted by the remote peer, this may be at considerable resource cost both locally and at the remote site, whereas the remote site might well have received the extra articles sent from another peer a fraction of a second later. Opening large numbers of connections is considered antisocial. The meanings of the various settings are: .RS .TP .B 0 no method Increase of connections up to \fBmax-connections\fP is unrestrained. .TP .B 1 maximize articles per second Connections are increased (up to \fBmax-connections\fP) and decreased so as to maximize the number of articles per second sent, while using the fewest connections to do this. .TP .B 2 set target queue length Connections are increased (up to \fBmax-connections\fP) and decreased so as to keep the queue of articles to be sent within the bounds set by \fBdynamic-backlog-low\fP and \fBdynamic-backlog-high\fP, while using the minimum resources possible. As the queue will tend to fill if the site is not keeping up, this method ensures that the maximum number of articles are offered to the peer while using the minimum number of connections to achieve this. .TP .B 3 combination This method uses a combination of methods 1 and 2 above. For sites accepting a large percentage of articles, method 2 will be used to ensure these sites are offered as complete a feed as possible. For sites accepting a small percentage of articles, method 1 is used, to minimize remote resource usage. For intermediate sites, an appropriate combination is used. .RE .TP .B dynamic-backlog-low This key requires an integer value between 0 and 100. It represents (as a percentage) the low water mark for the host queue. If the host queue falls below this level while using \fBdynamic-method\fP 2 or 3, and if 2 or more connections are open, innfeed will attempt to drop connections to the host. An IIR filter is applied to the value to prevent connection flap (see \fBdynamic-filter\fP). A value of 25.0 is recommended. This value must be smaller than \fBdynamic-backlog-high\fP. .TP .B dynamic-backlog-high This key requries an integer value between 0 and 100. It represents (as a percentage) the high water mark for the host queue. If the host queue rises above this level while using \fBdynamic-method\fP 2 or 3, and if less than \fBmax-connections\fP are open to the host, innfeed will attempt to open further connections to the host. An IIR filter is applied to the value to prevent connection flap (see \fBdynamic-filter\fP). A value of 50.0 is recommended. This value must be larger than \fBdynamic-backlog-low\fP. .TP .B dynamic-backlog-filter This key requires a floating-point value between 0 and 1. It represents the filter coefficient used by the IIR filter used to implement \fBdynamic-method\fP 2 and 3. The recommended value of this filter is 0.7, giving a time constant of 1/(1-0.7) articles. Higher values will result in slower response to queue fullness changes, lower values in faster response. .TP .B max-queue-size This key requires a positive integer value. It defines the maximum number of articles to process at one time when using streaming to transmit to a peer. Larger numbers mean more memory consumed as articles usually get pulled into memory (see the description of \fBuse-mmap\fP). .TP .B streaming This key requires a boolean value. It defines whether streaming commands are used to transmit articles to the peers. .TP .B no-check-high This key requires a floating-point number which must be in the range [0.0, 100.0]. When running transmitting with the streaming commands, innfeed attempts an optimization called ``no-CHECK'' mode. This involves \fInot\fP asking the peer if it wants the article, but just sending it. This optimization occurs when the percentage of the articles the peer has accepted gets larger than this number. If this value is set to 100.0, then this effectively turns off no-CHECK mode, as the percentage can never get above 100.0. If this value is too small, then the number of articles the peer rejects will get bigger (and your bandwidth will be wasted). A value of 95.0 usually works pretty well. NOTE: In innfeed 0.9.3 and earlier this value was in the range [0.0, 9.0]. .TP .B no-check-low: This key requires a floating-point number which must be in the range [0.0, 100.0), and it must be smaller that the value for \fBno-check-high\fP. When running in no-CHECK mode, as described above, if the percentage of articles the remote accepts drops below this number, then the no-CHECK optimization is turned off until the percentage gets above the \fBno-check-high\fP value again. If there is small difference between this and the \fBno-check-high\fP value (less than about 5.0), then innfeed may frequently go in and out of no-CHECK mode. If the difference is too big, then it will make it harder to get out of no-CHECK mode when necessary (wasting bandwidth). Keeping this to between 5.0 and 10.0 less than \fBno-check-high\fP usually works pretty well. .TP .B no-check-filter This is a floating point value representing the time constant, in articles, over which the CHECK / no-CHECK calculations are done. The recommended value is 50.0 which will implement an IIR filter of time constant 50. This roughly equates to making a decision about the mode over the previous 50 articles. A higher number will result in a slower response to changing percentages of articles accepted; a lower number will result in a faster response. .TP .B bindaddress This key requires a string value. It specifies which outgoing IPv4 address innfeed should bind the local end of its connection to. Must be an IPv4 address in dotted-quad format (nnn.nnn.nnn.nnn), "any", or "none". If not set or set to "any", innfeed defaults to letting the kernel choose this address. If set to "none", innfeed will not use IPv4 for outgoing connections to peers in this scope (i.e. it forces IPv6). The default value is unset. .TP .B bindaddress6 This key requires a string value. It behaves like \fBbindaddress\fP except for outgoing IPv6 connections. Must be in numeric IPv6 format, "any", or "none". If set to "none", innfeed will not use IPv6 for outgoing connections to peers in this scope. A value containing colons must be enclosed in double quotes. .TP .B port-number This key requires a positive integer value. It defines the tcp/ip port number to use when connecting to the remote. .TP .B force-ipv4 This key requires a boolean value. By default it is set to false. Setting it to true is the same as setting "bindaddress6: none" and removing "bindaddress: none" if it was set. .TP .B drop-deferred This key requires a boolean value. By default it is set to false. When set to true, and a peer replies with code 431 or 436 (try again later) just drop the article and don't try to re-send it. This is useful for some peers that keep on deferring articles for a long time to prevent innfeed from trying to offer the same article over and over again. .TP .B min-queue-connection This key requires a boolean value. By default it is set to false. When set to true, innfeed will attempt to use a connection with the least queue size (or the first empty connection). If this key is set to true, it is recommended that \fBdynamic-method\fP be set to 0. This allows for article propagation with the least delay. .TP .B no-backlog This key requires a boolean value. It specifies whether spooling should be enabled (false, the default) or disabled (true). Note that when no-backlog is set, articles reported as "spooled" are actually silently discarded. .TP .B backlog-limit This key requires a non-negative integer value. If the number is 0 then backlog files are allowed to grown without bound when the peer is unable to keep up with the article flow. If this number is greater than 0 then it specifies the size (in bytes) the backlog file should get truncated to when the backlog file reaches a certain limit. The limit depends on whether \fBbacklog-factor\fP or \fBbacklog-limit-highwater\fP is used. .TP .B backlog-factor This key requires a floating point value, which must be larger than 1.0. It is used in conjunction with the peer key \fBbacklog-limit\fP. If \fBbacklog-limit\fP has a value greater than zero, then when the backlog file gets larger than the value \fBbacklog-limit * backlog-factor\fP, then the backlog file will be truncated to the size \fBbacklog-limit\fP. For example if \fBbacklog-limit\fP has a value of 1000000, and \fBbacklog-factor\fP has a value of 2.0, then when the backlogfile gets to be larger than 2000000 bytes in size, it will be truncated to 1000000 bytes. The front portion of the file is removed, and the trimming happens on line boundaries, so the final size may be a bit less than this number. If \fBbacklog-limit-highwater\fP is defined too, then \fBbacklog-factor\fP takes precedence. .TP .B backlog-limit-highwater This key requires a positive integer value that must be larger than the value for \fBbacklog-limit\fP. If the size of the backlog file gets larger than this value (in bytes), then the backlog file will be shrunk down to the size of \fBbacklog-limit\fP. If both \fBbacklog-factor\fP and \fBbacklog-limit-highwater\fP are defined, then the value of \fBbacklog-factor\fP is used. .TP .B backlog-feed-first This key requires a boolean value. By default it is set to false. When set to true, the backlog is fed before new files. This is intended to enforce in-order delivery, so setting this to true when initial-connections or max-connections is more than 1 is inconsistent. .TP .B username This key requires a string value. If the value is defined, then innfeed tries to authenticate by ``AUTHINFO USER'' and this value used for user name. \fBpassword\fP must also be defined, if this key is defined. .TP .B password This key requires a string value. The value is the password used for ``AUTHINFO PASS''. \fBusername\fP must also be defined, if this key is defined. .TP .B deliver This key is used with imapfeed to authenticate to a remote host. It is optional. There are several parameters that must be included with deliver: .RS .TP .B deliver-authname The authname is who you want to authenticate as. .TP .B deliver-password This is the appropriate password for authname. .TP .B deliver-username The username is who you want to "act" as, that is, who is actually going to be using the server. .TP .B deliver-realm In this case, the "realm" is the realm in which the specified authname is valid. Currently this is only needed by the DIGEST-MD5 SASL mechanism. .TP .B deliver-rcpt-to A printf-style format string for creating the envelope recipient address. The pattern MUST include a single string specifier which will be replaced with the newgroup (e.g "bb+%s"). The default is "+%s". .TP .B deliver-to-header An optional printf-style format string for creating a To: header to be prepended to the article. The pattern MUST include a single string specifier which will be replaced with the newgroup (e.g "post+%s@domain"). If not specified, the To: header will not be prepended. .RE .\".................................................. .SH "PEER VALUES" As previously explained, the peer definitions can contain redefinitions of any of the key/value pairs described in the \fBGLOBAL PEER DEFAULTS\fP section above. There is one key/value pair that is specific to a peer definition. .TP .B ip-name This key requires a word value. The word is the host's FQDN, or the dotted quad ip-address. If this value is not specified then the name of the peer is taken to also be its ip-name. See the entry for data.ramona.vix.com in the example below. .\".................................................. .SH RELOADING .PP If innfeed gets a SIGHUP signal, then it will reread the config file. All values at global scope except for ``\fBbacklog-directory\fP'' can be changed (although note that ``\fBbindaddress\fP'' and ``\fBbindaddress6\fP'' changes will only affect new connections). Any new peers are added and any missing peers have their connections closed. .\".................................................. .SH EXAMPLE .PP Below is the sample innfeed.conf file. .RS .nf # # innfeed.conf file. See the comment block at the # end for a fuller description. # ## ## Global values. Not specific to any peer. These ## are optional, but if used will override the ## compiled in values. Command-line options used ## will override these values. ## pid-file: innfeed.pid debug-level: 0 use-mmap: false log-file: innfeed.log stdio-fdmax: 0 backlog-directory: innfeed backlog-rotate-period: 60 backlog-ckpt-period: 30 backlog-newfile-period: 600 dns-retry: 900 dns-expire: 86400 close-period: 3600 gen-html: false status-file: innfeed.status connection-stats: false host-queue-highwater: 200 stats-period: 600 stats-reset: 43200 max-reconnect-time: 3600 initial-reconnect-time: 30 ## ## Defaults for all peers. These must all exist at ## global scope. Any of them can be redefined ## inside a peer or group definition. ## article-timeout: 600 response-timeout: 300 initial-connections: 1 max-connections: 5 max-queue-size: 25 streaming: true no-check-high: 95.0 no-check-low: 90.0 no-check-filter: 50.0 port-number: 119 backlog-limit: 0 backlog-factor: 1.10 backlog-limit-highwater:0 dynamic-method: 3 dynamic-backlog-filter: 0.7 dynamic-backlog-low: 25.0 dynamic-backlog-high: 50.0 no-backlog: false ## ## Peers. ## peer decwrl { ip-name: news1.pa.dec.com } peer uunet { ip-name: news.uunet.uu.net max-connections: 10 } peer data.ramona.vix.com { # ip-name defaults to data.ramona.vix.com streaming: false } peer bb.home.vix.com { ip-name: 192.5.5.33 } # Blank lines are ignored. Everything after a '#' # is ignored too. # # Format is: # key : value # # See innfeed.conf(5) for a description of # necessary & useful keys. Unknown keys and their # values are ignored. # # Values may be a integer, floating-point, c-style # single-quoted characters, boolean, and strings. # # If a string value contains whitespace, or # embedded quotes, or the comment character # (``#''), then the whole string must be quoted # with double quotes. Inside the quotes, you may # use the standard c-escape sequence # (\\t,\\n,\\r,\\f,\\v,\\",\\'). # # Examples: # eg-string: "New\\tConfig\\tfile\\n" # eg-long-string: "A long string that goes # over multiple lines. The # newline is kept in the # string except when quoted # with a backslash \\ # as here." # eg-simple-string: A-no-quote-string # eg-integer: 10 # eg-boolean: true # eg-char: 'a' # eg-ctrl-g: '\\007' .fi .RE .SH HISTORY Written by James Brister for InterNetNews. .de R$ This is revision \\$3, dated \\$4. .. .R$ $Id: innfeed.conf.5 7778 2008-04-17 21:27:22Z iulius $ .SH SEE ALSO innfeed(1), newsfeeds(5)