+++ /dev/null
-.\" -*- nroff -*-
-.\"
-.\" Author: James A. Brister <brister@vix.com> -- berkeley-unix --
-.\" Start Date: Sat, 20 Jan 1996 15:50:56 +1100
-.\" Project: INN -- innfeed
-.\" File: innfeed.1
-.\" RCSId: $Id: innfeed.1 7798 2008-04-26 08:47:01Z iulius $
-.\" Description: Man page for innfeed(1)
-.\"
-.TH INNFEED 1
-.\"
-.\"
-.\"
-.\"
-.\"
-.SH NAME
-innfeed \- multi-host, multi-connection, streaming NNTP feeder.
-.SH SYNOPSIS
-.B innfeed
-[
-.B \-a spool-dir
-]
-[
-.BI \-b " directory"
-]
-[
-.B \-C
-]
-[
-.BI \-c " filename"
-]
-[
-.BI \-d " num"
-]
-[
-.BI \-e " bytes"
-]
-[
-.B \-h
-]
-[
-.BI \-l " filename"
-]
-[
-.B \-m
-]
-[
-.B \-M
-]
-[
-.B \-o bytes
-]
-[
-.B \-p file
-]
-[
-.B \-S file
-]
-[
-.B \-x
-]
-[
-.B \-y
-]
-[
-.B \-z
-]
-[
-.B \-v
-]
-[ file ]
-.\"
-.\"
-.\"
-.\"
-.\"
-.SH DESCRIPTION
-.PP
-.I Innfeed
-implements the NNTP protocol for transferring news between computers. It
-handles the standard IHAVE protocol as well as the CHECK/TAKETHIS
-streaming extension. Innfeed can feed any number of remote hosts at once
-and will open multiple connections to each host if configured to do so. The
-only limitations are the process limits for open file descriptors and memory.
-.PP
-As an alternative to using NNTP, INN may also be fed to an IMAP
-server. This is done by using an executable called imapfeed, which is
-identical to innfeed except for the delivery process. The new version
-has two types of connections: an LMTP connection to deliver regular
-messages and an IMAP connection to handle control messages. The
-startinnfeed process can then be told to start imapfeed instead of innfeed.
-(See the INSTALL file for how to do this.)
-.\"
-.\"
-.\"
-.\"
-.\"
-.SH MODES
-.PP
-.I Innfeed
-has three modes of operation: channel, funnel-file and batch.
-.PP
-Channel mode is used when no filename is given on the command line,
-the ``input-file'' keyword is \fInot\fP given in the config file, \fIand\fP
-the ``\fI\-x\fP'' option is \fInot\fP given.
-In channel mode innfeed runs with stdin connected via a pipe to
-innd. Whenever innd closes this pipe (and it has several reasons during
-normal processing to do so), innfeed will exit. It first will try to
-finish sending all articles it was in the middle of transmitting, before
-issuing a QUIT command. This means innfeed may take a while to exit
-depending on how slow your peers are. It never (well, almost never) just
-drops the connection.
-.PP
-The recommended way to restart innfeed when run in channel mode is
-therefore to tell innd to close the pipe and spawn a new innfeed process.
-This can be done with ``ctlinnd flush <feed>'' where <feed> is the name of
-the innfeed channel feed in ``\fInewsfeeds\fP''.
-.PP
-Funnel-file mode is used when a filename is given as an argument
-or the ``input-file'' keyword is given in the config file.
-In funnel file mode it reads the specified file for the same formatted
-information as innd would give in channel mode. It is expected that innd is
-continually writing to this file, so when innfeed reaches the end of the file
-it will check periodically for new information. To prevent the funnel file
-from growing without bounds, you will need to periodically move the file to
-the side (or simply remove it) and have innd flush the file. Then, after
-the file is flushed by innd, you can send innfeed a SIGALRM, and it too
-will close the file and open the new file created by innd. Something like:
-.PP
-.RS
-.nf
-innfeed -p /var/run/news/innfeed.pid my-funnel-file &
-while true; do
- sleep 43200
- rm -f my-funnel-file
- ctlinnd flush funnel-file-site
- kill -ALRM `cat /var/run/news/innfeed.pid`
-done
-.fi
-.RE
-.PP
-Batch mode is used when the ``\fI\-x\fP'' flag is used.
-In batch mode innfeed will ignore stdin, and will simply process any
-backlog created by a previously running innfeed. This mode is not normally
-needed as innfeed will take care of backlog processing.
-.\"
-.\"
-.\"
-.\"
-.\"
-.SH CONFIGURATION
-Innfeed expects a couple of things to be able to run correctly: a directory
-where it can store backlog files and a configuration file to describe which
-peers it should handle.
-.PP
-The configuration file is described in
-.IR innfeed.conf (5).
-The ``\fI\-c\fP''
-option can be used to specify a different file.
-.PP
-For each peer (say, ``\fIfoo\fP''), innfeed manages up to 4 files in the
-backlog directory: a ``\fIfoo.lock\fP'' file, which prevents other
-instances of innfeed from interfering with this one; a ``\fIfoo.input\fP''
-file which has old article information innfeed is reading for
-re-processing; a ``\fIfoo.output\fP'' file where innfeed is writing
-information on articles that couldn't be processed (normally due to a slow
-or blocked peer); and a ``\fIfoo\fP'' file.
-.PP
-This last file (``\fIfoo\fP'') is never created by innfeed, but if innfeed
-notices it, it will rename it to ``\fIfoo.input\fP'' at the next
-opportunity and will start reading from it. This lets you create a batch
-file and put it in a place where innfeed will find it. You should never
-alter the .input or .output files of a running innfeed.
-.PP
-The format of these last three files is one of the following:
-.PP
-.RS
-.nf
-/path/to/article <message-id>
-@token@ <message-id>
-.fi
-.RE
-.PP
-This is the same as the first two fields of the lines innd feeds to
-innfeed, and the same as the first two fields of the lines of the batch
-file innd will write if innfeed is unavailable for some reason. When
-innfeed processes its own batch files it ignores everything after the first
-two whitespace separated fields, so moving the innd-created batch file to
-the appropriate spot will work, even though the lines have extra fields.
-.PP
-The first field can also be a storage API token. The two types of lines can
-be intermingled; innfeed will use the storage manager if appropriate and
-otherwise treat the first field as a filename to read directly.
-.PP
-Innfeed writes its current status to the file ``\fIinnfeed.status\fP'' (or
-the file given by the ``\fI-S\fP'' option). This file contains details on
-the process as a whole, and on each peer this instance of innfeed is
-managing.
-.PP
-If innfeed is told to send an article to a host it is not managing, then
-the article information will be put into a file matching the pattern
-``\fIinnfeed-dropped.*\fP'', with part of the file name matching the pid of
-the innfeed process that is writing to it. Innfeed will not process this
-file except to write to it. If nothing is written to the file then it will
-be removed if innfeed exits normally.
-.\"
-.\"
-.\"
-.\"
-.\"
-.SH SIGNALS
-.PP
-Upon receipt of a SIGALRM innfeed will close the funnel-file specified on
-the command line, and will reopen it (see funnel file description above).
-.PP
-Innfeed with catch SIGINT and will write a large debugging snapshot of the
-state of the running system.
-.PP
-Innfeed will catch SIGHUP and will reload the config file.
-See
-.IR innfeed.conf (5)
-for more details.
-.PP
-Innfeed will catch SIGCHLD and will close and reopen all backlog files.
-.PP
-Innfeed will catch SIGTERM and will do an orderly shutdown.
-.PP
-Upon receipt of a SIGUSR1 innfeed will increment the debugging level by
-one; receipt of a SIGUSR2 will decrement it by one. The debugging level
-starts at zero (unless the ``-d'' option it used), in which case no debugging
-information is emitted. A larger value for the level means more debugging
-information. Numbers up to 5 are currently useful.
-.\"
-.\"
-.\"
-.\"
-.\"
-.SH SYSLOG ENTRIES
-.PP
-There are 3 different categories of syslog entries for statistics: Host,
-Connection and Global.
-.PP
-The Host statistics are generated for a given peer at regular intervals
-after the first connection is made (or, if the remote is unreachable, after
-spooling starts). The Host statistics give totals over all Connections that
-have been active during the given time frame. For example (broken here to
-fit the page, with ``vixie'' being the peer):
-.PP
-.nf
- May 23 12:49:08 data innfeed[16015]: vixie checkpoint
- seconds 1381 offered 2744 accepted 1286
- refused 1021 rejected 437 missing 0 spooled 990
- on_close 0 unspooled 240 deferred 10 requeued 25
- queue 42.1/100:14,35,13,4,24,10
-.fi
-.PP
-These meanings of these fields are:
-.nr XX \w'unspooled '
-.TP \n(XXu
-seconds
-The time since innfeed connected to the host or since the statistics
-were reset by a ``final'' log entry.
-.TP
-offered
-The number of IHAVE commands sent to the host if it is not in streaming mode.
-The sum of the number of TAKETHIS commands sent when no-CHECK mode
-is in effect plus the number CHECK commands sent in streaming mode (when
-no-CHECK mode is not in effect).
-.TP
-accepted
-The number of articles which were sent to the remote host and accepted
-by it.
-.TP
-refused
-The number of articles offered to the host that it it indicated it
-didn't want because it had already seen the Message-ID. The remote
-host indicates this by sending a 435 response to an IHAVE command or
-a 438 response to a CHECK command.
-.TP
-rejected
-The number of articles transferred to the host that it did not accept
-because it determined either that it already had the article or it did
-not want it because of the article's Newsgroups: or Distribution: headers,
-etc. The remote host indicates that it is rejecting the article by
-sending a 437 or 439 response after innfeed sent the entire article.
-.TP
-missing
-The number of articles which innfeed was told to offer to the host but
-which were not present in the article spool. These articles were probably
-cancelled or expired before innfeed was able to offer them to the host.
-.TP
-spooled
-The number of article entries that were written to the .output backlog file
-because the articles could not either be sent to the host or be refused
-by it. Articles are generally spooled either because new articles are
-arriving more quickly than they can be offered to the host, or because
-innfeed closed all the connections to the host and pushed all the
-articles currently in progress to the .output backlog file.
-.TP
-on_close
-The number of articles that were spooled when innfeed closed all the
-connections to the host.
-.TP
-unspooled
-The number of article entries that were read from the .input backlog
-file.
-.TP
-deferred
-The number of articles that the host told innfeed to retry later by
-sending a 431 or 436 response. Innfeed immediately puts these articles
-back on the tail of the queue.
-.TP
-requeued
-The number of articles that were in progress on connections when innfeed
-dropped those connections and put the articles back on the queue. These
-connections may have been broken by a network problem or became unresponsive
-causing innfeed to time them out.
-.TP
-queue
-The first number is the average (mean) queue size during the previous logging
-interval. The second number is the maximum allowable queue size.
-The third number is the percentage of the time that the queue
-was empty. The fourth through seventh numbers are the percentages of the
-time that the queue was >0% to 25% full, 25% to 50% full, 50% to 75%
-full, and 75% to <100% full. The last number is the percentage of the
-time that the queue was totally full.
-.PP
-If the ``\fI\-z\fP'' option is used (see below), then when the peer stats are
-generated, each Connection will log its stats too. For example, for
-connection number zero (from a set of five):
-.PP
-.nf
- May 23 12:49:08 data innfeed[16015]: vixie:0 checkpoint
- seconds 1381 offered 596 accepted 274
- refused 225 rejected 97
-.fi
-.PP
-If you only open a maximum of one Connection to a remote, then there will
-be a close correlation between Connection numbers and Host numbers, but in
-general you can't tie the two sets of number together in any easy or very
-meaningful way. When a Connection closes it will always log its stats.
-.PP
-If all Connections for a Host get closed together, then the Host logs its
-stats as ``final'' and resets its counters. If the feed is so busy that
-there's always at least one Connection open and running, then after some
-amount of time (set via the config file), the Host stats are logged as
-final and reset. This is to make generating higher level stats from log
-files, by other programs, easier.
-.PP
-There is one log entry that is emitted for a Host just after its last
-Connection closes and innfeed is preparing to exit. This entry contains
-counts over the entire life of the process. The ``seconds'' field is from the
-first time a Connection was successfully built, or the first time spooling
-started. If a Host has been completely idle, it will have no such log entry.
-.PP
-.nf
- May 23 12:49:08 data innfeed[16015]: decwrl global
- seconds 1381 offered 34 accepted 22
- refused 3 rejected 7 missing 0
-.fi
-.PP
-The final log entry is emitted immediately before exiting. It contains a
-summary of the statistics over the entire life of the process.
-.PP
-.nf
- Feb 13 14:43:41 data innfeed-0.9.4[22344]: ME global
- seconds 15742 offered 273441 accepted 45750
- refused 222008 rejected 3334 missing 217
-.fi
-.PP
-.\"
-.\"
-.\"
-.\"
-.\"
-.SH OPTIONS
-.TP
-.B \-a
-The ``\fI\-a\fP'' flag is used to specify the top of the article spool
-tree. Innfeed does a chdir(2) to this directory, so it should probably be
-an absolute path. The default is <patharticles\ in\ inn.conf>.
-.TP
-.B \-b
-The ``\fI\-b\fP'' flag may be used to specify a different directory for backlog
-file storage and retrieval. If the path is relative then it is relative
-to <pathspool\ in\ inn.conf>. The default is ``\fIinnfeed\fP''.
-.TP
-.B \-c
-The ``\fI\-c\fP'' flag may be used to specify a different config file from the
-default value. If the path is relative then it is relative to
-<pathetc\ in\ inn.conf>. The default is ``\fIinnfeed.conf\fP''.
-.TP
-.B \-C
-The ``\fI\-C\fP'' flag is used to have innfeed simply check the config
-file, report on any errors and then exit.
-.TP
-.B \-d
-The ``\fI\-d\fP'' flag may be used to specify the initial logging level. All
-debugging messages go to stderr (which may not be what you want, see the
-``\fI\-l\fP'' flag below).
-.TP
-.B \-e
-The ``\fI\-e\fP'' flag may be used to specify the size limit (in bytes) for the
-\fI\%.output\fP backlog files innfeed creates. If the output file gets bigger
-than 10% more than the given number, innfeed will replace the output file
-with the tail of the original version. The default value is 0, which means
-there is no limit.
-.TP
-.B \-h
-Use the ``\fI\-h\fP'' flag to print the usage message.
-.TP
-.B \-l
-The ``\fI\-l\fP'' flag may be used to specify a different log file from
-stderr. As innd starts innfeed with stderr attached to /dev/null, using this
-option can be useful in catching any abnormal error messages, or any
-debugging messages (all ``normal'' errors messages go to syslog).
-.TP
-.B \-M
-If innfeed has been built with mmap support, then the ``\fI\-M\fP'' flag
-turns OFF the use of mmap(); otherwise it has no effect.
-.TP
-.B \-m
-The ``\fI\-m\fP'' flag is used to turn on logging of all missing
-articles. Normally if an article is missing, innfeed keeps a count, but
-logs no further information. When this flag is used, details about
-message-id and expected pathname are logged.
-.TP
-.B \-o
-The ``\fI\-o\fP'' flag sets a value of the maximum number of bytes of article
-data innfeed is supposed to keep in memory. This doesn't work properly yet.
-.TP
-.B \-p
-The ``\fI\-p\fP'' flag is used to specify the filename to write the pid of the
-process into. A relative path is relative to <pathrun\ in\ inn.conf>. The
-default is ``\fIinnfeed.pid\fP''.
-.TP
-.B \-S
-The ``\fI\-S\fP'' flag specifies the name of the file to write the periodic
-staus to. If the path is relative it is considered relative to
-<pathlog\ in\ inn.conf>. The default is ``\fIinnfeed.status\fP''.
-.TP
-.B \-v
-When the ``\fI\-v\fP'' flag is given, version information is printed to stderr
-and then innfeed exits.
-.TP
-.B \-x
-The ``\fI\-x\fP'' flag is used to tell innfeed not to expect any article
-information from innd but just to process any backlog files that exist and
-then exit.
-.TP
-.B \-y
-The ``\fI\-y\fP'' flag is used to allow dynamic peer binding. If this flag is
-used and article information is received from innd that specifies an
-unknown peer, then the peer name is taken to be the IP name too, and an
-association with it is created. Using this it is possible to only
-have the global defaults in the
-.I innfeed.conf
-file, provided the peername as used by innd is the same as the ip name.
-Note that
-.I innfeed
-with ``\fI\-y\fP'' and no peer in
-.I innfeed.conf
-would cause a problem that
-.I innfeed
-drops the first article.
-.TP
-.B \-z
-The ``\fI\-z\fP'' flag is used to cause each connection, in a parallel feed
-configuration, to report statistics when the controller for the connections
-prints its statistics.
-.TP
-.\"
-.\"
-.\"
-.\"
-.\"
-.SH BUGS
-.PP
-When using the ``-x'' option, the config file entry's
-``initial-connections'' field will be the total number of connections
-created and used, no matter how many big the batch file, and no
-matter how big the ``max-connectiond'' field specifies. Thus a value
-of 0 for ``initial-connections'' means nothing will happen in ``-x''
-mode.
-.PP
-Innfeed does not automatically grab the file out of out.going--this needs
-to be prepared for it by external means.
-.PP
-Probably too many other bugs to count.
-.\"
-.\"
-.\"
-.\"
-.\"
-.SH FILES
-innfeed.conf config file.
-.br
-innfeed directory for backlog files.
-.\"
-.\"
-.\"
-.\"
-.\"
-.SH HISTORY
-Written by James Brister <brister@vix.com> for InterNetNews.
-.de R$
-This is revision \\$3, dated \\$4.
-..
-.R$ $Id: innfeed.1 7798 2008-04-26 08:47:01Z iulius $
-.SH SEE ALSO
-.IR innfeed.conf(5)
~ian / inn-innduct.git / blobdiff