3 pullnews - Pull news from multiple news servers and feed it to another
7 B<pullnews> [B<-hnqRx>] [B<-b> I<fraction>] [B<-c> I<config>] [B<-C> I<width>]
8 [B<-d> I<level>] [B<-f> I<fraction>] [B<-F> I<fakehop>] [B<-g> I<groups>]
9 [B<-G> I<newsgroups>] [B<-H> I<headers>] [B<-k> I<checkpt>] [B<-l> I<logfile>]
10 [B<-m> I<header_pats>] [B<-M> I<num>] [B<-N> I<timeout>] [B<-p> I<port>]
11 [B<-P> I<hop_limit>] [B<-Q> I<level>] [B<-r> I<file>] [B<-s> I<to-server>[:I<port>]]
12 [B<-S> I<max-run>] [B<-t> I<retries>] [B<-T> I<connect-pause>] [B<-w> I<num>]
13 [B<-z> I<article-pause>] [B<-Z> I<group-pause>] [I<from-server> ...]
17 The C<Net::NNTP> module must be installed. This module is available as part
18 of the libnet distribution and comes with recent versions of Perl. For
19 older versions of Perl, you can download it from L<http://www.cpan.org/>.
23 B<pullnews> reads a config file in the running user's home directory
24 (normally called F<~/.pullnews>) and connects to the upstream servers
25 given there as a reader client. By default, it connects to all servers
26 listed in the configuration file, but you can limit B<pullnews> to
27 specific servers by listing them on the command line: a whitespace-separated
28 list of server names can be specified, like I<from-server> for one of them.
29 For each server it connects to, it pulls over articles and feeds them to the
30 destination server via the IHAVE or POST commands. This means that the system
31 B<pullnews> is run on must have feeding access to the destination news server.
33 B<pullnews> is designed for very small sites that do not want to bother
34 setting up traditional peering and is not meant for handling large feeds.
40 =item B<-b> I<fraction>
42 Backtrack on server numbering reset. Specify the proportion (C<0.0> to C<1.0>)
43 of a group's articles to pull when the server's article number is less than
44 our high for that group. When I<fraction> is C<1.0>, pull all the articles on
45 a renumbered server. The default is to do nothing.
49 Normally, the config file is stored in F<~/.pullnews> for the user running
50 B<pullnews>. If B<-c> is given, I<config> will be used as the config file
51 instead. This is useful if you're running B<pullnews> as a system user on
52 an automated basis out of cron rather than as an individual user.
54 See L<CONFIG FILE> below for the format of this file.
58 Use I<width> characters per line for the progress table. The default value
63 Set the debugging level to the integer I<level>; more debugging output
64 will be logged as this increases. The default value is C<0>.
66 =item B<-f> I<fraction>
68 This changes the proportion of articles to get from each group to
69 I<fraction> and should be in the range C<0.0> to C<1.0> (C<1.0> being
72 =item B<-F> I<fakehop>
74 Prepend I<fakehop> as a host to the Path: header of articles fed.
78 Specify a collection of groups to get. I<groups> is a list of
79 newsgroups separated by commas (only commas, no spaces). Each group must
80 be defined in the config file, and only the remote hosts that carry those
81 groups will be contacted. Note that this is a simple list of groups, not
82 a wildmat expression, and wildcards are not supported.
84 =item B<-G> I<newsgroups>
86 Add the comma-separated list of groups I<newsgroups> to each server in the
87 configuration file (see also B<-g> and B<-w>).
91 Print a usage message and exit.
93 =item B<-H> I<headers>
95 Remove these named headers (colon-separated list) from fed articles.
97 =item B<-k> I<checkpt>
99 Checkpoint (save) the config file every I<checkpt> articles
100 (default is C<0>, that is to say at the end of the session).
102 =item B<-l> I<logfile>
104 Log progress/stats to I<logfile> (default is C<stdout>).
106 =item B<-m> I<header_pats>
108 Feed an article based on header matching. The argument is a number of
109 whitespace-separated tuples (each tuple being a colon-separated header and
110 regular expression). For instance:
112 Hdr1:regexp1 !Hdr2:regexp2
114 specifies that the article will be passed only if the C<Hdr1:> header
115 matches C<regexp1> and the C<Hdr2:> header does not match C<regexp2>.
119 Specify the maximum number of articles (per group) to process.
120 The default is to process all new articles. See also B<-f>.
124 Do nothing but read articles S<-- does> not feed articles downstream,
125 writes no B<rnews> file, does not update the config file.
127 =item B<-N> I<timeout>
129 Specify the timeout length, as I<timeout> seconds,
130 when establishing an NNTP connection.
134 Connect to the destination news server on a port other than the default of
135 C<119>. This option does not change the port used to connect to the source
138 =item B<-P> I<hop_limit>
140 Restrict feeding an article based on the number of hops it has already made.
141 Count the hops in the Path: header (I<hop_count>), feeding the article only
142 when I<hop_limit> is C<+num> and I<hop_count> is more than I<num>;
143 or I<hop_limit> is C<-num> and I<hop_count> is less than I<num>.
147 Print out less status information while running.
151 Set the quietness level (C<-Q 2> is equivalent to C<-q>). The higher this
152 value, the less gets logged. The default is C<0>.
156 Rather than feeding the downloaded articles to a destination server, instead
157 create a batch file that can later be fed to a server using B<rnews>. See
158 rnews(1) for more information about the batch file format.
162 Be a reader (use MODE READER and POST commands) to the downstream
163 server. The default is to use the IHAVE command.
165 =item B<-s> I<to-server>[:I<port>]
167 Normally, B<pullnews> will feed the articles it retrieves to the news
168 server running on localhost. To connect to a different host, specify a
169 server with the B<-s> flag. You can also specify the port with this same
172 =item B<-S> I<max-run>
174 Specify the maximum time I<max-run> in seconds for B<pullnews> to run.
176 =item B<-t> I<retries>
178 The maximum number (I<retries>) of attempts to connect to a server
179 (see also B<-T>). The default is C<0>.
181 =item B<-T> I<connect-pause>
183 Pause I<connect-pause> seconds between connection retries (see also B<-t>).
188 Set each group's high watermark (last received article number) to I<num>.
189 If I<num> is negative, calculate S<I<Current>+I<num>> instead (i.e. get the last
190 I<num> articles). Therefore, a I<num> of C<0> will re-get all articles on the
191 server; whereas a I<num> of C<-0> will get no old articles, setting the
192 watermark to I<Current> (the most recent article on the server).
196 If the B<-x> flag is used, an Xref: header is added to any article
197 that lacks one. It can be useful for instance if articles are fed
198 to a news server which has I<xrefslave> set in F<inn.conf>.
200 =item B<-z> I<article-pause>
202 Sleep I<article-pause> seconds between articles. The default is C<0>.
204 =item B<-Z> I<group-pause>
206 Sleep I<group-pause> seconds between groups. The default is C<0>.
212 The config file for B<pullnews> is divided into blocks, one block for each
213 remote server to connect to. A block begins with the host line, which
214 must have no leading whitespace and contains just the hostname of the
215 remote server, optionally followed by authentication details (username
216 and password for that server).
218 Following the host line should be one or more newsgroup lines which start
219 with whitespace followed by the name of a newsgroup to retrieve. Only one
220 newsgroup should be listed on each line.
222 B<pullnews> will update the config file to include the time the group was
223 last checked and the highest numbered article successfully retrieved and
224 transferred to the destination server. It uses this data to avoid doing
225 duplicate work the next time it runs.
229 <host> [<username> <password>]
230 <group> [<time> <high>]
231 <group> [<time> <high>]
233 where the <host> line must not have leading whitespace and the <group>
236 A typical configuration file would be:
238 # Format group date high
240 rec.bicycles.racing 908086612 783
241 rec.humor.funny 908086613 18
242 comp.programming.threads
243 nnrp.vix.com pull sekret
246 Note that an earlier run of B<pullnews> has filled in details about the
247 last article downloads from the two rec.* groups. The two comp.* groups
248 were just added by the user and have not yet been checked.
250 The nnrp.vix.com server requires authentication, and B<pullnews> will use
251 the username C<pull> and the password C<sekret>.
257 =item I<pathbin>/pullnews
259 The Perl script itself used to pull news from upstream servers and feed
260 it to another news server.
262 =item I<$HOME>/.pullnews
264 The default config file. It is in the running user's home directory
265 (normally called F<~/.pullnews>).
271 B<pullnews> was written by James Brister for INN. The documentation was
272 rewritten in POD by Russ Allbery <rra@stanford.edu>.
274 Geraint A. Edwards greatly improved B<pullnews>, adding no more than S<16 new>
275 recognized flags, fixing some bugs and integrating the B<backupfeed>
276 contrib script by Kai Henningsen, adding again S<6 other> flags.
278 $Id: pullnews.pod 7853 2008-05-27 19:07:45Z iulius $
282 incoming.conf(5), rnews(1).