NEWS SETUP ON DRAGON CLUSTER -- Peter Maydell , 08/1998 I have mnementh set up so I can read news offline, to save on phone bills. This document describes the configuration and software required to do this. NB: I wrote this a while after actually configuring everything, so I might have omitted something. If something doesn't work or is obviously wrong, let me know and I'll correct this document. HUGE WARNING: this is currently *wrong* for chiark users, as it doesn't do posting to ucam.* correctly! I have a system (more or less) which does all the Right Things and doesn't break any of the chiark rules, but I haven't had time to write it up yet; email me for details. -- PMM 12/1999 BIG WARNING: you may well need to check with your news server administrators before setting this up! Although most ISPs probably don't care (to them it looks like just another newsreader), chiark users in particular should carefully read the chiark rules about this and will need to inform Ian Jackson about what they are doing... I believe that this config is compatible with the chiark bulk news transfer rules, but you should obviously verify this for yourself :-> SMALL WARNING: this document is steadily drifting away from general relevance to non-chiark users, as chiark's rules for bulk news transfer are rather different to those of random ISPs. Firstly, you will need to run a local newsserver. I recommend INN, because I tried C-News and couldn't work out how to configure it. INN is apparently a bit bigger and more memory-hungry than C-News, but C-News is dying out and memory is cheap nowadays... The Debian INN package works fine; I currently have version 1.7.2-4 installed. You will probably find when you try to install the inn package that it wants to replace inews with inewsinn. If you do things in the right order you can avoid dpkg/apt trying to remove all your installed newsreader packages... The other piece of software you will need is newsx (or something that does the same job). newsx takes outgoing news spooled by INN and sends it to a remote server using POST (so it acts like a normal newsreader client as far as the server is concerned). It also fetches incoming news from that server and passes it to INN. It can submit news using IHAVE, which is a requirement for chiark. You need newsx 1.2 or better, to provide the AUTHINFO GENERIC support. You also currently need a patch to do MODE READER before authenticating even in the IHAVE case. [Submitted as a Debian bug; will presumably be fixed upstream too eventually.] The patch is at the end of this file. You'll also need the md5cookie1way program. The source for this is on chiark in /usr/local/src/nntp-merge/ (possibly /usr/src/nntp-merge). Compile this ("make md5cookie1way" will work) and stick the binary in /usr/local/bin/. (It does need to be on the PATH set in /etc/ppp/ip-up). Create the file /etc/news/md5cookies.read containing lines such as: chiark e8:50:9b:d5:47:9a:29:cd # Do not use the usenet2 cookie without reading /info/news-usenet2.text. chiark-usenet2 eb:2c:95:3e:9e:ad:28:f3 You can find the true cookie values in the file of that name on chiark if you have permission to use them. The values above are not the true cookies. [Actually, they used to be, because I was dumb enough to include the file verbatim in this web document; they have since been changed.] This is one of the files searched by md5cookie1way when looking for a shared secret. If you have a personal cookie you could put it in /etc/news/md5cookies and make that owned by news.news and non-world-readable. NB: the nature of this setup is that every local reader on your machine will have to use the same chiark cookie. I'm the only user on my machine (and I don't have a personal cookie anyway) so this doesn't matter to me. Now you need to configure newsx and INN. The INN documentation is rather large and not very comprehensible, IMHO. The newsx stuff is easier to read but there's not quite enough of it... The easiest thing to do is to go through the 'Quick Start with INN' section of /usr/doc/newsx/README.Debian.gz and the 'INN Quickstart for PPP', comparing with my config files. Have a look at the /etc/ppp/ip-up.d/newsx-pmm script later in this file for the right set of command line options and ssh invocation but don't run it automatically yet... Here's my /etc/news/newsfeeds: ---begin--- # The ME line is magic. The 'subscription list' (everything between first : and /) # is prepended to the entries for all other feeds in this file. # (those entries can override defaults set here: last match in the line # always wins.) # A distribution subfield means that only articles matching that distribution # are *accepted* (ie filters on incoming). # General note: we *never* accept articles which are not posted to groups # mentioned in the active file. This is enforced by both inews and newsx. # newsx also reads this file to see which groups to fetch from each feed. # this is the default: send everything except control or junk or local.*. # (default also has /!local to not accept local articles from other (misconfigured) # sites. We don't want that since we want local-to-chiark posts to be accepted.) ME:*,!control,!junk,!local.*:: # Magic for doing overview database (threading?) overview:*:Tc,WO:/usr/lib/news/bin/overchan # This is the recommended newsx outgoing batch incantation: # it causes outgoing news to be put in a spool file for # newsx to handle later. # In general, the 'excludes' list for each outgoing feed contains the # path entries for each incoming feed. This ensures that we never attempt # to feed articles from demon to chiark or vice versa; only locally # generated articles are ever emitted. # We exchange only chiark.*, ucam.*, oxbridge.* and net.* with chiark. # USENET 2 (net.*) is a separate feed to the same machine, so that # we can avoid sending articles crossposted outside net.*. # This means we aren't an incoming leak (although our spool will # still have such crossposted articles if we get them from demon, # we won't offer them to chiark.) # U2 feed: accepts only articles posted to net.* and no groups outside net.* # and only articles with Distribution: 4gh chiark-u2/ewrotcd,news.demon.co.uk:@*,net.*/4gh:Tf,Wfb: # Feed for chiark-local, non-U2 groups. Articles posted to net.* and # a non-net.* group aren't sent because of the '@net.*' line. chiark/ewrotcd,news.demon.co.uk:!*,chiark.*,ucam.*,oxbridge.*,@net.*,!junk*:Tf,Wfb: # everything else comes from Demon; make sure we don't leak # chiark/ucam/usenet 2... The @ is like ! except that it # means 'don't propagate even if crossposted to a group that # would be propagated'. So crossposts between eg misc.test and # ucam.chat aren't propagated. They would go via chiark. # The use of Tm means that INN will 'funnel' outgoing articles # into the same outbound file as the chiark feed. # This is done because we send all outbound articles via chiark but # still wish to pull inbound news from demon. demon/news.demon.co.uk,ewrotcd:*,@chiark.*,@ucam.*,@net.*,@oxbridge.*,@junk*:Tm:chiark # This is the old config; it would produce a file to be sent to demon by newsx. #demon/news.demon.co.uk,ewrotcd:*,@chiark.*,@ucam.*,@net.*,@oxbridge.*,@junk*:Tf,Wfb: ---end--- See newsfeeds(5) for more details on the syntax. Broadly speaking, you will want to edit the bit between the first and second ':'s on the demon/chiark lines to set which groups are sent to/taken from which servers. I recommend that if you have a non-chiark server you should use it to pull everything except chiark-only newsgroups: it will be faster and put less load on chiark. The two lines above are complementary: there's no point getting a group both from demon and from chiark, and generally you shouldn't be leaking chiark-only groups to demon (it probably won't accept them anyway.) WARNING: one of the chiark-specific rules is that you can't suck news from chiark and also post articles to another server using POST. Hence the setup above where demon's server is only used to pull incoming articles but all outgoing news is sent via chiark and IHAVE. Note that newsfeeds is being read by *two* programs. innd reads it to find out where it should send outgoing news to. It batches up articles that match the pattern into a named spoolfile, which newsx will read later. newsx reads newsfeeds to find out which groups to request from the server in the fetching phase: it only asks for groups which are in the active file and match the newsfeeds pattern. Start small, with just a few groups (eg chiark.test and demon.test), and don't try to automate newsx fetching until you're sure it's all working. The first time you fetch a new newsgroup it will take rather longer because it's fetching all the articles on the remote server. (Also, chiark has a big gap in its article numbering with a few articles from 1997 still floating around, so newsx requests lots of non-existent article numbers. [may be fixed in a later release of newsx?]) Get a connection to the Demon (or whatever) server working before trying to get chiark working, if you can: it's somewhat simpler. You'll probably want to tweak /etc/news/expire.ctl, which sets how long you keep news on the spool file. I recommend: /remember/:14 *:A:1:14:60 control:A:1:7:60 junk:A:1:7:60 of which the important figure is the 14. Expiry times set here are from the time the article first arrives on your local spool, rather than date of posting. See expire.ctl(5). You may also wish to add "-c0" to the FLAGS definition in the /etc/news/boot script. This means that INN never rejects an article from a feed because it's too old (based on time of posting). This is a useful option because the most likely reason for receiving old articles is simply that we haven't downloaded news recently (eg been on holiday) or that this is a group we've only just decided to download. We don't want to lose the backlog of unread articles in this situation. When you're ready to automate newsx fetching, you should create a script /etc/ppp/ip-up.d/newsx-pmm similar to this one. The script is commented to say what it's doing. Note the funky ssh stuff to talk to chiark. Make sure that root can ssh to your chiark account without getting a password prompt before you try to run the script... Notice also that the chiark news cookie to use is embedded in this script ('chiark-usenet2'). If you don't read USENET-2 then you should use 'chiark' instead. Don't use the U2 cookie without having read the USENET-2 rules (http://www.usenet2.org/, I think) and the /info/ file on chiark about USENET-2 access. [I'm supposed to ensure that people don't just copy my configuration files without having read the rules, so don't do that, please :->] Also, it's probably a good idea to set up an alias so that mail to usenet2@yourmachine goes to somewhere sensible. ---begin script--- #!/bin/sh # # Don't run for local-to-dragon-cluster connections if [ x$PPP_IPPARAM = xdragon ]; then exit 0; fi # # Run newsx to send and receive news when the PPP link goes up. # Note that this script is *not* named newsx in case the Debian # package ever includes an ip-up.d script... # # We have to use logger to get messages into the syslog, sigh. # # Ensure that inn puts all outgoing messages into the # spool files ready for newsx to read them. su news -c "/usr/sbin/ctlinnd flush ''" # Most news is sucked from Demon. We tell newsx not to try to post # news; there shouldn't be any anyway as INN will have routed it all # into the same file used for the chiark feed's outbound news. su news -c "/usr/lib/news/bin/newsx --nopost --inn -dd --window 10 --rnews --keeppath demon news.demon.co.uk" 2>&1 | logger -p news.info -t newsx # chiark is for ucam.*, chiark.*, net.* only. # net.* is a separate feed ('chiark-u2') to ensure that we are sound and # don't send crossposted articles. # It's also a bit hairy to connect to, since it # (a) requires MD5 authentication via AUTHINFO GENERIC # (b) requires us to connect using ssh to forward local port 9090 # to chiark's NNTP port... # Whip together a fake .newsrc to persuade chiark that we're still reading # news on there (so it doesn't stop fetching groups we read...) # All stuff goes out via chiark, so everything goes in the newsrc except # control and junk... ~/.newsrc' # First we initiate the port forwarding via ssh: # note -C to indicate compression of the data (probably a good idea # for usenet over a modem...) # We have to specify a command: "sleep 30" gives us 30 seconds to # open the local port. Once a connection is using the forwarded port # ssh will not exit until it is closed, which is what we want. # NB: root had better be set up to ssh to the target user/machine # without a password being requested :-> ssh -f -C -L 9090:localhost:119 pmaydell@login.chiark.greenend.org.uk 'sleep 30' # Now we run newsx, using the options to do AUTHINFO GENERIC # and MODE READER before AUTHINFO, telling it that the news server # is on localhost port 9090... # We use the non-U2 cookie to ensure that we can't post net.* articles via # this feed: if there are any then newsfeeds is probably screwed... # Note that chiark rules mandate use of IHAVE. su news -c "export NNTPAUTH='md5cookie1way chiark'; /usr/lib/news/bin/newsx --inn -dd --window 10 --rnews --authgeneric --reader --readbeforeauth --keeppath --ihave chiark localhost 9090" 2>&1 | logger -p news.info -t newsx # Now the usenet-2 feed: note that we use local port 9091 in case # the previous ssh hasn't released 9090 yet... ssh -f -C -L 9091:localhost:119 pmaydell@login.chiark.greenend.org.uk 'sleep 30' su news -c "export NNTPAUTH='md5cookie1way chiark-usenet2'; /usr/lib/news/bin/newsx --inn -dd --window 10 --rnews --authgeneric --reader --readbeforeauth --keeppath --ihave chiark-u2 localhost 9091" 2>&1 | logger -p news.info -t newsx # Let user know news transfers are done logger -p news.notice -t newsx 'All newsx transfers completed.' ---end script--- inews: I had problems with inews producing invalid message-IDs. This will only be a problem if hostname --fqdn doesn't produce a valid fully qualified domain name (my machine gives mnementh.local for obscure reasons). You can test by running "inews -h -D" and entering a test article on standard input such as: Newsgroups: chiark.test Subject: foo body and ending with ctrl-D. The test article won't go anywhere, but inews will print on standard out what it would have submitted to the news server. If the message-ID created is bogus, you'll need to hack the inews source to get it to put something better in. (It's in the debian inn source package. Look at lib/getfqdn.c.) You'll probably want to get hold of a suitable file for /var/lib/news/newsgroups giving all the descriptive names for groups. It's OK for this to have groups the server doesn't carry, so just snarf chiark's version. I can't think of any other configuration stuff I've missed, so I'll stop here :-> Postscript: here's the patch against Debian's newsx package 1.2-2, for file src/doit.c. This fixes a bug whereby the --readbeforeauth flag didn't do the right thing in conjunction with --ihave. ===begin=== --- doit.c.dist Sat Oct 23 23:13:34 1999 +++ doit.c Sat Oct 23 23:20:02 1999 @@ -116,19 +116,22 @@ time(&starttime); /* We might need to do MODE READER before AUTHINFO or afterwards. - * readbeforeauth_opt switches which way round we do things. -- PMM + * readbeforeauth_opt switches which way round we do things. + * The fact that we don't do MODE READER until after IHAVE + * transfers complicates things a little... -- PMM */ /* if authinfo details supplied, then use 'em */ if (!readbeforeauth_opt && ai_username) - do_authinfo(ai_username,ai_password); + do_authinfo(ai_username,ai_password); /* switch INN to nnrpd instead of innd if needed */ - if (!ihave_opt && mode_reader_opt > 0) - do_mode_reader(); - - if (readbeforeauth_opt && ai_username) - do_authinfo(ai_username,ai_password); + if (!ihave_opt && mode_reader_opt > 0) { + do_mode_reader(); + /* If we haven't already authenticated, do that now */ + if (readbeforeauth_opt && ai_username) + do_authinfo(ai_username,ai_password); + } /* get the active newgroup list from the server */ if (group_list || group_newlist) { @@ -150,8 +153,14 @@ despool(article_name); /* might need to do the switch here */ - if (ihave_opt && mode_reader_opt >= 0) + if (ihave_opt && mode_reader_opt >= 0) { do_mode_reader(); + /* If we postponed authentication until after MODE READER, + * better do it now. + */ + if (readbeforeauth_opt && ai_username) + do_authinfo(ai_username,ai_password); + } /* and pull any new articles */ if (!nopull_opt) { ===endit===