fix so successful only connect really not blocker for another connect; lowvol in...

[innduct.git] / TODO

This is a rough and informal list of suggested improvements to INN, parts
of INN that need work, and other tasks yet undone.  Some of these may be
in progress, in which case the person working on them will be noted in
square brackets and should be contacted if you want to help.  Otherwise,
let inn-workers@isc.org know if you'd like to work on any item listed
below.

The list is divided into changes already tentatively scheduled for a
particular release, higher priority changes that will hopefully be done in
the near future, small or medium-scale projects for the future, and
long-term, large-scale problems.  Note that just because a particular
feature is scheduled for a later release doesn't mean it can't be
completed earlier if someone decides to take it on.  The association of
features with releases is intended to be a rough guide for prioritization
and a set of milestones to use to judge when a new major release is
justified.

Also, one major thing that is *always* welcome is additions to the test
suite, which is currently very minimal.  Any work done on the test suite
to allow more portions of INN to be automatically tested will make all
changes easier and will be *greatly* appreciated.

Last modified $Id: TODO 7575 2006-09-11 22:59:38Z eagle $.


Scheduled for INN 2.5

* Rewrite configure, breaking all of the tests out into separate files
  using the new capabilities in autoconf 2.5x.  Replace our local macros
  with the more general features provided by autoconf.  At the same time,
  configure.in and Makefile.global.in should be fixed to use the same
  names as each other for various parameters.  [Russ plans to work on
  this.]

* Add support for groups, nesting, and vectors to the new configuration
  parsing code.  [Russ plans on doing this.]

* Convert readers.conf and storage.conf (and related configuration files)
  to use the new parsing system and break out program-specific sections
  of inn.conf into their own groups.

* The current WIP cache and history cache should be integrated into the
  history API, things like message ID hashing should become a selectable
  property of the history file, and the history API should support
  multiple backend storage formats and automatically select the right one
  for an existing history file based on stored metainformation.

* The interface to embedded filters needs to be reworked.  The information
  about which filters are enabled should be isolated in the filtering API,
  and there should be standard API calls for filtering message IDs, remote
  posts, and local posts.  As part of this revision, all of the Perl
  callbacks should be defined before any of the user code is loaded, and
  the Perl loading code needs considerable cleanup.  At the same time as
  this is done, the implementation should really be documented; we do some
  interesting things with embedded filters and it would be nice to have a
  general document describing how we do it.  [Russ is planning on working
  on this at some point, but won't get upset if someone starts first.]

* All of INN's documentation should be written in POD, with text and man
  pages generated from the POD source.  Anyone is encouraged to work on
  this by just taking any existing documentation in man format and convert
  it to POD while checking that it's still accurate and adding any
  additional useful information that was missed.

* Replace the current innshellvars.pl file with a real INN Perl module for
  Perl programs, and include the necessary glue so that other Perl modules
  can be added to INN's build tree and installed with INN, allowing their
  capabilities to be available to the portions of INN written in Perl.

* Switch nnrpd over to using the new wildmat routines rather than breaking
  apart strings on commas and matching each expression separately.  This
  involves a lot of surgery, since PERMmatch is used all over the place,
  and may change the interpretation of ! and @ in group permission
  wildmats.

* Rework and clean up the storage API.  The major change is that the
  initialization function should return a pointer to an opaque struct
  which stores all of the state of the storage subsystem, rather than all
  of that being stored in static variables, and then all other functions
  should take that pointer.  More of the structures should also be opaque,
  all-caps structure names should be avoided in favor of named structures,
  SMsetup and SMinit should be combined into one function that takes
  flags, SMerrno and SMerrorstr should be replaced with functions that
  return that information, and the wire format utilities should be moved
  into libinn.

* Rework and clean up the overview API.  The major change is that the
  initialization function should return a pointer to an opaque struct
  which stores all of the state of the overview subsystem, rather than all
  of that being stored in static variables, and then all other functions
  should take that pointer.  OVctl possibly should instead take and return
  a struct rather than using an ioctl-style interface.  Currently, the
  overview functions do a lot of breaking apart of Xref headers and
  parsing them, which is very ugly; consider having the overview interface
  always key off a newsgroup name and article number, even for storing.
  OVadd should probably take a structure and OVsearch should probably
  return a structure.


Scheduled for INN 2.6

* Add a generic, modular anti-spam and anti-abuse filter, off by default,
  but coming with INN and prominently mentioned in the INSTALL
  documentation.  [Andrew Gierth has work in progress that may be usable
  for this.]

* A unified configuration file combining the facilities of newsfeeds,
  incoming.conf, and innfeed.conf, but hopefully more readable and easier
  for new INN users to edit.  This should have all of the capabilities of
  the existing configuration files, but specifying common things (such as
  file feeds or innfeed feeds) should be very simple and straightforward.
  This configuration file should use the new parsing infrastructure.

* Convert all remaining INN configuration files to the new parsing
  infrastructure.

* INN really should be capable of both sending and receiving a
  headers-only feed (or even an overview-only feed) similar to Diablo and
  using it for the same things that Diablo does, namely clustering,
  pull-on-demand for articles, and the like.  This should be implementable
  as a new backend, although the API may need a few more hooks.  Both a
  straight headers-only feed that only pulls articles down via NNTP from a
  remote server and a caching feed where some articles are pre-fed, some
  articles are pulled down at first read, and some articles are never
  stored locally should be possible.  [Patches for a header-only feed have
  already been written and submitted to inn-workers.]

* The libinn, libstorage, and other library interfaces should be treated
  as stable libraries and properly versioned using libtool's
  recommendation for library versioning when changes are made so that they
  can be installed as shared libraries and work properly through releases
  of INN.  This is currently waiting on a systematic review of the
  interface and removal of things that we don't want to support long-term.

* The include files necessary to use libinn, libstorage, and other
  libraries should be installed in a suitable directory so that other
  programs can link against them.  All such include files should be under
  include/inn and included with <inn/header.h>.  All such include files
  should only depend on other inn/* header files and not on, e.g.,
  config.h.  All such include files should be careful about namespace to
  avoid conflicts with other include files used by applications.


High Priority Projects

* Modulo warnings from system headers and warnings where the compiler is
  simply wrong and there's no equally readable way to rewrite the code,
  INN should compile cleanly under "make warnings".  It should be possible
  for maintainers to routinely compile INN with make warnings to catch
  problems.  Note that -Wcast-qual warnings cannot be avoided entirely
  because we don't want to write redundant functions for regular and const
  strings and because of such things as struct iovec; -Wcast-qual will be
  removed from make warnings when this task is reasonably complete.

* INN shouldn't flush all feeds (particularly all program feeds) on
  newgroup or rmgroup.  Currently it reloads newsfeeds to reparse all of
  the wildmat patterns and rebuild the peer lists associated with the
  active file on group changes, and this forces a flush of all feeds.
  The best fix is probably to stash the wildmat pattern (and flags) for
  each peer when newsfeeds is read and then just using the stashed copy on
  newgroup or rmgroup, since otherwise the newsfeeds loading code would
  need significant modification.  But in general, innd is too
  reload-happy; it should be better at making incremental changes without
  reloading everything.

* Add authenticated Path support, based on the current USEFOR draft or the
  behavior of some other servers (such as Diablo).  [Andrew Gierth wrote a
  patch for part of this a while back, which Russ has.  Marco d'Itri
  expressed some interest in working on this.]

* Various parts of INN are using write or writev; they should all use
  xwrite or xwritev instead.  Even for writes that are unlikely to ever be
  partial, on some systems system calls aren't restartable and xwrite and
  xwritev properly handle EINTR returns.

* Apparently on Solaris open can also be interrupted by a signal; we may
  need to have an xopen wrapper that checks for EINTR and retries.

* tradspool has a few annoying problems.  Deleted newsgroups never have
  their last articles expired, and there is no way of forcibly
  resynchronizing the articles stored on disk with what overview knows
  about unless tradindexed is used.  Some sort of utility program to take
  care of these and to do things like analyze the tradspool.map file
  should be provided.

* Rewrite inndstart as a helper program that only binds the relevant
  sockets and then returns them to innd.  Since file descriptors are
  shared by child processes, this can be done with a program spawned by
  innd.  This may have gotten more complicated with IPv6.  Drop
  startinnfeed entirely in favor of recommending people use ulimit in the
  news init script.

* contrib/mkbuf and contrib/reset-cnfs.c should be combined into a utility
  for creating and clearing cycbuffs, perhaps combined with cnfsheadconf,
  and the whole thing moved into storage/cnfs rather than frontends (along
  with cnfsstat).  pullart.c may also stand to be merged into the same
  utility (cnfs-util might not be a bad name).


Documentation Projects

* Add man pages for all libinn interfaces.  There should be a subdirectory
  of doc/pod for this since there will be a lot of them; installing them
  as libinn_<section>.3 seems to make the most sense (so, for example,
  error handling routines would be documented in libinn_error.3).

* Better documentation of and support for UUCP feeds.  send-uucp is now
  easier to use, but there's still a paucity of documentation covering the
  whole theory and mechanisms of UUCP feeding.

* Everything installed by INN should have a man page.  Currently, there
  are several binaries and configuration files that don't have man pages.
  (In some cases, the best thing to do with the configuration file may be
  to merge it into another one or find a way to eliminate it.)

* Document the internal formats of the various overview methods, CNFS,
  timehash, and timecaf.  A lot of this documentation already exists in
  various forms, but it needs to be cleaned up and collected in one place
  for each format, preferrably as a man page.

* Add documentation for slave servers.  [Russ has articles from
  inn-workers that can be used as a beginning.]

* Write complete documentation for all of our extensions to RFC 977 or RFC
  1036, preferrably in a format that could be suitable for future
  inclusion into new revisions of the RFCs.

* Audit readers.conf.5 against perm.c for missing options ("include" at
  least is missing from the documentation).

* The distributions file is undocumented.


Code Cleanup Projects

* Eliminate everything in the LEGACY section of config.h.

* Move all compile-time configuration in config.h either into a separate
  header (such as inn/options.h) or turn it into a configuration file
  directive or a command-line option.  In particular, the rnews
  configuration should probably be an rnews-specific section of inn.conf.

* Move include/paths.h to include/inn/paths.h and change _PATH as a prefix
  to INN_PATH to move the identifiers out of the C reserved namespace.
  Check to be sure we still need all of the #defines and look at adding
  anything needed by innfeed (and eliminating the separate innfeed header
  serving the same purpose).

* Move include/nntp.h to include/inn/nntp.h and at the same time look at
  standardizing the names of all of the #defines it provides, including
  the message class.  [Russ has a start on this.]

* Get rid of GetTimeInfo and TIMEINFO.  All the struct is is a struct
  timeval plus time zone information.  All of the parts of INN that deal
  with time zone information are isolated in lib/date.c.  The rest of INN
  uses GetTimeInfo where a plain call to time would often work fine, or
  at most gettimeofday, and there's no reason to compute the time zone
  everywhere.  Plus, it makes the code more readable to use standard
  functions and data types.

* putman.sh should be merged into support/install-sh (which would mean
  giving up any pretext of using the standard install-sh script, but that
  should be fine).

* Use vectors or cvectors everywhere that argify and friends are currently
  used and eliminate the separate implementation in nnrpd/misc.c.

* Break up the remainder of libinn.h into multiple inn/* include files for
  specific functions (such as memory management, wildmat, date handling,
  NNTP commands, etc.), with an inn/util.h header to collect the remaining
  random utilities.  Consider adding some sort of prefix, like inn_, to all
  functions that aren't part of some other logical set with its own prefix.

* Break the CNFS and tradspool code into multiple source files to make it
  easier to understand the logical divisions of the code and consider
  doing the same with the other overview and storage methods.

* Examine the (mostly socket) code that currently should probably be
  compiled with -fno-strict-aliasing on gcc and move the relevant casts
  to within function calls.  [Russ knows about this.]

* Clean up the use of #ifdef for sockets and IPv6, perhaps involving
  addition of more to include/portable/socket.h.


Needed Bug Fixes

* tradspool currently uses stdio to write out tradspool.map, which can
  cause problems if more than 256 file descriptors are in use for other
  things (such as incoming connections or tradindexed overview cache).
  It should use write() instead.

* LIST NEWSGROUPS should probably only list newsgroups that are marked in
  the active file as valid groups.

* INN's startup script should be sure to clean out old lock files and PID
  files for innfeed.  Be careful, though, since innfeed may still be
  running, spawned from a previous innd.

* makedbz should be more robust in the presence of malformed history
  lines, discarding with them or otherwise dealing with them.

* CNFS, if the cycbuff is larger than 2GB and it doesn't have large file
  support, reports a mysterious file not found error because it assumes
  all errors from stat are the result of the cycbuff not being found.

* Some servers reject some IHAVE, TAKETHIS, or CHECK commands with 500
  syntax errors (particularly for long message IDs), and innfeed doesn't
  handle this particularly well at the moment.  It really should have an
  error handler for this case.  [Sven Paulus has a preliminary patch that
  needs testing.]

* Editing the active file by hand can currently munge it fairly badly even
  if the server is throttled unless you reload active before restarting
  the server.  This could be avoidable for at least that particular case
  by checking the mtime of active before and after the server was
  throttled.

* innreport silently discards news.notice entries about most of the errors
  innfeed generates.  It should ideally generate some summary, or at least
  note that some error has occurred and the logs should be examined.

* INN's message ID parser should be more forgiving about surrounding
  whitespace.  Right now, it will reject messages with a trailing space in
  the Message-ID header.

* nnrpd doesn't check the message ID of a posted article for syntactic
  validity before remailing it to the moderator, since normally it relies
  on innd to check the message ID.  The message ID checking code from
  innd/art.c should be moved into lib so that nnrpd can use it as well.

* Currently, if the list of newsgroups on an Xref slave is out of sync
  with the newsgroups on the master, receiving an article crossposted to
  one of the groups that doesn't exist on the slave will cause the slave
  to throttle.  This isn't the best behavior; the server should either
  optionally create the missing newsgroup or just ignore that crossposted
  group (and modify Xref accordingly?).

* Handling of compressed batches needs to be thoroughly reviewed by
  someone who understands how they're supposed to work.  It's not clear
  that _PATH_GZIP is being used correctly at the moment and that
  compressed batch handling will work right now on systems that don't have
  gzip installed (but that do have uncompress).

* innfeed's statistics don't add up properly all the time.  All of the
  article dispositions don't add up to the offered count like they should.
  Some article handling must not be recorded properly.

* innd's counting of article size doesn't always work properly, and it can
  accept articles that are larger than its configured limit.  It's not
  clear exactly where this is happening.

* If a channel feed exits immediately, innd respawns it immediately,
  causing thrashing of the system and a huge spew of errors in syslog.  It
  should mark the channel as dormant for some period of time before
  respawning it, perhaps only if it's already died multiple times in a
  short interval.

* ctlinnd begin <site-name> was causing innd to core dump.

* Handling of innfeed's dropped batches needs looking at.  There are three
  places where articles can fall between the cracks:  an innfeed.togo file
  written by innd when the feed can't be spawned, a batch file named after
  the feed name which can be created under similar circumstances, and the
  dropped files written by innfeed itself.  procbatch can clean these up,
  but has to be run by hand.

* When using tradspool, groups are not immediately added to tradspool.map
  when created, making innfeed unable to find the articles until after
  some period of time.  Part of the problem here is that tradspool only
  updates tradspool.map on a lazy basis, when it sees an article in that
  group, since there is no storage hook for creation of a new group.

* nntpget doesn't handle long lines in messages.

* WP feeds break if there are spaces in the Path header, and the inn.conf
  parser doesn't check for this case and will allow people to configure
  their server that way.  (It's not clear that the latter is actually a
  bug, given the new USEFOR attempt to allow folding of Path headers, but
  the space needs to be removed for WP feeds.)

* Error handling in the history backend needs to be reviewed, since it
  currently is always printing out errno regardless of whether it's
  meaningful.  The error handling needs to record errno if it's useful and
  the reporting function should only print it out if it's useful for that
  error.

* innd returns 437 for articles that were accepted but filed in the junk
  group.  It should probably return the appropriate 2xx status code in
  that case instead.

* Someone should go through the BUGS sections of all of the manpages and
  fix those for which the current behavior is unacceptable.


Requested New Features

* Consider implementing the HEADERS command as discussed rather
  extensively in news.software.nntp.  [Greg Andruk has a preliminary
  patch.]

* There have been a few requests for the ability to programmatically set
  the subject of the report generated by news.daily, with escapes that are
  filled in by the various pieces of information that might be useful.

* A bulk cancel command using the MODE CANCEL interface.  Possibly through
  ctlinnd, although it may be a bit afield of what ctlinnd is currently
  for.

* Sven Paulus's patch for nnrpd volume reports should be integrated.  See
  <ftp://ftp.tin.org/pub/news/servers/inn/unofficial-patches/
  patch-inn-2.2.x-artstat+list+overstat>.

* Lots of people encrypt X-Trace in various ways.  Should that be offered
  as a standard option?  The first data element should probably remain
  unencrypted so that the O flag in newsfeeds doesn't break.

  Should there also be an option not to generate X-Trace?  And this whole
  area may change if USEFOR ever standardizes poster trace information;
  it's been proposed to put it in the path tail instead.  The current
  USEFOR trend as of January, 2001 appears to be towards an Injector-Info
  header with this information, allowing a token or an injecting hostname.
  For a token, one really wants it to be hierarchically structured for
  spam filtering even if it's encrypted (in other words, to get a "group"
  of clients, one could just match the first n bytes of the token instead
  of the whole thing).

  Olaf Titz suggests:

      This can be done by formatting the (rest of) the header in a way
      that fields are always a multiple of 8 bytes and applying a 64 bit
      block cipher in ECB mode on it.  But then we would be better off
      using binary fields, as the timestamp is 9 bytes and an IP address
      10-12 bytes.

      Combining the timestamp and PID into one block, adding an
      authenticated user field and omitting the redundant formatted time
      would give the following format:

      X-Trace: g212.hadiko.de [395109AA000016FF] [AC14302A00000000] [...]
                               time    |    pid   ip      |reserved  user

* ctlinnd flushlogs currently renames all of the log files.  It would be
  nice to support the method of log rotation that most other daemons
  support, namely to move the logs aside and then tell innd to reopen its
  log files.  Ideally, that behavior would be triggered with a SIGHUP.
  scanlogs would have to be modified to handle this.

  The best way to support this seems to be to leave scanlogs as is by
  default, but also add two additional modes.  One would flush all the
  logs and prepare for the syslog logs to be rotated, and the other would
  do all the work needed after the logs have been rotated.  That way, if
  someone wanted to plug in a separate log rotation handler, they could do
  so and just call scanlogs on either side of it.  The reporting portions
  of scanlogs should be in a separate program.

* Several people have Perl interfaces to pieces of INN that should ideally
  be part of the INN source tree in some fashion.  Greg Andruk has a bunch
  of stuff that Russ has copies of, for example.

* Investigate using the new, stricter date parsing code in libinn for
  nnrpd rather than the extremely lenient parsedate routine.

* There are various available patches for Cancel-Lock and an Internet
  draft; support should be added to INN for both generation and
  verification (definitely optional and not on by default at this point).

* It would be nice to be able to reload inn.conf (although difficult, due
  to the amount of data that's generated from it and stashed in various
  places).  This will need to wait for the new configuration parsing
  library and an inn.conf parser that uses it.

* remembertrash currently rejects and remembers articles with syntax
  errors as well as things like unwanted newsgroups and unwanted
  distributions, which means that if a peer sends you a bunch of mangled
  articles, you'll then also reject the correct versions of the articles
  from other peers.  This should probably be rethought.

* Additional limits for readers.conf:  Limit on concurrent parallel reader
  streams, limit on KB/second download (preliminary support for this is
  already in), and a limit on maximum posted articles per day (tied in
  with the backoff stuff?).  These should be per-IP or per-user, but
  possibly also per-access group.  (Consider pulling the -H, -T, -X, and 
  -i code out from innd and using it here.)

* timecaf should have more configurable parameters (at the least, how
  frequently to switch to a new CAF file should be an option).
  storage.conf should really be extended to allow method-specific
  configuration for things like this (and to allow the cycbuff.conf file
  to be merged into storage.conf).

* Allow generation of arbitrary additional information that could go in
  overview by using embedded Perl or Python code.  This might be a cleaner
  way to do the keywords code, which really wants Perl's regex engine
  ideally.  It would also let one do something like doing MD5 hashes of
  each article and putting that in the overview if you care a lot about
  making sure that articles aren't corrupted.

* Allow some way of accepting articles regardless of the Date header, even
  if it's far into the future.  Some people are running into articles that
  are dated years into the future for some reason that they still want to
  store on the server.

* There was a request to make --program-suffix and the other name
  transformation options to autoconf work.  The standard GNU package does
  this with really ugly sed commands in the Makefile rules; we could
  probably do better, perhaps by substituting the autoconf results into
  support/install-sh.

* INN currently uses hash tables to store the active file internally.  It
  would be worth trying ternary search trees to see if they're faster; the
  data structure is simpler, performance may be comparable for hits and
  significantly better for misses, sizing and resizing becomes a non-issue,
  and the space penalty isn't too bad.  A generic implementation is already
  available in libinn.  (An even better place to use ternary search trees
  may be the configuration parser.)

* Provide an innshellvars equivalent for Python.

* inncheck should check the syntax of all the various files that are
  returned by LIST commands, since having those files present with the
  wrong syntax could result in non-compliant responses from the server.
  Possibly the server should also refuse to send malformatted lines to
  the client.

* ctlinnd reload incoming.conf could return a count of the hosts that
  failed, or even better a list of them.  This would make pruning old
  stuff out of incoming.conf much easier.

* nnrpd could use sendfile(2), if available, to send articles directly
  to the socket (for those storage methods where to-wire conversion is
  not needed).  This would need to be added to the storage API.

* Somebody should look at keeping the "newsgroups" file more accurate
  (e.g. newgroups for existing groups should change description, better
  checkgroups handling, checking for duplicates)

* The by-domain statistics innreport generates for nnrpd count all local
  connections (those with no "." in the hostname) in with the errors as
  just "?".  The host2dom function could be updated to group these as
  something like "Local".

* news.daily could detect if expire segfaults and unpause the server.

* When using SSL, track the amount of data that's been transferred to the
  client and periodically renegotiate the session key.

* When using SSL, use SSL_get_peer to get a verified client certificate,
  if available, and use it to create an additional header line when
  posting articles (X-Auth-Poster?).  This header could use:

      X509_NAME_oneline(X509_get_subject_name(peer),...)

  for the full distinguished name, or

      X509_name_get_text_by_NID(X509_get_subject_name(peer),
                                NID_commonName, ...)
   
  for the client's "common name" alone.

* When using SSL, use the server's key to generate an HMAC of the body of
  the message (and most headers?), then include that digest in the
  headers.  This allows a news administrator to determine if a complaint
  about the content of a message is fradulent since the message was
  changed after transmission.


General Projects

* All the old packages in unoff-contrib should be reviewed for integration
  into INN.

* It may be better for INN on SysV-derived systems to use poll rather than
  select.  The semantics are better, and on some systems (such as Solaris)
  select is limited to 1024 file descriptors whereas poll can handle any
  number.  Unfortunately, the API is drastically different between the
  two and poll isn't portable, so supporting both cleanly would require a
  bit of thought.

* Currently only innd and innfeed increase their file descriptor limits.
  Other parts of INN, notably makehistory, may benefit from doing the same
  thing if they can without root privileges.

* The Tcl filtering support code has undergone serious bitrot and needs
  some work to fix it and make it work with modern versions of Tcl and the
  current version of INN.  It also lacks a lot of the functionality of the
  Perl and Python filters, if anyone cares.

* Revisit support for aliased groups and what nnrpd does with them.
  Should posts to the alias automatically be redirected to the real group?
  Regardless, the error return should provide useful information about
  where to post instead.  Also, the new overview API, for at least some of
  the overview methods, truncated the group status at one character and
  lost the name of the group to which a group is aliased; that needs to be
  fixed.

* More details as to why a message ID is bad would be useful to return to
  the user, particularly for rnews, inews, etc.  innd also rejects message
  IDs with trailing spaces, which can be hard to check.

* Support putting the active file and history file in different
  directories without hand-editing a bunch of files.

* nnrpd's NNTP command parsing interacts poorly with AUTHINFO and
  passwords containing spaces.  The correct solution isn't clear; check
  with the current NNTP RFC draft and how existing clients handle it?

* frontends/pullnews and contrib/backupfeed solve the same problem; the
  best ideas of both should be unified into one script.

* actsyncd could stand a rewrite and cleaner handling of both
  configuration and syncing against multiple sources which are canonical
  for different sets of groups.

* send-nntp and nntpsend basically do the same thing; send-nntp could
  probably be removed (possibly with some extra support in nntpsend for
  doing simpler things).


Long-Term Projects

* Look at turning header parsing into a library of some sort.  Lots of INN
  does this, but different parts of INN need subtly different things, so
  the best best API is unclear.

* INN's header handling needs to be checked against the current USEFOR
  draft.  This may want wait until after we have a header parsing library.

* The innd filter should be able to specify additional or replacement
  groups into which an article should be filed, or even spool the article
  to a local disk file rather than storing it.  (See the stuff that the
  nnrpd filter can already do.)

* Add authentication via SASL to nnrpd.  This is a boatload of additional
  issues, particularly if we want to add authentication methods like
  Kerberos that require their own separate libraries (although we should
  use Cyrus's SASL libraries, which will simplify a lot of that).
  [Jeffrey Vinocur is working on a standard for this.]

* When articles expire out of a storage method with self-expire
  functionality, the overview and history entries for those articles
  should also be expired immediately.  Otherwise, things like the GROUP
  command don't give the correct results.  This will likely require a
  callback that can be passed to CNFS that is called to do the overview
  and history cleanup for each article overwritten.  It will also require
  the new history API.

* Feed control, namely allowing your peers to set policy on what articles
  you feed them (not just newsgroups but max article size and perhaps even
  filter properties like "non-binary").  Every site does this a bit
  differently.  Some people have web interfaces, some people use GUP, some
  people roll their own alternate things.  It would really be nice to have
  some good way of doing this as part of INN.  It's worth considering an
  NNTP extension for this purpose, although the first step is to build a
  generic interface that an NNTP extension, a web page, etc. could all
  use.  (An alternate way of doing this would be to extend IHAVE to pass
  the list of newsgroups as part of the command, although this doesn't
  seem as generally useful.)

* Traffic classification as an extension of filtering.  The filter should
  be able to label traffic as binary (e.g.) without rejecting it, and
  newsfeeds should be extended to allow feeding only non-binary articles
  (e.g.) to a peer.

* External authenticators should also be able to do things like return a
  list of groups that a person is allowed to read or post to.  Currently,
  maintaining a set of users and a set of groups, each of which some
  subset of the users is allowed to access, is far too difficult.  For a
  good starting list of additional functionality that should be made
  available, look at everything the Perl authentication hooks can do.
  This should probably wait for the configuration file parsing rewrite.

* Allow nnrpd to spawn long-running helper processes.  Not only would this
  be useful for handling authentication (so that the auth hooks could work
  without execing a program on every connection), but it may allow for
  other architectures for handling requests (such as a pool of helpers
  that deal only with overview requests).  More than that, nnrpd should
  *be* a long-running helper process that innd can feed open file
  descriptors to.  [Aidan Culley has ideas along these lines.]

* The tradspool storage method requires assigning a number to every
  newsgroup (for use in a token).  Currently this is maintained in a
  separate tradspool.map file, but it would be much better to keep that
  information in the active file where it can't drop out of sync.  A code
  assigned to each newsgroup would be useful for other things as well,
  such as hashing the directories for the tradindexed overview.  For use
  for that purpose, though, the active file would have to be extended to
  include removed groups, since they'd need to be kept in the active file
  to reserve their numbers until the last articles expired.

* The locking of the active file leaves something to be desired; in
  general, the locking in INN (for the active file, the history file,
  spool updates, overview updates, and the like) needs a thorough
  inspection and some cleanup.  A good place to start would be tracing
  through the pause and throttle code and write up a clear description of
  what gets locked where and what is safely restarted and what isn't.
  Long term, there needs to be a library locking routine used by
  *everything* that needs to write to the history file, active file, etc.
  and that keeps track of the PID of the process locking things and is
  accessible via ctlinnd.

* There is a fundamental problem with the current design of the
  control.ctl file.  It combines two things:  A database of hierarchies,
  their maintainers, and related information, and a list of which
  hierarchies the local server should honor.  These should be separated
  out into the database (which could mostly be updated from a remote
  source like ftp.isc.org and then combined with local additions) and a
  configured list of hierarchies (or sub-hierarchies within hierarchies)
  that control messages should be honored for.  This should be reasonably
  simple although correct handling of checkgroups could get a mite tricky.

* Possible NNTP extension:  Compression of the protocol, using gzip,
  bzip2, or some other technique.  Particularly useful for long lists like
  the active file information or the overview information, but possibly
  useful in general for other things.

* Install wizards.  Configuring INN is currently very complex even for an
  experienced news admin, and there are several fairly standard
  configurations that shouldn't be nearly that complicated to get running
  out of the box.  A little interactive Perl script asking some simple
  questions could probably get a lot of cases easily right.

* One ideally wants to be able to easily convert between different
  overview formats or storage methods, refiling articles in place.  This
  should be possible once we have a history API that allows changing the
  storage location of an article in-place.

* Set up the infrastructure required so that INN can use alloca.  This
  would significantly decrease the number of calls to malloc needed and
  would be a lot more convenient.

* A serious investigation into whether INN could use a garbage collector
  is probably a good idea.  The network buffers probably need to be
  handled with decidated code, but there are a lot of other incidental
  allocations and deallocations that may be much more efficient and safer
  using a garbage collector.

* Look at integrating asprintf and vasprintf.  Russ already tried this
  once and couldn't see a good way of doing it (particularly vasprintf)
  without hooking deep into an sprintf implementation, because the simple
  hack of calling vsnprintf first, allocating that much memory, and then
  calling it again on the new buffer doesn't work for vasprintf (you can't
  reprocess the arguments).

* Support building in a separate directory than the source tree.  It may
  be best to just support this via lndir rather than try to do it in
  configure, but it would be ideal to add support for this to the autoconf
  system.  Unfortunately, the standard method requires letting configure
  generate all of the makefiles, which would make running configure and
  config.status take much longer than it does currently.

* Look at adding some kind of support for MODE CANCEL via network sockets
  and fixing up the protocol so that it could possibly be standardized
  (the easiest thing to do would probably be to change it into a CANCEL
  command).  If we want to get to the point where INN can accept and even
  propagate such feeds from dedicated spam filters or the like, there must
  also be some mechanism of negotiating policy in order to decide what
  cancels the server wants to be fed.

* The "possibly signed" char data type is one of the inherent flaws of C.
  Some other projects have successfully gotten completely away from this
  by declaring all of their strings to be unsigned char, defining a macro
  like U that casts strings to unsigned char for use with literal strings,
  and always using unsigned char everywhere.  Unfortunately, this also
  requires wrappering all of the standard libc string functions, since
  they're prototyped as taking char rather than unsigned char.  The
  benefits include cleaner and consistent handling of characters over 127,
  better warnings from the compiler, consistent behavior across platforms
  with different notions about the signedness of char, and the elimination
  of warnings from the <ctype.h> macros on platforms like Solaris where
  those macros can't handle signed characters.  We should look at doing
  this for INN.

* It would clean up a lot of code considerably if we could just use mmap
  semantics regardless of whether the system has mmap.  It may be possible
  to emulate mmap on systems that don't have it by reading the entirety of
  the file into memory and setting the flags that require things to call
  mmap_flush and mmap_invalidate on a regular basis, but it's not clear
  where to stash the file descriptor that corresponds to the mapped file.

* Figure out some Samba library that we can link against for the Samba
  authenticator so that we can get all the Samba code back out of INN's
  source tree; we don't want to maintain it.

* Consider replacing the awkward access: parameter in readers.conf with
  separate commands (e.g. "allow_newnews: true") or otherwise cleaning up
  the interaction between access: and read:/post:.  Note that at least
  allownewnews: can be treated as a setting for overriding inn.conf and
  should be very easy to add.

* Add a localport: parameter (similar to localaddress:) to readers.conf
  auth groups.  With those two parameters (and ssl_required:) we
  essentially eliminate the need to run multiple instances of nnrpd just to
  use different configurations.

* Various things may break when trying to use data written while compiled
  with large file support using a server that wasn't so compiled (and vice
  versa).  The main one is the history file, but tradindexed is also
  affected and buffindexed has been reported to have problems with this
  as well.  Ideally, all of INN's data files should be as portable as
  possible.


Complete Code Reorganization

At some point, we should probably abandon and archive the current CVS
repository, reimport all of the current source files, and start with a
fresh repository with a better revision control system such as Subversion.
A better revision control system would let us rename and move things
around arbitrarily, something CVS doesn't handle at all well.  Should this
ever be done, we should consider doing all of the following at the same
time:

* Don't include any generated files in the CVS tree.  Maintainers should
  have autoconf and friends, pod2text and pod2man, and bison around anyway.
  This would save a bunch of extra check-ins, remove the danger of the
  generated files getting out of sync, and drastically reduce the
  repository size in the case of configure.

* Don't include any of the generated man pages in the CVS tree, as an
  additional case of the above.  All of the documentation should be in POD
  and we can generate the man pages as part of the snapshot process.

* storage should be reserved just for article storage; the overview
  methods should be in a separate overview tree.

* The split between frontends and backends is highly non-intuitive.  Some
  better organization scheme should be arrived at.  Perhaps something
  related to incoming and outgoing, with programs like cnfsstat moved into
  the storage directory with the other storage-related code?

* Add a separate utils directory for things like convdate, shlock,
  shrinkfile, and the like.  Some of the scripts may possibly want to go
  into that directory too.

* The lib directory possibly should be split so that it contains only code
  always compiled and part of INN, and the various replacements for
  possibly missing system routines are in a separate directory (such as
  replace).  These should possibly be separate libraries; there are things
  that currently link against libinn that only need the portability
  pieces.

* The doc directory really should be broken down further by type of
  documentation or section or something; it's getting a bit unwieldy.

* Untabify and reformat all of the code according to a consistent coding
  style which would then be enforced for all future check-ins.