.\" Automatically generated by Pod::Man v1.37, Pod::Parser v1.32 .\" .\" Standard preamble: .\" ======================================================================== .de Sh \" Subsection heading .br .if t .Sp .ne 5 .PP \fB\\$1\fR .PP .. .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" Set up some character translations and predefined strings. \*(-- will .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left .\" double quote, and \*(R" will give a right double quote. \*(C+ will .\" give a nicer C++. Capital omega is used to do unbreakable dashes and .\" therefore won't be available. \*(C` and \*(C' expand to `' in nroff, .\" nothing in troff, for use with C<>. .tr \(*W- .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' .ie n \{\ . ds -- \(*W- . ds PI pi . if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch . if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch . ds L" "" . ds R" "" . ds C` "" . ds C' "" 'br\} .el\{\ . ds -- \|\(em\| . ds PI \(*p . ds L" `` . ds R" '' 'br\} .\" .\" If the F register is turned on, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.Sh), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .if \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . nr % 0 . rr F .\} .\" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .hy 0 .if n .na .\" .\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2). .\" Fear. Run. Save yourself. No user-serviceable parts. . \" fudge factors for nroff and troff .if n \{\ . ds #H 0 . ds #V .8m . ds #F .3m . ds #[ \f1 . ds #] \fP .\} .if t \{\ . ds #H ((1u-(\\\\n(.fu%2u))*.13m) . ds #V .6m . ds #F 0 . ds #[ \& . ds #] \& .\} . \" simple accents for nroff and troff .if n \{\ . ds ' \& . ds ` \& . ds ^ \& . ds , \& . ds ~ ~ . ds / .\} .if t \{\ . ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u" . ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u' . ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u' . ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u' . ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u' . ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u' .\} . \" troff and (daisy-wheel) nroff accents .ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V' .ds 8 \h'\*(#H'\(*b\h'-\*(#H' .ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#] .ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H' .ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u' .ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#] .ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#] .ds ae a\h'-(\w'a'u*4/10)'e .ds Ae A\h'-(\w'A'u*4/10)'E . \" corrections for vroff .if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u' .if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u' . \" for low resolution devices (crt and lpr) .if \n(.H>23 .if \n(.V>19 \ \{\ . ds : e . ds 8 ss . ds o a . ds d- d\h'-1'\(ga . ds D- D\h'-1'\(hy . ds th \o'bp' . ds Th \o'LP' . ds ae ae . ds Ae AE .\} .rm #[ #] #H #V #F C .\" ======================================================================== .\" .IX Title "OVDB 5" .TH OVDB 5 "2008-04-06" "INN 2.4.5" "InterNetNews Documentation" .SH "NAME" ovdb \- Overview storage method for INN .SH "DESCRIPTION" .IX Header "DESCRIPTION" Ovdb is a storage method that uses the BerkeleyDB library to store overview data. It requires version 2.6.x or later of the BerkeleyDB library, but has mostly been tested with version 3 and 4. .PP Ovdb makes use of the full transaction/logging/locking functionality of the BerkeleyDB environment. BerkeleyDB may be downloaded from and is needed to build the ovdb backend. .SH "UPGRADING" .IX Header "UPGRADING" This is version 2 of ovdb. If you have a database created with a previous version of ovdb (such as the one shipped with \s-1INN\s0 2.3.0) your database will need to be upgraded using \fIovdb_init\fR\|(8). See the man page \&\fIovdb_init\fR\|(8) for upgrade instructions. .SH "INSTALLATION" .IX Header "INSTALLATION" To build ovdb support into \s-1INN\s0, specify the option \f(CW\*(C`\-\-with\-berkeleydb\*(C'\fR when running the configure script. By default, configure will search for a BerkeleyDB tree in several likely locations, and choose the highest version (based on the name of the directory, e.g., \fIBerkeleyDB.3.0\fR) that it finds. There will be a message in the configure output indicating the chosen pathname. .PP You can override this pathname by adding a path to the option, e.g., \&\f(CW\*(C`\-\-with\-berkeleydb=/usr/BerkeleyDB.3.1\*(C'\fR. This directory is expected to have subdirectories \fIinclude\fR and \fIlib\fR, containing \fIdb.h\fR, and the library itself, respectively. .PP The ovdb database will take up more disk space for a given spool than the other overview methods. Plan on needing at least 1.1 \s-1KB\s0 for every article in your spool (not counting crossposts). So, if you have 5 million articles, you'll need at least 5.5 \s-1GB\s0 of disk space for ovdb. With BerkeleyDB 2.x, the db files are 'grow only'; the library will not shrink them, even if data is removed. So, reserving extra space above the estimate is a good idea. Plus, you'll need additional space for transaction logs: at least 100 \s-1MB\s0. By default the transaction logs go in the same directory as the database. To improve performance, they can be placed on a different disk \*(-- see the \s-1DB_CONFIG\s0 section. .SH "CONFIGURATION" .IX Header "CONFIGURATION" To enable ovdb, set the \fIovmethod\fR parameter in \fIinn.conf\fR to \f(CW\*(C`ovdb\*(C'\fR. The ovdb database is stored in the directory specified by the \&\fIpathoverview\fR paramter in \fIinn.conf\fR. This is the \*(L"\s-1DB_HOME\s0\*(R" directory. To start out, this directory should be empty (other than an optional \&\fI\s-1DB_CONFIG\s0\fR file; see \s-1DB_CONFIG\s0 for details) and \fBinnd\fR (or \&\fBmakehistory\fR) will create the files as necessary in that directory. Make sure the directory is owned by the news user. .PP Other parameters for configuring ovdb are in the \fIovdb.conf\fR\|(5) configuration file. See also the sample \fIovdb.conf\fR. .IP "cachesize" 4 .IX Item "cachesize" Size of the memory pool cache, in kilobytes. The cache will have a backing store file in the \s-1DB\s0 directory which will be at least as big. In general, the bigger the cache, the better. Use \f(CW\*(C`ovdb_stat \-m\*(C'\fR to see cache hit percentages. To make a change of this parameter take effect, shut down and restart \s-1INN\s0 (be sure to kill all of the nnrpds when shutting down). Default is 8000, which is adequate for small to medium sized servers. Large servers will probably need at least 20000. .IP "numdbfiles" 4 .IX Item "numdbfiles" Overview data is split between this many files. Currently, \fBinnd\fR will keep all of the files open, so don't set this too high or \fBinnd\fR may run out of file descriptors. \fBnnrpd\fR only opens one at a time, regardless. May be set to one, or just a few, but only do that if your \s-1OS\s0 supports large (>2G) files. Changing this parameter has no effect on an already-established database. Default is 32. .IP "txn_nosync" 4 .IX Item "txn_nosync" If txn_nosync is set to false, BerkeleyDB flushes the log after every transaction. This minimizes the number of transactions that may be lost in the event of a crash, but results in significantly degraded performance. Default is true. .IP "useshm" 4 .IX Item "useshm" If useshm is set to true, BerkeleyDB will use shared memory instead of mmap for its environment regions (cache, lock, etc). With some platforms, this may improve performance. Default is false. This parameter is ignored if you have BerkeleyDB 2.x .IP "shmkey" 4 .IX Item "shmkey" Sets the shared memory key used by BerkeleyDB when 'useshm' is true. BerkeleyDB will create several (usually 5) shared memory segments, using sequentially numbered keys starting with 'shmkey'. Choose a key that does not conflict with any existing shared memory segments on your system. Default is 6400. This parameter is only used with BerkeleyDB 3.1 or newer. .IP "pagesize" 4 .IX Item "pagesize" Sets the page size for the \s-1DB\s0 files (in bytes). Must be a power of 2. Best choices are 4096 or 8192. The default is 8192. Changing this parameter has no effect on an already-established database. .IP "minkey" 4 .IX Item "minkey" Sets the minimum number of keys per page. See the BerkeleyDB documentation for more info. Default is based on page size: .Sp .Vb 1 \& default_minkey = MAX(2, pagesize / 2048 \- 1) .Ve .Sp The lowest allowed minkey is 2. Setting minkey higher than the default is not recommended, as it will cause the databases to have a lot of overflow pages. Changing this parameter has no effect on an already-established database. .IP "maxlocks" 4 .IX Item "maxlocks" Sets the BerkeleyDB \*(L"lk_max\*(R" parameter, which is the maxmium number of locks that can exist in the database at the same time. Default is 4000. .IP "nocompact" 4 .IX Item "nocompact" The nocompact parameter affects expireover's behavior. The expireover function in ovdb can do its job in one of two ways: by simply deleting expired records from the database, or by re-writing the overview records into a different location leaving out the expired records. The first method is faster, but it leaves 'holes' that result in space that can not immediately be reused. The second method 'compacts' the records by rewriting them. .Sp If this parameter is set to 0, expireover will compact all newsgroups; if set to 1, expireover will not compact any newsgroups; and if set to a value greater than one, expireover will only compact groups that have less than that number of articles. .Sp Experience has shown that compacting has minimal effect (other than making expireover take longer) so the default is now 1. This parameter will probably be removed in the future. .IP "readserver" 4 .IX Item "readserver" Normally, each nnrpd process directly accesses the BerkeleyDB environment. The process of attaching to the database (and detaching when finished) is fairly expensive, and can result in high loads in situations when there are lots of reader connections of relatively short duration. .Sp When the readserver parameter is \*(L"true\*(R", the nnrpds will access overview via a helper server (\fBovdb_server\fR \*(-- which is started by \fBovdb_init\fR). This can also result in cleaner shutdowns for the database, improving stability and avoiding deadlocks and corrupted databases. If you are experiencing any instability in ovdb, try setting this parameter to true. Default is false. .IP "numrsprocs" 4 .IX Item "numrsprocs" This parameter is only used when \fIreadserver\fR is true. It sets the number of ovdb_server processes. As each ovdb_server can process only one transaction at a time, running more servers can improve reader response times. Default is 5. .IP "maxrsconn" 4 .IX Item "maxrsconn" This parameter is only used when \fIreadserver\fR is true. It sets a maximum number of readers that a given ovdb_server process will serve at one time. This means the maximum number of readers for all of the ovdb_server processes is (numrsprocs * maxrsconn). Default is 0, which means an umlimited number of connections is allowed. .SH "DB_CONFIG" .IX Header "DB_CONFIG" A file called \fI\s-1DB_CONFIG\s0\fR may be placed in the database directory to customize where the various database files and transaction logs are written. By default, all of the files are written in the \*(L"\s-1DB_HOME\s0\*(R" directory. One way to improve performance is to put the transaction logs on a different disk. To do this, put: .PP .Vb 1 \& DB_LOG_DIR /path/to/logs .Ve .PP in the \fI\s-1DB_CONFIG\s0\fR file. If the pathname you give starts with a /, it is treated as an absolute path; otherwise, it is relative to the \*(L"\s-1DB_HOME\s0\*(R" directory. Make sure that any directories you specify exist and have proper ownership/mode before starting \s-1INN\s0, because they won't be created automatically. Also, don't change the \s-1DB_CONFIG\s0 file while anything that uses ovdb is running. .PP Another thing that you can do with this file is to split the overview database across multiple disks. In the \fI\s-1DB_CONFIG\s0\fR file, you can list directories that BerkeleyDB will search when it goes to open a database. .PP For example, let's say that you have \fIpathoverview\fR set to \&\fI/mnt/overview\fR and you have four additional file systems created on \&\fI/mnt/ov?\fR. You would create a file \*(L"/mnt/overview/DB_CONFIG\*(R" containing the following lines: .PP .Vb 5 \& set_data_dir /mnt/overview \& set_data_dir /mnt/ov1 \& set_data_dir /mnt/ov2 \& set_data_dir /mnt/ov3 \& set_data_dir /mnt/ov4 .Ve .PP (For BerkeleyDB 2.x, replace \f(CW\*(C`set_data_dir\*(C'\fR with \f(CW\*(C`DB_DATA_DIR\*(C'\fR.) .PP Distribute your ovNNNNN files into the four filesystems. (say, 8 each). When called upon to open a database file, the db library will look for it in each of the specified directories (in order). If said file is not found, one will be created in the first of those directories. .PP Whenever you change \s-1DB_CONFIG\s0 or move database files around, make sure all news processes that use the database are shut down first (including nnrpds). .PP The \s-1DB_CONFIG\s0 functionality is part of BerkeleyDB itself, rather than something provided by ovdb. See the BerkeleyDB documentation for complete details for the version of BerkeleyDB that you're running. .SH "RUNNING" .IX Header "RUNNING" When starting the news system, \fBrc.news\fR will invoke \fBovdb_init\fR. \&\fBovdb_init\fR must be run before using the database. It performs the following tasks: .IP "\(bu" 4 Creates the database environment, if necessary. .IP "\(bu" 4 If the database is idle, it performs a normal recovery. The recovery will remove stale locks, recreate the memory pool cache, and repair any damage caused by a system crash or improper shutdown. .IP "\(bu" 4 Starts the \s-1DB\s0 housekeeping processes (\fBovdb_monitor\fR) if they're not already running. .PP And when stopping \s-1INN\s0, \fBrc.news\fR kills the ovdb_monitor processes after the other \s-1INN\s0 processes have been shut down. .SH "DIAGNOSTICS" .IX Header "DIAGNOSTICS" Problems relating to ovdb are logged to news.err with \*(L"\s-1OVDB\s0\*(R" in the error message. .PP \&\s-1INN\s0 programs that use overview will fail to start up if the ovdb_monitor processes aren't running. Be sure to run \fBovdb_init\fR before running anything that accesses overview. .PP Also, \s-1INN\s0 programs that use overview will fail to start up if the user running them is not the \*(L"news\*(R" user. .PP If a program accessing the database crashes, or otherwise exits uncleanly, it might leave a stale lock in the database. This lock could cause other processes to deadlock on that stale lock. To fix this, shut down all news processes (using \f(CW\*(C`kill \-9\*(C'\fR if necessary) and then restart. \fBovdb_init\fR should perform a recovery operation which will remove the locks and repair damage caused by killing the deadlocked processes. .SH "FILES" .IX Header "FILES" .IP "inn.conf" 4 .IX Item "inn.conf" The \fIovmethod\fR and \fIpathoverview\fR parameters are relevant to ovdb. .IP "ovdb.conf" 4 .IX Item "ovdb.conf" Optional configuration file for tuning. See \s-1CONFIGURATION\s0 above. .IP "\fIpathoverview\fR" 4 .IX Item "pathoverview" Directory where the database goes. BerkeleyDB calls it the '\s-1DB_HOME\s0' directory. .IP "\fIpathoverview\fR/DB_CONFIG" 4 .IX Item "pathoverview/DB_CONFIG" Optional file to configure the layout of the database files. .IP "\fIpathrun\fR/ovdb.sem" 4 .IX Item "pathrun/ovdb.sem" A file that gets locked by every process that is accessing the database. This is used by \fBovdb_init\fR to determine whether the database is active or quiescent. .IP "\fIpathrun\fR/ovdb_monitor.pid" 4 .IX Item "pathrun/ovdb_monitor.pid" Contains the process \s-1ID\s0 of \fBovdb_monitor\fR. .SH "TO DO" .IX Header "TO DO" Implement a way to limit how many databases can be open at once (to reduce file descriptor usage); maybe using something similar to the cache code in ov3.c .SH "HISTORY" .IX Header "HISTORY" Written by Heath Kehoe for InterNetNews .SH "SEE ALSO" .IX Header "SEE ALSO" \&\fIinn.conf\fR\|(5), \fIinnd\fR\|(8), \fInnrpd\fR\|(8), \fIovdb_init\fR\|(8), \fIovdb_monitor\fR\|(8), \&\fIovdb_stat\fR\|(8) .PP BerkeleyDB documentation: in the \fIdocs\fR directory of the BerkeleyDB source distribution, or on the Sleepycat web page: .