For most people, the cache file will not be needed: compressing the logfile using a standard compression utility such as gzip will be sufficient. Compressing a logfile is very efficient owing to the large number of repeated strings: I find about 12 times compression in practice. That in itself may solve your filespace problems, without needing to throw away any information.
The cache file is also not the best format for post-processing the data or feeding it into a spreadsheet. For that you should use the computer readable output style.
Many people have trouble using the cache file, and end up accidentally recording corrupt data. You do need to understand what you're doing before you throw away your logfiles. See the discussion on Procedures below.
If you are going to use the cache file feature, it is also very important that you understand what is and what is not recorded. The summary is that all INCLUDE and EXCLUDE commands, including FROM and TO, and any ALIASes and LOGTIMEOFFSETs, must be applied when you create the cache file, not when you read it later. If you want different sets of options, you must create several cache files from the same logfile.
The reason for this is that it is not possible to reconstruct everything of interest in the logfile from the cache file. The cache file does contain information about the total number of requests for each host and each file, but not about, for example, which files were read by which hosts. (To do so would take up as much disk space as the compressed logfile.) So you cannot later look at only one file and see which hosts read that file.
Another way to look at this: if you do, for example, a HOSTEXCLUDE when reading the cache file, you are not doing a genuine HOSTEXCLUDE because files that that host read will still be included. You are only excluding those hosts from the Host Report, Organisation Report and Domain Report. This is why you must do all the inclusions and exclusions you want when you create the cache file.
When analog reads in a cache file, it does not apply any more aliases to the items. This is to avoid double-aliasing. So you must do any aliases you want at the time you create the cache file. Similarly, it does not obey the LOGTIMEOFFSET variable, to avoid double-offsetting, so any offset you want must be applied at cache-creation time too.
A couple of other minor points. (1) The pattern of failed requests and redirected requests over time is not recorded in the cache file. So although the total number will still be correct, the number in the last 7 days can be under-reported subsequently if the "last 7 days" for a report overlaps the period of the cache file. (2) Times are only recorded to five-minute resolution.
CACHEOUTFILE noneto turn it off again. You will still get the regular output as well as the cache output, unless you request OUTPUT NONE. To avoid overwriting, you cannot set the CACHEOUTFILE to be a file which already exists. (Disclaimer: on some systems, race conditions may very occasionally thwart this check. Also on a few systems, making the file writeable but not readable will allow it to be overwritten). You can include the date in the name of the CACHEOUTFILE in the same way as described earlier for the OUTFILE.
You can read in a previously-made cache file with the CACHEFILE command, or with the +U command line option. As with the LOGFILE command, you can use commas and wild cards to read in several cache files, and read compressed cache files using the UNCOMPRESS mechanism. Note that if you don't want to read a logfile as well as the cache file, you will have to explicitly set the LOGFILE to none.
Sometimes you don't want to record all the types of item in the cache file. You might want to forget about which hosts had accessed your web site, for example, and only remember how many times each file was requested. You can choose not to include one type of item in the cache file by setting its LOWMEM to 3; for example, specify
HOSTLOWMEM 3to exclude hosts from the cache file. Because this is a serious step, analog will produce a warning if you do this. You can even set all six LOWMEMs to 3 if you just want to remember the pattern of requests over time, not even which files were requested.
Here is one way to use the cache files correctly. It's not the only correct way, but I think it's conceptually the simplest. The idea is that whenever you start a new logfile, you make a cache file out of the old logfile. So each cache file contains all the data from one, and only one, logfile. You never use old cache files to make new ones: so you never have a CACHEFILE and a CACHEOUTFILE command in the same configuration file.
Here is the procedure.
Need help with analog? Use the analog-help mailing list.
[ Top | Up | Prev | Next | Map | Index ]