This README describes analog0.91beta. For the latest version of analog, see the analog home page.
This program analyses logfiles in both the common log format and NCSA old format from WWW servers. It is designed to be fast on long logfiles and to produce attractive statistics. For more details, see the
For examples of the output see This program may be freely distributed and modified provided full credit is given to Stephen Turner (sret1@cam.ac.uk), and that this condition is retained. However, please only distribute it in tact, including the domains file and this README. No warranty of any sort is given or implied for this program.analog.c The program. analogo.gif A logo for the top of the output. analhead.h A header file, with various settable options. domains.tab A file matching internet domains to their locations.
DOMAINSFILE and LOGFILE, and will want
to change HOSTNAME and
HOSTURL. You can change any of the rest to your desires, but equally it's
probably OK to leave them as they are, at least at first.
ad Andorra ae United Arab Emirates [...]There can be arbitrary space between the code and the corresponding location. The codes are converted to lower case. Use
? (or anything starting with
?) for the name if you want the domain
to be recognised, but don't want the name to be printed out.
After each domain can come subdomains of that domain in the same format. These
will be printed out in the domain report after the domain to which they apply,
in that order. So, for example, the domains file I use contains the lines
ug Uganda uk United Kingdom cam.ac.uk University of Cambridge statslab.cam.ac.uk ? um US Minor Outlying Islandsand the output looks like
103118 : 46.60% : .uk (United Kingdom) ( 64975):( 35.46%): cam.ac.uk (University of Cambridge) ( 47133):( 20.56%): statslab.cam.ac.uk 49271 : 12.49% : .edu (USA Educational)Numerical subdomains (which have most significant part on the left) can occur before the first domain. They will look like
131 The Ever-Popular 131 domainor
131.111 ?Comments can occur in the domains file. They are introduced by the character
#.
So you could write, for example
uk United Kingdom cam.ac.uk University of Cambridge statslab.cam.ac.uk ? # This host # Any other UK ones?
FILEALIAS file1 file2Whenever
file1 occurs in the logfile, it is to be replaced by
file2. Analog already understands that
/dir/index.html is the same as /dir/ and
translates `escaped' entities (e.g., %7E is the same
as ~) so these don't need to be specified separately.
If * is placed at the end of the first entry, then all
filenames starting with file1 will be changed to start
with file2. So, for example, after the command
FILEALIAS /~sret1/statprog/* /~sret1/analog/a filename looking like
/~sret1/statprog/statprog/stat.c
will be understood as /~sret1/analog/statprog/stat.c.
(Note that the conversion is done only once for each filename; you
don't get /~sret1/analog/analog/stat.c).
The second command that can occur in the configuration file is
HOSTALIAS host1 host2This is useful if your server records local hosts in the logfile instead of full internet names. Also, if a host has two names, they can be combined in this way. So, for example, I might find it convenient to use
HOSTALIAS lion lion.statslab.cam.ac.uk HOSTALIAS www lion.statslab.cam.ac.uk HOSTALIAS www.statslab.cam.ac.uk lion.statslab.cam.ac.ukAgain, only one conversion is done per host, which is why I need both the second and the third line.
Next there are some commands for analysing only part of the logfile. Use of these commands can slow the program down a bit. The command
FILEIGNORE filenameignores altogether logfile entries due to certain files. Again, an asterisk (
*) can appear at the end of the name. Another command
FILEONLY filenamelooks at only that file. If two
FILEONLY commands appear, the
program will look at entries matching either of them. For example
FILEONLY /~sret1/* FILEONLY /~sret2/* FILEIGNORE /~sret1/backgammon/* FILEIGNORE /~sret2/home.htmlwill look at my files and
sret2's files, but not my backgammon
files or sret2's home page. These commands are applied after
any aliasing of filenames has been done, but the files given in these names
are not subject to any aliasing.
There are similar commands for including or excluding certain hosts. For example,
HOSTONLY *.statslab.cam.ac.uk HOSTIGNORE lion.statslab.cam.ac.ukwill only include hosts from my local site, but not
lion.statslab. In this case, *.statslab.cam.ac.uk
matches the host statslab.cam.ac.uk. If you want to exclude it,
you can give a HOSTIGNORE line for it.
Also, all of the commandline arguments can have values specified in the configuration file. These values are overridden by the commandline values, but override the default values given in the header file. This is useful if you often process the logfile in two different ways; then you can get one of them just by supplying a configuration file without the need to recompile the program or specify a long list of commandline arguments. The legal values for variables are exemplified below.
LOGFILE logfile # the logfile to be analysed
DOMAINSFILE domainsfile # where to get the domains from
LOGOURL logourl # the URL where the logo for the report title is.
LOGO on # or off. Whether to use the logo.
HOSTNAME "Stephen's pages" # our organisation
HOSTURL http://www.statslab.cam.ac.uk/ # where to find us
REQFLOOR 1 # if sorting by requests or alphabetically, the
# fewest requests needed to get on the request
# report; if by bytes, the min traffic in 1/100ths
# of a percent; if negative, do a 'top n' report.
DIRFLOOR 1 # directory report
DOMFLOOR -10 # domain report
REQSORTBY byrequests # how to sort the request report
DIRSORTBY alphabetical # directory report
DOMSORTBY bybytes # domain report
MARKCHAR * # the character for graphical displays
MARKCHAR '#' # in quotes so that it isn't a comment
PAGEWIDTH 65 # the width of the output pages
MONTHLY on # do a monthly report?
DAILY off # a daily report?
HOURLY on # an hourly report?
DOMAIN on # a domain report?
DIRECTORY on # a directory report?
REQUEST off # a request report?
DIRLEVEL 2 # the level of the directory report
REQTYPE pages # what to print in the request report; pages or all
PAGELINKS on # whether to link to pages in the request report
COUNTHOSTS on # count the total number of hosts who have visited?
LASTSEVEN on # statistics for the last 7 days?
NUMLOOKUP off # try and resolve numerical addresses; NB SLOW!
MONTHLYUNIT 1000 # the size of the character in the graphical displays
HOURLYUNIT 0 # 0 represents 'choose something sensible automatically'
DAILYUNIT 0
gcc analog.c -o analog -O2to use the gcc compiler.
The program usage is
analog [logfile | -] [options]The name of the logfile can be omitted, in which case the default (as specified in analhead.h) will be used.
- will use stdin. This is useful for
constructing pipes, e.g.,
cat access_log1 access_log2 | analog -(This is safe, because the program works properly even if the logfile it is analysing is not in chronological order).
The legal options are as follows. All of them have default values which can be changed in the header file or configuration file. If two contradictory options are specified, the latter is heeded (this might be useful if you have an alias set up, for example). Note that items in square brackets are optional; also that there can be no space between an option and any arguments to it. See the pages cited above for the meaning of the various reports.
-m Don't do a monthly report.
+m[n] Do a monthly report. Make each character in the graph
worth n requests; if n = 0, choose something sensible.
-d | +d[n] Don't do/Do a daily summary.
-h | +h[n] Don't do/Do an hourly summary.
-o Don't do a domain report.
+o[a|b|r][n] Do a domain report. Sort alphabetically/by bytes/
by requests. If n is positive, only print domains with
at least n requests (r or a) or at least n/100ths of a
percent of the traffic (b). If n is negative, print the
top n (well, top -n actually) on the report.
-i | +i[a|b|r][n] Directory report.
-r | +r[a|b|r][n] Request report; report pages only.
-R | +R[a|b|r][n] Request report; report all filenames.
-s | +s Don't/do count number of distinct hosts.
-7 | +7 Don't/do give statistics for last 7 days.
-ln Level (or depth) of directory report.
-k | +k | +kk Don't link in the request report / link to pages /
link to everything.
-cchar The character to use in the graphical displays.
-wn The width of the page for output. 65 is about right.
-nhostname The name of your organisation or WWW server. (Printed
in the title of the output page).
-uhosturl The URL of your front page. Use -u- if you don't want the
top line of your output to be linked to anywhere.
-fdomainsfile The domains file you want to use.
+gconffile The configuration file you want to use.
-g Don't use a configuration file.
-p Don't put a logo at the top (p for picture)
+p[logourl] Do put a logo. Find it at the given URL.
-1 / +1 Don't / do try and look up all numerical address.
(May be slow).
The figures for the last 7 days are for the 7 days before the program is run, not before the last entry in the logfile. Moreover, they are actually for any times after 7 days before the program is started; so `future' requests will be included if some more requests come in while the program is reading the logfile (or if the clock on the computer running the program is not sychronised with the one on the computer that recorded the logs).
If the logfile contains any old-style lines, they will be read and understood, but as they do not contain any data about bytes transferred, such data will no longer be collected. If any of the reports was to be sorted by bytes, a warning will be issued and sorting will be by requests instead.
If we are doing a `top n' report and two entries tie for nth place, only one will be printed.
The reported `running time' is elapsed real time, not CPU time.
If you specify +oa-10 you really do get the top ten domains
alphabetically.
Subdomains will be reported in the domain report if and only if their parent domain is, independent of their usage.
The program uses a rather naïve method to determine what is a `page'
for the purpose of the request report. A page is anything that ends in
.html or .htm or /.
The bytes aren't reported correctly. This is really a bug in the logfile. Servers don't actually measure the number of bytes transferred, they measure the size of the file they are about to transfer, so if a connection is interrupted, they may write down more bytes than were actually transferred.
The following list contains in no particular order things that I think would be good to be able to do with analog, as well as all things that people have suggested that I have not yet implemented. I welcome comments on how high a priority people place on the various items, and what other things would be useful.
*.cam.ac.uk and
(not statslab).cam.ac.uk.
Thanks are also due to the following who helped in the early stages of writing this program.
Page last modified: 04-Jul-95