Gentoo Linux: Web-Logs Analysis with AWStats

Sometimes it may be useful to examine the statistics of the web server (frequently visited pages, downloaded files, etc.). As a basic tool can be leveraged the Google Analytics tool, having the plugin for wordpress available. However, why would someone share his/her own statistics with Google, when there is an opensource tool for it in Portage...

To be correct, there are many commercial and open-source solutions for the same. This post has the only aim which is to describe "a free powerful and featureful tool that generates advanced web, streaming, ftp or mail server statistics, graphically", the AWStats.

Proper Installation of Apache, PHP, and Perl

Someone may find useful to have Apache, PHP and Perl support threads/ithreads. It is strictly important to have all three compiled with or without it. Any other combination will not work. Going forward with support included.

$ nano /etc/make.conf

and insert:

APACHE_MODULES="ssl alias log_config mime mime_magic unique_id vhost_alias threads authz_host auth_basic auth_default rewrite dir cgid"
APACHE2_MPMS="event"

then add the specific USE flags (either /etc/make.conf or /etc/portage/package.use):

echo "www-servers/apache ssl threads" >> /etc/portage/package.use
echo "dev-lang/php apache2 berkdb bzip2 cgi cli crypt ctype exif fileinfo filter gd gdbm hash iconv ipv6 json mhash mysql nls phar posix readline session simplexml sqlite3 ssl threads tokenizer truetype unicode xml xpm xsl zip zlib" >> /etc/portage/package.use
echo "dev-lang/perl berkdb gdbm ithreads"

Now, all three (and their dependencies) can be compiled.

$ emerge -av www-servers/apache dev-lang/php dev-lang/perl www-apache/mod_perl

Once done, it may be also usefull to recompile all perl dependent packages (to be compiled with ithreads support) by executing:

$ perl-cleaner --modules ; perl-cleaner --force --libperl

If everything OK, you can go forward. Otherwise, it is necessary to go through the compilation errors and find which USE flag may have brought some confusion into the compilation process.

---

Apache Setup

Assuming the HTTP/HTTPS availability, modify the APACHE2_OPTS directive in /etc/conf.d/apache2 as follows:

APACHE2_OPTS="-D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5"

Modify the default VirtualHost setup in /etc/apache2/vhosts.d/00_default_vhost.conf. Replace the DOMAIN.TLD with the real domain you manage:

Listen 80
NameVirtualHost *:80

    DocumentRoot "/var/www/DOMAIN.TLD"
    ServerName www.DOMAIN.TLD
    ServerAlias DOMAIN.TLD *.DOMAIN.TLD
    ErrorLog "/var/log/apache2/DOMAIN.TLD-error_log"
    CustomLog "/var/log/apache2/DOMAIN.TLD-access_log" combined
        
            Options Indexes Includes FollowSymLinks MultiViews
            AllowOverride All
            Order allow,deny
            Allow from all

Modify the default SSL_VirtualHost setup in /etc/apache2/vhosts.d/00_default_ssl_vhost.conf:

Listen 443
NameVirtualHost *:443

    DocumentRoot "/var/www/DOMAIN.TLD"
    ServerName www.DOMAIN.TLD:443
    ServerAlias *.DOMAIN.TLD:443
    ServerAdmin admin@DOMAIN.TLD
    ErrorLog "/var/log/apache2/DOMAIN.TLD-error_log"
    CustomLog "/var/log/apache2/DOMAIN.TLD-access_log" combined
    SSLEngine on
    SSLProtocol all -SSLv2
    SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5
    # Do not forget to generate the SSL certificate for Apache or get one for free from StartSSL
    SSLCertificateFile "/etc/ssl/apache2/www.crt"
    SSLCertificateKeyFile "/etc/ssl/apache2/www.key"
    # The following TWO only in case of a free StartSSL certificate
    SSLCertificateChainFile "/etc/ssl/apache2/sub.class1.server.ca.pem"
    SSLCACertificateFile /etc/ssl/apache2/ca.pem

    
        SSLOptions +StdEnvVars
    
    
        SSLOptions +StdEnvVars
    

    BrowserMatch "MSIE [2-5]" 
         nokeepalive ssl-unclean-shutdown 
         downgrade-1.0 force-response-1.0

    # The following can replace the original CustomLog above
    CustomLog "/var/log/apache2/DOMAIN.TLD-ssl_request_log" 
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x "%r" %b"
    # Or this one can replace it as well, especially, for a multi-vhost Apache server and log analysis using AWStats
    CustomLog /var/log/apache2/DOMAIN.TLD-access_log vhost
    CustomLog /var/log/apache2/DOMAIN.TLD-ssl-request_log 
        "%t %h %{HTTPS}x %{SSL_PROTOCOL}x %{SSL_CIPHER}x %{SSL_CIPHER_USEKEYSIZE}x %{SSL_CLIENT_VERIFY}x "%r" %b"

Do not forget to make sure the access rights do always belong to the owner under which is the http daemon executed (apache:apache by default). Or simply:

$ chown apache:apache -R /var/www/DOMAIN.TLD

Now, it is highly recommended to restart the Apache server:

/etc/init.d/apache2 restart

If everything OK, you can go forward. Otherwise, it is necessary to go through the execution errors and find which statement may have caused it. It may be also useful to go through the above configured log files (CustomLog and ErrorLog).

---

Installation and AWStats Set Up

Now it's time for AWStats and some useful packages (GeoIP location, IPv6 support, etc.)

$ echo "dev-libs/geoip perl-geoipupdate" >> /etc/portage/package.use
$ echo "www-misc/awstats ipv6 geoip apache2 vhost"  >> /etc/portage/package.use
$ emerge -av dev-libs/geoip www-misc/awstats

Now, copy the example configuration file of AWStats to the one you are going to modify accordingly:

$ cp /etc/awstats/awstats.model.conf /etc/awstats/awstats.DOMAIN.TLD.conf

Modify the configuration accordingly as follows:

LogFile="/var/log/apache2/DOMAIN.TLD-access_log"
LogType=W
LogFormat=1
SiteDomain="DOMAIN.TLD"
HostAliases="localhost 127.0.0.1 REGEX[DOMAIN.TLD$]"
DNSLookup=1
DirData="/var/lib/awstats"
DirCgi="/stats/cgi-bin"
DirIcons="/stats/icon"
LoadPlugin=ipv6
LoadPlugin="geoip GEOIP_STANDARD /usr/share/GeoIP/GeoIP.dat"

Now make sure, the /var/lib/awstats directory as owned by apache:apache, or simply

$ chown apache:apache -R /var/lib/awstats

Once saved, the configuration needs to be executed as follows:

$ awstats.pl -config=DOMAIN.TLD -update -showdropped

If no error occurred, you can go forward. Otherwise, recheck the AWStats's configuration file.

Finally, create an hourly script in CRON:

$ nano /etc/cron.hourly/awstats

and insert:

#!/bin/sh
cd /var/www/awstats_host_name/cgi-bin
awstats.pl -config=DOMAIN.TLD -update > /dev/null 2>&1

and make it executable

$ chmod +x /etc/cron.hourly/awstats

You should also make sure that you will parse logs shortly before every log rotation. To do it you need to add prerotate command to your log rotation configuration. Consult example showing my log rotation configuration for apache access_log files stored in /etc/logrotate.d/apache2 configuration file.

/var/log/apache2/*access_log {
  daily
  missingok
  notifempty
  rotate 365
  dateext
  olddir /var/log/old/apache2
  sharedscripts
  nocompress
  nocreate
  prerotate
        /etc/cron.hourly/awstats
  endscript
  postrotate
        /etc/init.d/apache2 reload > /dev/null 2>&1 || true
  endscript
}

---

Display the AWStats

To be able to display the statistics via a web browser, it is necessary to copy the AWStats into the web browser directory and configure Apache appropriately.

Create a directory in WWW root and copy the awstats structure:

$ mkdir /var/www/DOMAIN.TLD/stats
$ cp -R /usr/share/awstats/wwwroot/* /var/www/DOMAIN.TLD/stats

Then create a simple PHP script to forward you to the AWStats page:

$ nano /var/www/DOMAIN.TLD/stats/index.php

and insert

Make sure the whole stats directory is owned by apache, or simple:

$ chown apache:apache -R /var/www/DOMAIN.TLD/stats

Finally, the apache VirtualHost section needs to be updated by inserting the following code to make the AWStats available via SSL and appropriate credentials (/etc/apache2/vhosts.d/00_default_ssl_vhost.conf).

        Alias /awstatsclasses "/var/www/DOMAIN.TLD/stats/classes/"
        Alias /awstatscss "/var/www/DOMAIN.TLD/stats/css/"
        Alias /awstatsicons "/var/www/DOMAIN.TLD/stats/icon/"
        ScriptAlias /awstats "/var/www/DOMAIN.TLD/stats/cgi-bin/"

        
                Options -Indexes FollowSymLinks
                AllowOverride All
                AuthType Basic
                AuthName "Please, provide the appropriate credentials."
                AuthUserFile /etc/awstats/.htpasswd
                require valid-user
                Order allow,deny
                Allow from all
                SSLRequireSSL
        
        
                Options ExecCGI -Indexes FollowSymLinks
                SetHandler cgi-script
                Order allow,deny
                Allow from all
                SSLRequireSSL
        
        
                AuthType Basic
                AuthName "Please, provide the appropriate credentials."
                AuthUserFile /etc/awstats/.htpasswd
                require valid-user
                SSLRequireSSL

Finally, open the browser and put https://DOMAIN.TLD/stats.

A big thank sending to Andrzej Kardaś.