Tags: #Gentoo Linux #statistics #AWStats #log #analysis #Apache #PHP #perl
Gentoo Linux: Web-Logs Analysis with AWStats
Sometimes it may be useful to examine the statistics of the web server (frequently visited pages, downloaded files, etc.). As a basic tool can be leveraged the Google Analytics tool, having the plugin for wordpress available. However, why would someone share his/her own statistics with Google, when there is an opensource tool for it in Portage…
To be correct, there are many commercial and open-source solutions for the same. This post has the only aim which is to describe “a free powerful and featureful tool that generates advanced web, streaming, ftp or mail server statistics, graphically”, the AWStats.
Proper Installation of Apache, PHP, and Perl
Someone may find useful to have Apache, PHP and Perl support threads/ithreads. It is strictly important to have all three compiled with or without it. Any other combination will not work. Going forward with support included.
$ nano /etc/make.conf
and insert:
APACHE_MODULES="ssl alias log_config mime mime_magic unique_id vhost_alias threads authz_host auth_basic auth_default rewrite dir cgid"
APACHE2_MPMS="event"
then add the specific USE
flags (either /etc/make.conf
or /etc/portage/package.use
):
echo "www-servers/apache ssl threads" >> /etc/portage/package.use
echo "dev-lang/php apache2 berkdb bzip2 cgi cli crypt ctype exif fileinfo filter gd gdbm hash iconv ipv6 json mhash mysql nls phar posix readline session simplexml sqlite3 ssl threads tokenizer truetype unicode xml xpm xsl zip zlib" >> /etc/portage/package.use
echo "dev-lang/perl berkdb gdbm ithreads"
Now, all three (and their dependencies) can be compiled.
$ emerge -av www-servers/apache dev-lang/php dev-lang/perl www-apache/mod_perl
Once done, it may be also usefull to recompile all perl dependent packages (to be compiled with ithreads support) by executing:
$ perl-cleaner --modules ; perl-cleaner --force --libperl
If everything OK, you can go forward. Otherwise, it is necessary to go through the compilation errors and find which USE
flag may have brought some confusion into the compilation process.
Apache Setup
Assuming the HTTP/HTTPS availability, modify the APACHE2_OPTS
directive in /etc/conf.d/apache2
as follows:
APACHE2_OPTS="-D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5"
Modify the default VirtualHost
setup in /etc/apache2/vhosts.d/00_default_vhost.conf
. Replace the DOMAIN.TLD
with the real domain you manage:
Listen 80
NameVirtualHost *:80
DocumentRoot "/var/www/DOMAIN.TLD"
ServerName www.DOMAIN.TLD
ServerAlias DOMAIN.TLD *.DOMAIN.TLD
ErrorLog "/var/log/apache2/DOMAIN.TLD-error_log"
CustomLog "/var/log/apache2/DOMAIN.TLD-access_log" combined
Options Indexes Includes FollowSymLinks MultiViews
AllowOverride All
Order allow,deny
Allow from all
Modify the default SSL_VirtualHost
setup in /etc/apache2/vhosts.d/00_default_ssl_vhost.conf
:
Listen 443
NameVirtualHost *:443
DocumentRoot "/var/www/DOMAIN.TLD"
ServerName www.DOMAIN.TLD:443
ServerAlias *.DOMAIN.TLD:443
ServerAdmin admin@DOMAIN.TLD
ErrorLog "/var/log/apache2/DOMAIN.TLD-error_log"
CustomLog "/var/log/apache2/DOMAIN.TLD-access_log" combined
SSLEngine on
SSLProtocol all -SSLv2
SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5
# Do not forget to generate the SSL certificate for Apache or get one for free from StartSSL
SSLCertificateFile "/etc/ssl/apache2/www.crt"
SSLCertificateKeyFile "/etc/ssl/apache2/www.key"
# The following TWO only in case of a free StartSSL certificate
SSLCertificateChainFile "/etc/ssl/apache2/sub.class1.server.ca.pem"
SSLCACertificateFile /etc/ssl/apache2/ca.pem
SSLOptions +StdEnvVars
SSLOptions +StdEnvVars
BrowserMatch "MSIE [2-5]"
nokeepalive ssl-unclean-shutdown
downgrade-1.0 force-response-1.0
# The following can replace the original CustomLog above
CustomLog "/var/log/apache2/DOMAIN.TLD-ssl_request_log"
"%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x "%r" %b"
# Or this one can replace it as well, especially, for a multi-vhost Apache server and log analysis using AWStats
CustomLog /var/log/apache2/DOMAIN.TLD-access_log vhost
CustomLog /var/log/apache2/DOMAIN.TLD-ssl-request_log
"%t %h %{HTTPS}x %{SSL_PROTOCOL}x %{SSL_CIPHER}x %{SSL_CIPHER_USEKEYSIZE}x %{SSL_CLIENT_VERIFY}x "%r" %b"
Do not forget to make sure the access rights do always belong to the owner under which is the http
daemon executed (apache:apache
by default). Or simply:
$ chown apache:apache -R /var/www/DOMAIN.TLD
Now, it is highly recommended to restart the Apache server:
/etc/init.d/apache2 restart
If everything OK, you can go forward. Otherwise, it is necessary to go through the execution errors and find which statement may have caused it. It may be also useful to go through the above configured log files (CustomLog
and ErrorLog
).
Installation and AWStats Set Up
Now it’s time for AWStats and some useful packages (GeoIP location, IPv6 support, etc.)
$ echo "dev-libs/geoip perl-geoipupdate" >> /etc/portage/package.use
$ echo "www-misc/awstats ipv6 geoip apache2 vhost" >> /etc/portage/package.use
$ emerge -av dev-libs/geoip www-misc/awstats
Now, copy the example configuration file of AWStats to the one you are going to modify accordingly:
$ cp /etc/awstats/awstats.model.conf /etc/awstats/awstats.DOMAIN.TLD.conf
Modify the configuration accordingly as follows:
LogFile="/var/log/apache2/DOMAIN.TLD-access_log"
LogType=W
LogFormat=1
SiteDomain="DOMAIN.TLD"
HostAliases="localhost 127.0.0.1 REGEX[DOMAIN.TLD$]"
DNSLookup=1
DirData="/var/lib/awstats"
DirCgi="/stats/cgi-bin"
DirIcons="/stats/icon"
LoadPlugin=ipv6
LoadPlugin="geoip GEOIP_STANDARD /usr/share/GeoIP/GeoIP.dat"
Now make sure, the /var/lib/awstats
directory as owned by apache:apache
, or simply
$ chown apache:apache -R /var/lib/awstats
Once saved, the configuration needs to be executed as follows:
$ awstats.pl -config=DOMAIN.TLD -update -showdropped
If no error occurred, you can go forward. Otherwise, recheck the AWStats’s configuration file.
Finally, create an hourly script in CRON:
$ nano /etc/cron.hourly/awstats
and insert:
#!/bin/sh
cd /var/www/awstats_host_name/cgi-bin
awstats.pl -config=DOMAIN.TLD -update > /dev/null 2>&1
and make it executable
$ chmod +x /etc/cron.hourly/awstats
You should also make sure that you will parse logs shortly before every log rotation. To do it you need to add prerotate command to your log rotation configuration. Consult example showing my log rotation configuration for apache access_log files stored in /etc/logrotate.d/apache2
configuration file.
/var/log/apache2/*access_log {
daily
missingok
notifempty
rotate 365
dateext
olddir /var/log/old/apache2
sharedscripts
nocompress
nocreate
prerotate
/etc/cron.hourly/awstats
endscript
postrotate
/etc/init.d/apache2 reload > /dev/null 2>&1 || true
endscript
}
Display the AWStats
To be able to display the statistics via a web browser, it is necessary to copy the AWStats into the web browser directory and configure Apache appropriately.
Create a directory in WWW root and copy the awstats structure:
$ mkdir /var/www/DOMAIN.TLD/stats
$ cp -R /usr/share/awstats/wwwroot/* /var/www/DOMAIN.TLD/stats
Then create a simple PHP script to forward you to the AWStats page:
$ nano /var/www/DOMAIN.TLD/stats/index.php
and insert
Make sure the whole stats directory is owned by apache
, or simple:
$ chown apache:apache -R /var/www/DOMAIN.TLD/stats
Finally, the apache VirtualHost
section needs to be updated by inserting the following code to make the AWStats available via SSL and appropriate credentials (/etc/apache2/vhosts.d/00_default_ssl_vhost.conf
).
Alias /awstatsclasses "/var/www/DOMAIN.TLD/stats/classes/"
Alias /awstatscss "/var/www/DOMAIN.TLD/stats/css/"
Alias /awstatsicons "/var/www/DOMAIN.TLD/stats/icon/"
ScriptAlias /awstats "/var/www/DOMAIN.TLD/stats/cgi-bin/"
Options -Indexes FollowSymLinks
AllowOverride All
AuthType Basic
AuthName "Please, provide the appropriate credentials."
AuthUserFile /etc/awstats/.htpasswd
require valid-user
Order allow,deny
Allow from all
SSLRequireSSL
Options ExecCGI -Indexes FollowSymLinks
SetHandler cgi-script
Order allow,deny
Allow from all
SSLRequireSSL
AuthType Basic
AuthName "Please, provide the appropriate credentials."
AuthUserFile /etc/awstats/.htpasswd
require valid-user
SSLRequireSSL
Finally, open the browser and put https://DOMAIN.TLD/stats.
A big thank sending to Andrzej Kardaś.