by ·0 Comments

One of my most used means of monitoring what’s going on at a remote Linux server, is looking at the log lines that each daemon or client program writes to the disk. Using this technique you can see a real time view about what is happening and find out if there are any outstanding errors or optimizations in your configuration. I am going to show how I tackle this task and what tricks I use to make tons of lines of dull text more readable.

Continue reading

by ·3 Comments
GoAccess - web log analyzer

GoAccess - web log analyzer

Description

GoAccess is a very handy tool, that runs in the terminal on most *nix Systems (e.g. Linux, Mac OS X, BSD etc.) and displays relatime data gathered from Apache’s access logfile.

It offers a very structured display with different “modules” that break down the parsed access.log data into sections:

  • General Statistics – Information analyzed from log file – Unique totals
  • Unique visitors per day – Including spiders
  • Requested files (Page-URL)
  • Requested static files (images, stylesheets etc.)
  • Referrer URLs
  • HTTP 404 Not Found response code
  • Operating Systems
  • Browsers
  • Hosts
  • HTTP Status Codes
  • Top Referring Sites
  • Top key phrases used on Google

Each module has a detail view that can be opened by pressing “o”. The detail view gives you a scrollable list with more information on the currently selected data.

Installation

Since I couldn’t find a package for GoAccess in Debian lenny’s apt sources, I took the manual compile route. But the installation is still really easy and fast.

  • Make sure you’ve got the GLIB2, ncurses & GeoIP packages installed. 
    aptitude install libglib2.0-dev, libncurses5-dev, libgeoip-dev
  • Download the latest GoAccess tarball from http://goaccess.prosoftcorp.com/download and uncompress it on your webserver.
    At the time of writing the current version was GoAccess 0.3.3.
  • Go into the newly created folder where the uncompressed files are and type “./configure”
    (include the dot and slash before the command!)
  • If all tests pass, you should see those three lines at the end of the output: 
    config.status: creating Makefile
config.status: creating config.h
config.status: executing depfiles commands
  • Type “make” to start the compile process.
  • When the program is readily compiled, it is time to install the binaries on your system.
    For that gain root access (or use “sudo”) and type “make install“.

Running GoAccess

Once you are done installing you can start the program by typing in “goaccess” on the command prompt and pressing enter. If you don’t add any command line options, GoAccess will print a few lines of help text.

GoAccess - 0.3.3

Usage: goaccess [ -b ][ -s ][ -e IP_ADDRESS][ -f log_file ]

The following options can also be supplied to the command:

  -f  - Path to input log
  -b  - Enable total bandwidth consumption. To achieve
        faster parsing, do not enable this flag.
  -s  - Enable/report HTTP status codes. To avoid overhead
        while parsing, this has been disabled by default.
  -e  - Exclude an IP from being counted under the HOST
        module. This has been disabled by default. 

For more details visit: http://goaccess.prosoftcorp.com

If you want all the features you can start it with:

goaccess -b -s -f /var/log/apache/access.log

Give it some time to parse the logfile (could take a very long time, depending on your logfile size), then you should see the mainscreen with live updating data and can start to use GoAccess.

GoAccess - Operating Systems module report

GoAccess - Operating Systems module report

by ·1 Comment

Log4Twitter is very similar to an idea Herbert an me had a couple of years ago, to use a Jabber messaging framework to get distributed logging and alerts from several servers/applications.
It is a Java class that allows you to “log” to Twitter. This would easily allow you to set up an application that sends messages to a private Twitter Account that you can subscribe to. Retrieving those messages should then be possible by IM or even SMS, seems like a perfect set up for me.

Using Log4Twitter is as easy as the following:

Setting Classpath
Add log4twitter-1.0.jar to your application’s classpath.
Note that log4twitter-1.0.jar has to be loaded by the same classloader that will load the logging framework.

Setting Logger
Edit your logging framework’s configuration to enable Log4Twitter.
The fully qualified class name of log4twitter is “log4twitter.FRAMEWORK_NAME.TwitterAppender”.

See the Log4Twitter page for some examples.

Now who’s got the time to code a Linux syslogd replacement or supplement, so I can receive important log messages via Twitter? 🙂