Systems Manageability Part 7 - Log Management and Analysis

by kishi on August 07, 2007 01:57pm

Level-Set – Log Management: This section includes open-source technology directed primarily on host-based logging, log file rotation and log file analysis.  Many of these tools are very common free and open-source software tools that are distributed and preconfigured with most of the major Linux systems, including major vendors such as RedHat and Novell.

I. Logrotate

Logrotate is a very popular application utilized in a number of Linux systems, including all RedHat and SUSE based systems.  The logrotate utility typically runs periodically via cron, a task scheduling application.  The utility will read a configuration file (/etc/logrotate.conf), and archive and compress log files according to the configuration.  Administrators can configure when log files should be rotated based on age and size, and how long backlogs should be maintained.  Older archived log files can then be swapped out and replaced with newer archives.

II. Syslogd and klogd

Typical Linux systems utilize a syslog daemon to capture log messages from userspace applications and write them to text-based log files or send them to a logging host over the network.  The syslogd daemon is often accompanied by a klogd application which is designed to capture and log kernel messages.

The behavior of the syslog daemon can be configured via the /etc/syslog.conf configuration file.  All messages captured by syslog are categorized by facility and priority.  Messages can then be sent to particular log files or logging hosts, or dropped completely based on their facility and priority attributes.

List of syslog facilities and priorities.

III. Syslog-ng

The syslog-ng application aims to be an enhanced drop-in replacement for the traditional syslog daemon.  It provides many of the same features of the standard syslog daemon, but includes additional features such as advanced message filtering based on content, remote logging via UDP or TCP, and the ability to write log files to a database such as MySQL or PostgreSQL.  More recent SUSE-based systems such as SLES10 have switched to syslog-ng as the default syslog server.

IV. Viewing Logs

Most log files on a Linux system are stored in plain-text, which means they can be viewed and parsed using a number of different command-line tools.  Typical utilities such as tail, head, grep, cat, less, more, sed and awk can be used to view and filter log messages via the command line.

There are also a myriad of utilities designed to parse and view log files via a GUI or web browser.  Some utilities are even designed to handle specific log formats, such as those generated by Linux’s Netfilter firewall subsystem.

GNOME System Log Viewer

The GNOME system includes a GTK-based system log viewing application that displays system logs via the GUI.

YaST System Log Module

SUSE-based systems using YaST typically include a module called View System Log (called internally as view_anymsg).  Similar to the GNOME System Log viewer, the YaST module allows an administrator to view many of the various system logs without using the command-line.

V. Log Analysis

LogWatch

The logwatch utility is designed to parse system logs and located any entries that might indicate security threat or system failure and send an email report to a designated address.  Logwatch is distributed with RedHat Enterprise Linux systems.  The following is an excerpt from the RPM description:

“LogWatch is a customizable log analysis system. LogWatch parses through your system's logs for a given period of time and creates a report analyzing areas that you specify, in as much detail as you require. LogWatch is easy to use and claims that it will work right out of the package on almost all systems. Note that LogWatch now analyzes Samba logs.”

LogWatch is typically executed periodically via cron, a task scheduling application.  

LogCheck

The logcheck utility is a part of the Sentry Tools project that also includes portsentry, a utility designed to detect port scans.  Similar to the LogWatch utility, the software is designed to parse system log files, find log entries that may indicate security problems and send an email to a preconfigured address.  Also similar to the LogWatch utility, logcheck relies on the standard cron utility to be periodically executed.

That does it for Log Management and Analysis section. We have one last blog to go and certainly hope that you found the information we have captured for you useful. If you’re running any special toolsets or customizable scripts for log management and analysis and would like to share your experience with us, please send us your feedback and as always, THANK YOU for tuning into Port25.