Short Introduction to Log Parser Lizard

  • Article

I came across a neat tool to assist in writing LogParser scripts that’s FREE!!! As someone who isn’t really good with LogParser (me), the IntelliSense helps out a lot.

Here’s what they say at https://www.lizard-labs.net/log_parser_lizard.aspx

Short Introduction to Log Parser Lizard

Log Parser is a very powerful and versatile query software tool that provides universal query access to text-based data, such as log files, XML files, and CSV files, as well as key data sources on the Microsoft Windows operating system, such as the event log, IIS log, the registry, the file system, and the Active Directory services.

Because the command-line interface is not very intuitive, we have created Log Parser Lizard, a FREE GUI tool for managing queries and exporting results to Excel and charts. In addition, I have added input filters for RegEx and log4net input log formats (with support for regular expressions) and SQL server T-SQL queries. There are some helpful examples included in the installation package to help you start using Log Parser Lizard as your query software,web log analyzer and system log analyzer.

Pre requirements for installation are Microsoft Log Parser 2.2 and Microsoft.Net 2.0. View the Log Parser Forums for additional information on the product.

Enjoy using Log Parser and Log Parser Lizard GUI.

Impressions of our streamlined user interface

 

 

 

 

 

 

 

 

Click here to register your copy and support future development of this product. As a reward you will gain access to a nice Office 2007 like user interface

About Microsoft LogParser

Microsoft has produced its Log Parser as a query software and log analyzing command line tool. Its current release is version 2.2. It is available as a free download from Microsoft on their Log Parser page.

With LogParser you use queries written in a dialect of the SQL language to specify the operations that transform input records generated by an Input Format into output records that are delivered to an Output Format. While many GUI tools are out there that provide filters, even those that allow the user to build custom filters can't compare with the power of writing a custom SQL query in Log Parser. Log parser can accept most common log formats and output it into one of many formats of your choosing for analysis of relevant data.

Log Parser is made up of three components

  • Input Formats are generic record provider (records are equivalent to rows in a SQL table. Log Parser's built-in Input Formats can retrieve data from the following sources:
    • IIS log files (W3C, IIS, NCSA, Centralized Binary Logs, HTTP Error logs, URLScan logs, ODBC logs)
    • Windows Event Log
    • Generic XML, CSV, TSV and W3C - formatted text files (e.g. Exchange Tracking log files, Personal Firewall logfiles, Windows Media Services logfiles, FTP log files, SMTP log files, apache log files etc.)
    • Windows Registry
    • Active Directory Objects
    • File and Directory information
    • NetMon .cap capture files
    • Extended/Combined NCSA log files
    • ETW traces
    • Custom plugins (through a public COM interface)
    • In addition to these input formats Log Parser Lizard GUI have added input filters for:
      • Parsing text based log files line-by-line with Regular Expressions (RegEx Input Format)
      • Log4net and log4j file format (also with support for regular expressions). This input format is also used for parsing multiline text log files (one record is spread through one or more text lines) from various sources
      • SQL server T-SQL queries, retrieves data from SQL server tables and views. For a quick way to export some data from SQL Server to Excel file or to a chart image.
  • A SQL-Like Engine Core processes the records generated by an Input Format, using a dialect of the SQL language that includes common SQL clauses (SELECT, WHERE, GROUP BY, HAVING, ORDER BY), aggregate functions (SUM, COUNT, AVG, MAX, MIN), and a rich set of functions (e.g. SUBSTR, CASE, COALESCE, REVERSEDNS, etc.); the resulting records are then sent to an Output Format.
  • Output Formats are generic consumers of records; they can be thought of as SQL tables that receive the results of the data processing.
    Log Parser's built-in Output Formats can:
    • Write data to text files in different formats (CSV, TSV, XML, W3C, user-defined, etc.)
    • Send data to a SQL database
    • Send data to a SYSLOG server
    • Create charts and save them in either GIF or JPG image files
    • Display data to the console or to the screen

Using Log Parser

Let see an example of using Log Parser from a command line. Run windows command prompt, set current directory path to the directory wherein lies the executable "logparser.exe" (default "C:\Program Files\Log Parser 2.2") and enter the following command line:

LogParser -i:EVT -fullText:OFF -o:CSV -tabs:OFF "SELECT * INTO output.csv FROM SYSTEM"

This will save all records from System Event Log on the local system in a comma separated text file “output.csv”. This example show the Log Parser command is made up of the SQL query and the Input and Output formats options. These kinds of commands are very powerful in scripts for automatic execution and monitoring of the servers. For more information about using Log Parser from a command line, please refer to the help file or look at resources paragraph on this page.

If you are comfortable and familiar with SQL queries and command prompt commands and switches, there would be no problem using the Log Parser, but if you aren't you will have to learn basics of SQL to effectively work with this tool.

Although LogParser is fantastic, there are a few things that some users didn't like about it:

  1. Command line interface can be difficult to learn and adopt for new users.
  2. The graphing output and charts are good, but it's also a MS Office dependency. You can’t graph something on a machine without Office installed (like most of production servers)
  3. No support for a custom text file formats and multiline text log files

This is why we developed a new Log Parser GUI and share it with you and hoping to become your log reader of choice.

In short, if you keep and analyze any type of log this will make your life easier. By getting to know this tool and its capabilities you'll have better management of your systems, improved development process and have a forensic tool that you'll find new uses for with every time you use it. This is a must have for any systems engineer who needs to take a proactive approach in system monitoring.

Basics of writing a Logparser SQL Query

A basic SQL query must have, at a minimum two basic building blocks: the SELECT clause, and the FROM clause. For starters: start Log Parser Lizard, click on the “New Query” button on the toolbar, from a drop down list select “Windows Event Log” and in the Query text box in the bottom of the window write the following command:

SELECT * FROM System

The SELECT clause is used to specify which input record fields we want to appear in the output. The FROM clause is used to specify which specific data source we want the Input Format to process. Different Input Formats interpret the value of the FROM clause in different ways; for instance, the EVT Input Format requires the value of the FROM clause to be the name of a Windows Event Log, which in our example is the "System" Event Log.

The special "*" wildcard after a SELECT keyword means "all the fields" (like in standard SQL). Most of the times, an output of all of the fields of the log records might not be desired. You might only want to see only the fields that are of your interest. To accomplish this, instead of the "*" wildcard in the SELECT clause, you will have to write a comma-separated list of the names of the fields you wish to be displayed.

SELECT TimeGenerated, EventTypeName, SourceName FROM System.

The Log Parser SQL-Like language also supports a wide variety of functions, including arithmetical functions (e.g. ADD, SUB, MUL, DIV, MOD, QUANTIZE, etc.), string manipulation functions (e.g. SUBSTR, STRCAT, STRLEN, EXTRACT_TOKEN, etc.), and timestamp manipulation functions (e.g. TO_DATE, TO_TIME, TO_UTCTIME, etc.). Functions can also appear as arguments of other functions.

SELECT TO_DATE(TimeGenerated), TO_UPPERCASE( EXTRACT_TOKEN(EventTypeName, 0, ' ') ), SourceName FROM System

То change the name of a field-expression in the SELECT clause by using an alias you can use the AS keyword followed by the new name of the field.

SELECT TO_DATE(TimeGenerated) AS DateGenerated, TO_UPPERCASE( EXTRACT_TOKEN(EventTypeName, 0, ' ') ) AS TypeName, SourceName FROM System

When retrieving data from an Input Format, it is often needed to filter out unneeded records and only keep those that match specific criteria. To accomplish this task, you can use another basic building block of the Log Parser SQL language: the WHERE clause which is used to specify a Boolean expression that must be satisfied by an input record for that record to be listed in the output. Input records that do not satisfy the condition will be discarded. Conditions specified in the WHERE clause can be more complex, making use of comparison operators (such as ">", "<=", "<>", "LIKE", "BETWEEN", etc.) and boolean operators (such as "AND", "OR", "NOT"). The WHERE clause must immediately follow the FROM clause.

SELECT TimeGenerated, EventTypeName, SourceName FROM System WHERE ( SourceName = 'Service Control Manager' AND EventID >= 7024)

The ORDER BY clause can be used to specify that the output records should be sorted according to the values of selected fields. By default, output records are sorted according to ascending values. We can change the sort direction by appending the DESC (for descending) or ASC (for ascending) keywords to the ORDER BY clause.

SELECT SourceName, EventID, TimeGenerated FROM System ORDER BY TimeGenerated

Sometimes we might need to aggregate multiple input records together and perform some operation on groups of input records. To accomplish this, the Log Parser SQL like language has a set of aggregate functions (also referred to as "SQL functions") that can be used to perform basic calculations on multiple records. These functions include SUM, COUNT, MAX, MIN, and AVG. The GROUP BY clause is used to specify which fields we want the group subdivision to be based on. After the input records have been divided into these groups, all the aggregate functions in the SELECT clause will be calculated separately on each of these groups, and the query will return an output record for each group created.

SELECT EventTypeName, Count(*) FROM system GROUP BY EventTypeName

For filtering results from groups you can use the HAVING clause. The HAVING clause works just like the WHERE clause, with the only difference being that the HAVING clause is evaluated after groups have been created, which makes it possible for the HAVING clause to specify aggregate functions.

SELECT EventTypeName, Count(*) from system group by EventTypeName HAVING EventTypeName =’Error event'

The DISTINCT keyword is used to indicate that the output of a query should consist of unique records. Duplicate output records are discarded. It is also possible to use the DISTINCT keyword inside the COUNT aggregate function, in order to retrieve the total number of different values appearing in the data.

SELECT DISTINCT SourceName from System

SELECT COUNT( DISTINCT SourceName) from System

Use the TOP keyword in the SELECT clause to return only a few records at the top of the ordered output.

SELECT TOP 10 SourceName, Count(*) as Total FROM System GROUP BY SourceName ORDER BY Total DESC

These are simple queries, but they are good example that this log tool is more powerful for analyzing syslog events than any other event log viewer. For more samples, you can always look in examples provided with the program. They don’t all work out-of-a-box but can be very helpful.

There are many additional resources for learning about and using Log Parser on the Internet. Please check the following links.

  1. Logparser Download
  2. Logparser Forums
  3. Using Log Parser Lizard with SharePoint
  4. Examples (SQL) queries for IIS Analysis
  5. Under the hood of Logparser
  6. Microsoft Script Center page - The Microsoft logparser overview page
  7. Forensic IIS log exploration with LogParser
  8. Using the Logparser Utility to Analyze Exchange/IIS Logs
  9. LogParser 2.2 and ASP.NET
  10. Auditing the Event Logs
  11. Log Parser Plus
  12. Aggressive Virus Defense

Using Regular Expression and log4net input formats with Log Parser Lizard.

Both these formats are based on regular expressions to parse the log lines but the difference is that RegEx input format is parsing the log files line by line (if there is some line that doesn’t match the regular expression, it will be marked as error). Log4Net input format also uses regular expressions to parse the log file but the logged messages is spread in more than one line. Field “Full Message” is what goes after the first line that matches the regular expression and “Exception” field isn’t null only if the “Full Message” begins with “Exception:” word (this was made for my own needs but maybe someone else will find it useful). Regex and Log4Net input formats are not the ultimate solution for every text-log-file-format but they are flexible enough to meet most of your needs.

Here is a step by step guide for using RegExp and log4net input formats:

  1. In Log Parser Lizard install path create XML file which defines regular expression and its fields and their data types. Something like this:

    <?xml version="1.0" encoding="UTF-8"?>
    <config xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation= "C:\src\LogParserCSWebServiceInputFormat\LogParserRegexInputFormat.xsd">
    <regex>^(?&lt;DateTime&gt;(?:\d{4})-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3})\s+(?&lt;ThreadID&gt;\d*)\s*\[(?&lt;ProcessID&gt;\d+)\]\s+(?&lt;LogType&gt;\w+)\s+(?&lt;Loger&gt;\w+)\s+-\s+(?&lt;Message&gt;.*)$</regex>
    <fields>
    <field name="DateTime" type="Timestamp" format="yyyy-MM-dd HH:mm:ss,fff"/>
    <field name="ThreadID" type="Integer"/>
    <field name="ProcessID" type="Integer"/>
    <field name="LogType" type="String"/>
    <field name="Loger" type="String"/>
    <field name="Message" type="String"/>
    </fields>
    </config>

    For writing regular expression you can use Expresso, excellent and free tool for building regular expressions and for editing XML files you can use XML Notepad from Microsoft.

  2. Create a new query

  3. From a drop down list select “Regular expression input format” or “log4net input format”

  4. Click on properties button (next to the drop down list) and set “config file” property to configuration file name that you have created in step 1.

  5. Test created the query against your text file (ex. select * from c:\mylog.txt) and if you have some troubles try to fix the config file.

You can look at log4net examples provided in installation directory of LogParser Lizard. And remember, the difference between RegEx input format and log4net input format is that RegEx log files are read as one record per text line. In log4net log files, one record can be in more text lines (for instance when exception is logged).

If you like Log Parser Lizard, please consider helping to make Lizard Labs better.