Introducing the Netlogon Parser (v1.0.1) for Message Analyzer 1.1

Hi all! Brandon Wilson here again, and this time I will be giving you an introduction for the new Netlogon parser for Message Analyzer.

Do you have authentication issues? Are you unable to contact the domain? Wondering what’s going on and tired of pulling your hair out? Then the Netlogon parser may be able to help you! This parser was created to help you troubleshoot your own issues relating to Netlogon issues with a goal of self-diagnosis, but more importantly, to try to translate all of this information into plain English for you. This is something we will continue to expand on, and suggestions are always welcome! Look for more cool features for self-diagnosis in the future as well!

A special thank you goes out to all of those who helped me with the development of this parser: Paul Long, Yiming Cao, Meng Jin, Serge Mera, and the entire Message Analyzer team; along with countless Premier Field Engineers and customers who provided data for testing and analysis to help make the Netlogon parser a useful tool!

As with all of our parsers, this is provided as is, however there are a few different feedback mechanisms for you to use. You can use the feedback link in Message Analyzer (see the bottom of this blog for a highlighted screenshot for that; but every screenshot shows the feedback button), reach out in the forums (see reference links at the bottom of the blog), or to the MANetlogon@microsoft.comemail address where you can submit ideas and any problems or buggy behavior you may run across.

You can also read up more on Message Analyzer itself at http://blogs.technet.com/MessageAnalyzer!

In this introduction, I will walk you through the following:

The anatomy of the Netlogon parser for Message Analyzer

Opening a Netlogon log file in Message Analyzer

What does Message Analyzer show me?

What if I want to see the log in its native format?

Common problems the Netlogon parser finds for you!

Reference links

 

 

The anatomy of the Netlogon parser for Message Analyzer

It’s important to talk about the anatomy of the parser because this isn’t your average one-to-five liner. It has taken quite some time for the parser to come this far, and there will be many updates in the future to enhance the ease of use and blatantly point out problems. The Netlogon parser is made available for its public debut in Message Analyzer v1.1, which can be downloaded from http://www.microsoft.com/en-us/download/details.aspx?id=44226. You MUST have v1.1 in order to make use of the parser. It does not function in v1.0. So if you are running Message Analyzer v1.0, be sure to update as soon as you can. Outside of the Netlogon parser availability, there were numerous enhancements to Message Analyzer that make this a must have update to the tool!

As of today (9/2014), the parser has the capability to break out authentication calls for NTLM, Digest authentication, Kerberos PAC validation, and other authentication types and puts them into a nice little “group” for you to review. Due to the nature of the way the Netlogon files are written though, sometimes these groupings can be a little thrown off (aka disordered). In addition, the parser will also group together DNS calls, MAILSLOT calls (think LDAP pings here), performance counter information, critical messages, missing site/subnet associations, service initialization, and any other undefined logon messages not captured by other parsing mechanisms (there’s more, but I’m sure you get the point by now). You also have the ability to hide operations, which will give you a flat view of the file in the same format as you would see in the Netlogon log by looking at it directly via another mechanism such as notepad.exe.

And finally, and we’ll talk more about this shortly, the parser does do some identification of known issues that can cause you a headache and tells you flat out that you have a problem. At this time, those self-diagnosis mechanisms are identification of MaxConcurrentApi issues and winsock/RPC buffer (port) exhaustion. Trust me; it’s a really cool feature!

There are some known issues and limitations to speak about here too that I want to mention…I will work to correct all of these items in future versions going forward. The goal was to get you the ability to troubleshoot ninja style as soon as possible, and the Netlogon parser will most definitely help you do that in its current state!

Issues:

1. Some wording specific to Kerberos PAC validation may not directly reference PAC validation in the summary of the frame.

a. For now, just know that ANY reference to Kerberos authentication is referring directly to Kerberos PAC validation.

2. Some authentication calls may be grouped incorrectly and show incorrect authentication duration intervals.

a. This is something that reared its ugly head when analyzing some real world data internally after initial testing. This was noticed predominantly with Exchange and SharePoint servers, but may happen on other machine types/roles as well. It’s an intermittent issue that occurs when there are discontinuous authentications at the beginning and end of the Netlogon log for the same username from the same machine (whether proxied or interactive authentication). For any of you whom have dug your head into the Netlogon logs, you have probably seen how the beginning of the log file may contain some authentication “return” calls that are rolled over from the previous Netlogon.log file…and there is the cause.

Limitations:

1. At this time, only authentication calls are grouped together into individual groupings. 1 user authentication = 1 authentication grouping

2. DNS, MAILSLOT, CRITICAL, PERF, No client site, CHANGELOG, INIT, DOMAIN, MSA (managed service account), SESSION, MISC, and any other non-identifiable lines are grouped into a single large grouping.

a. Over time, these individual areas will be broken down into more usable forms where applicable.

 

 

Opening a Netlogon log file in Message Analyzer

So now that we’ve talked a bit about the anatomy of the parser, let’s take a look at how to open a Netlogon log file in Message Analyzer. It’s a fairly straight-forward process, and these steps are applicable to not only Netlogon log files, but also opening any text based log file that Message Analyzer has a parser available for; it’s simply a matter of choosing your parser selection from the dropdown, as you will see.

So let’s walk through this nice and slow…

1. Open Message Analyzer

a. Notice the version number at the bottom right corner of the Message Analyzer window. Version 4.0.7056.0 is equal to Message Analyzer v1.1

clip_image002

2. There are two primary methods to open a Netlogon log (or other text log). You can drag and drop the file, or you can use the File menu (File|Quick Open).

a. There will be a small delay the first time you open a text log based file due to Message Analyzer analyzing the available parsers on the first run before you can make your selections. Opening additional text log files while Message Analyzer is open should not experience this delay.

3. Once the parsers available are analyzed, you will be presented with the below window. From this window, select the appropriate text log configuration parser (in this case, Netlogon), then click the Start button.

clip_image004

clip_image006

4. Message Analyzer will now begin parsing the log file. Depending on the log file size, this may take some time. The default log file size of 20MB typically will take between 8-10 minutes, depending on the number and type of messages included in the file.

clip_image008

5. When parsing is complete, the bottom left corner of Message Analyzer will say “Ready”, as seen in the screenshot above.

 

 

What does Message Analyzer show me?

First, let’s talk about the columns. We won’t go into depth on this today, as that is for the next blog (Troubleshooting Problems with the Netlogon Parser (v1.0.1) for Message Analyzer)…but suffice it to say, the columns can be adjusted to reflect whatever view you desire. By default, these columns are MessageNumber, DiagnosisTypes (which is an interesting icon that is a mix of a red X, yellow exclamation, and blue informational bubble), TimeElapsed, Source, Destination, Module, Summary, and Type.

If you want to streamline your view, you don’t need the Source, Destination, Module, or Type columns for reviewing the logs. You can remove columns by right clicking any of the desired columns and selecting “Remove”; or you can add columns by selecting the “Choose Columns…” option, which opens a nice little tree view on the right side of the screen as seen below.

clip_image010

Now let’s discuss what the columns of use for the Netlogon parser mean….

Column name

Column Purpose

MessageNumber

This column typically contains the line number within the log file and is useful for tracking specific calls in the Netlogon log directly if you desire to do so (this really shouldn’t be necessary).

The message number is not always an exact match to the line number in the log file though, as any lines that are not parsed are not counted/tallied against the MessageNumber column. Operational “groupings” will appear in blue as seen in the screenshot above with a “+” box next to them to be expanded. “Ungrouped” lines that have been parsed will appear with a green icon and will not be expandable.

DiagnosticTypes

This column is denoted by the jazzy looking icon that is a combination of an informational bubble under a diagnostic error under an exclamation warning; an icon so unique the column needs no labeled name… This column contains any diagnostic messages that may occur in relation to parsing the log file or creating operational groupings.

In future versions of the Netlogon parser, this column will also contain further support information, such as hotfix references, steps to take based on specific problems, and references to resources to help you troubleshoot the problem on your own!

TimeStamp

This column contains the timestamp of the specific entry being parsed. In the case of operationally grouped messages, the timestamp will be the time defined on the initial entry to the operational grouping. In the case of an authentication, this means it would be when the authentication was “entered”.

TimeElapsed

This column, as its name implies, contains the time elapsed between the first entry and the last entry in the operational grouping. In regards to the Netlogon parser, if this column is blank, it means an operation occurred within the same second. In the above screenshot you can see 4 NTLM authentication attempts, each with a blank TimeElapsed; this indicates the authentication “entered” and “returned” in the same second. Since as of this writing, the Netlogon log does not currently track milliseconds, the smallest increment that can be reported in the TimeElapsed column is 1 second and beyond.

Summary

This column contains the pertinent information regarding the line parsed in the Netlogon log file. Wherever possible, the Summary column contains a simplified English translation for the information in the log to simplify the review of Netlogon logs greatly and take some of the “what in the world does that mean!?” out of your life!

 

Now that I dug a bit more into the columns that I planned for this article, let me get to the point for this section. What will you see…?

Detection Point

Detection Description

Authentication attempts (NTLM, Kerberos PAC validation, etc)

This is actually a culmination of a large number of parsers that include detection for all authentication types that utilize the Netlogon RPC channel for both legacy (Windows 7/2008 R2 and below) and current (Windows 8/2012 and above) Windows operating systems. This also includes detection of Kerberos PAC validation attempts. This is useful for identifying specific errors authentication attempts may be encountering.

Missing site/subnet associations

Identifies NO_CLIENT_SITE entries in the Netlogon logs of domain controllers and groups them together for easier analysis. This is very useful for detecting gaps in your environment where you may have straying clients and correcting your environment.

Change notifications

Identifies CHANGELOG notifications. This will typically not be used for troubleshooting purposes.

Mailslot ping messages

Identifies MAILSLOT messages (LDAP pings). This is useful for troubleshooting issues contacting domain controllers.

Domain related messages

Contains DOMAIN messages relating to trusted domains, computer name updates, etc. In many cases, this may not be directly applicable to troubleshooting, however in certain circumstances it may be of interest.

Critical Netlogon messages

Contains CRITICAL lines not captured by other parsing mechanisms. This can include some errors and debug information that is not captured by the MaxConcurrentApi or Winsock/RPC buffer (port) exhaustion self-diagnosis parsers. This may be of interest during troubleshooting to provide clues to the cause of a problem.

Session establishment

Contains SESSION establishment calls with the domain that may include machine password changes, domain controller discovery, and domain information among other things. This may be useful in troubleshooting; however in many cases the information provided will also be provided through other detection points.

MISC lines

Contains information items such as Netlogon events being logged, DsGetDcName calls, user/machine account status (ie; disabled), etc. This may be useful in troubleshooting some issues.

Netlogon performance counter information

Contains performance counter information for Netlogon. This can include performance data for session establishment, the number of authentication timeouts that have occurred, the average semaphore hold times, and other Netlogon performance counter information. This information is useful in troubleshooting for an at a glance view of average hold times before authentication occurs and how large of an impact MaxConcurrentApi issues may be having.

DNS call information

Contains DNS specific calls such as record scavenging and SRV record registration. This is useful for troubleshooting if you have DNS related issues, such as missing SRV records in expected locations in DNS.

Site related information

Contains SITE information, such as GC/DC listings for sites, site information derived from the local cache (DynamicSiteName), and site auto-coverage information. This can be useful for troubleshooting when you have issues with automatic site coverage or “site straying” by clients or servers.

Logon messages (typically non-authentication)

Identifies any lines tied to the LOGON parameter of Netlogon that are not specific to the “entered” or “return” calls. This can include the Netlogon service “picking” a domain to proxy authentication requests for and domain trust identification/enumeration. This can be useful for troubleshooting if there are trust issues, but the information can also be provided by other detection points in the Netlogon parser.

Service initialization

Contains the startup/initialization of the Netlogon service. This is useful for determining configuration parameters for the Netlogon service.

Managed Service Accounts

Identifies actions for lookups and usage of managed service accounts. This is useful for troubleshooting issues with managed service accounts in Windows 7/2008 R2 and up.

MaxConcurrentApi

Identifies MaxConcurrentApi issues on the local machine where the Netlogon log was collected from.

Winsock/RPC (port) exhaustion

Identifies RPC/winsock port exhaustion issues present in the Netlogon log.

Other non-specified lines (lines without headers, and other unparsed or unformatted lines)

Other lines not captured by existing parsers in the Netlogon parser will be thrown into a collection of unspecified lines labeled “The lines grouped here are typically not useful for troubleshooting!”.

A table is great, but seeing what this looks like in Message Analyzer is better! Below I will show you examples of what’s covered in the table above with 2 exceptions that merit their own section, and those are MaxConcurrentApi and Winsock/RPC buffer (port) exhaustion, which I will show you an example of later in this blog. Other self-diagnosis items similar to MaxConcurrentApi and RPC port exhaustion will also be added to the Netlogon parser in the future!

 
Detection point: Authentication attempts (NTLM, Kerberos PAC, Digest, etc)

Detection description: A culmination of a large number of parsers that include detection for all authentication types that utilize the Netlogon RPC channel for both legacy (Windows 7/2008 R2 and below) and current (Windows 8/2012 and above) Windows operating systems. This also includes detection of Kerberos PAC validation attempts. This is useful for identifying specific errors authentication attempts may be encountering.

clip_image012

If we expand these authentication attempts out, we can see the details of the authentication attempt; the “entered” and “returns” calls. Expanding any of the detection points will reveal more information to you. From this point forward for these samples, I will show you the expanded views.

clip_image014

Notes on this detection point:

1. Kerberos PAC validation (not shown in the screenshots above) does not contain usernames however it does show the domain name as well as any proxying/authenticating machine, so as a result the information gleaned from these validation requests is limited to success or failure determination and cannot be used to trend specific user authentication attempts.

Detection point: Missing site/subnet associations

Detection description: Identifies NO_CLIENT_SITE entries in the Netlogon logs of domain controllers and groups them together for easier analysis. This is very useful for detecting gaps in your environment where you may have straying clients and correcting your environment.

clip_image016

 
Detection point: Change notifications

Detection description: Identifies CHANGELOG notifications. This will typically not be used for troubleshooting purposes.

clip_image018

 
Detection point: Mailslot ping messages

Detection description: Identifies MAILSLOT messages (LDAP pings). This is useful for troubleshooting issues contacting domain controllers.

clip_image020

 
Detection point: Domain related messages

Detection description: Contains DOMAIN messages relating to trusted domains, computer name updates, etc. In many cases, this may not be directly applicable to troubleshooting, however in certain circumstances it may be of interest.

clip_image022

 
Detection point: Critical Netlogon messages

Detection description: Contains CRITICAL lines not captured by other parsing mechanisms. This can include some errors and debug information that is not captured by the MaxConcurrentApi or Winsock/RPC buffer (port) exhaustion self-diagnosis parsers. This may be of interest during troubleshooting to provide clues to the cause of a problem.

clip_image024

Notes on this detection point:

1. The CRITICAL message operational grouping will exclude MaxConcurrentApi and Winsock/RPC buffer (port) exhaustion related lines. These will be included in the applicable self-diagnosis area.

 
Detection point: Session establishment

Detection description: Contains SESSION establishment calls with the domain that may include machine password changes, domain controller discovery, and domain information among other things. This may be useful in troubleshooting; however in many cases the information provided will also be provided through other detection points.

clip_image026

 
Detection point: MISC lines

Detection description: Contains information items such as Netlogon events being logged, DsGetDcName calls, user/machine account status (ie; disabled), etc. This may be useful in troubleshooting some issues.

clip_image028

 
Detection point: Netlogon performance counter information

Detection description: Contains performance counter information for Netlogon. This can include performance data for session establishment, the number of authentication timeouts that have occurred, the average semaphore hold times, and other Netlogon performance counter information. This information is useful in troubleshooting for an at a glance view of average hold times before authentication occurs and how large of an impact MaxConcurrentApi issues may be having.

clip_image030

 
Detection point: DNS call information

Detection description: Contains DNS specific calls such as record scavenging and SRV record registration. This is useful for troubleshooting if you have DNS related issues, such as missing SRV records in expected locations in DNS.

clip_image032

 
Detection point: Site related information

Detection description: Contains SITE information, such as GC/DC listings for sites, site information derived from the local cache (DynamicSiteName), and site auto-coverage information. This can be useful for troubleshooting when you have issues with automatic site coverage or “site straying” by clients or servers.

clip_image034

 
Detection point: Logon messages (typically non-authentication)

Detection description: Identifies any lines tied to the LOGON parameter of Netlogon that are not specific to the “entered” or “return” calls. This can include the Netlogon service “picking” a domain to proxy authentication requests for and domain trust identification/enumeration. This can be useful for troubleshooting if there are trust issues, but the information can also be provided by other detection points in the Netlogon parser.

clip_image036

clip_image038

Notes on this detection point:

1. Notice in the first screenshot that there is one grouped area, but trusted domain detection has, in part, been excluded from the grouping. Specifically, the name and the GUID. This was done to allow an at a glance view of trusted domain information, which can be handy if a trust has been added or removed and you are evaluating the status.

 
Detection point: Service initialization

Detection description: Contains the startup/initialization of the Netlogon service. This is useful for determining configuration parameters for the Netlogon service.

clip_image040

 
Detection point: Managed Service Accounts

Detection description: Identifies actions for lookups and usage of managed service accounts. This is useful for troubleshooting issues with managed service accounts in Windows 7/2008 R2 and up.

clip_image042

 
Detection point: Other non-specified lines (lines without headers, and other unparsed or non-formatted lines)

Detection description: Other lines not captured by existing parsers in the Netlogon parser will be thrown into a collection of unspecified lines labeled “The lines grouped here are typically not useful for troubleshooting!”.

clip_image044

Notes on this detection point:

1. These lines are random and include lines without headers, as well as capturing some lines that might exhibit some “buggy” behavior that may combine multiple lines.

 

 

What if I want to see the log in its native format?

Let’s say you want to see the log as it sits natively; well, there is a way, however you may find it not quite as useful as utilizing the operations when it comes to Netlogon log reviews in Message Analyzer. All you need to do is click the “Hide operations” button in the ribbon.

clip_image046

And then once you do that, it will change the view you see above to something similar to this:

clip_image048

Notice the message numbers are out of sequence. If you are hiding the operational groupings, then to get the closest as possible view of the actual Netlogon log (as seen via notepad.exe), click the MessageNumber column to order the lines and get a view similar to this:

clip_image050

 

 

Common problems the Netlogon parser finds for you!

Now, there are a couple of common headaches that we wanted to help you figure out without doing much searching. Hopefully this will lessen your headaches for you…

The “self-diagnosis” detections added are for MaxConcurrentApi issues, as well and winsock/RPC buffer (port) exhaustion.

And as promised, here are some examples of the detection mechanisms from the “Detection Points” table that point you straight to your problem!

Now keep in mind, this is your problem, but the solution may not be as straight forward!

Detection point: MaxConcurrentApi

Detection description: Identifies MaxConcurrentApi issues on the local machine where the Netlogon log was collected from.

clip_image052

 
Detection point: Winsock/RPC buffer (Port) exhaustion

Detection description: Identifies RPC/winsock port exhaustion issues present in the Netlogon log.

clip_image054

Notes on this detection point:

1. This is an easy way to identify RPC port exhaustion issues even if authentication failures aren’t being reported! As you all know, RPC port exhaustion will cause some massive headaches, and isn’t always that easy to find. Well now, if you happen to have Netlogon logging enabled, then this should make it much easier to identify!

 

 

Reference links

Message Analyzer v1.1 download

http://www.microsoft.com/en-us/download/details.aspx?id=44226

Quick Reference: Troubleshooting Netlogon Error Codes (By: Brandon Wilson)

http://blogs.technet.com/b/askpfeplat/archive/2013/01/28/quick-reference-troubleshooting-netlogon-error-codes.aspx

Quick Reference: Troubleshooting, Diagnosing, and Tuning MaxConcurrentApi Issues (By: Brandon Wilson)

http://blogs.technet.com/b/askpfeplat/archive/2014/01/13/quick-reference-troubleshooting-diagnosing-and-tuning-maxconcurrentapi-issues.aspx

Message Analyzer Forum

http://social.technet.microsoft.com/Forums/en-US/home?forum=messageanalyzer

Message Analyzer blog site

http://blogs.technet.com/MessageAnalyzer

Just to recap; please send us any suggestions or problems you identify through the Message Analyzer forum, via email to MANetlogon@microsoft.com, or using the integrated feedback button in Message Analyzer as seen below!

clip_image056

 

Thanks, and talk to you folks next time!

-Brandon “Long Winded” Wilson