New Features in the Netlogon Parser (v1.1.4) for Message Analyzer

Hi all! Brandon Wilson here again talking to you about some new features added in the updated Netlogon parser (v1.1.4) for Message Analyzer. This parser was developed with fixes that are specific to Message Analyzer 1.2; however this version is backwards compatible with Message Analyzer 1.1 (with some caveats that are explained below). As such, Message Analyzer 1.2 will be the focus that I speak about in this blog…

The interface for Message Analyzer v1.2 has changed a bit, and I’ll try to touch on the areas pertinent to the Netlogon parser here, but outside of the GUI changes the pertinent methods for troubleshooting and parsing using the Netlogon parser are the same as we’ve went over in the previous blog posts. If you haven’t reviewed the previous blog posts, these are essential reading for proper usage of the Netlogon parser, and you should review the Introduction blog and the Troubleshooting Basics for the Netlogon Parser for Message Analyzer blog as pre-requisites, which cover some of the main features and troubleshooting techniques that were available in v1.0.1 of the Netlogon parser (the initial public release). It would also be a good idea to get a handle on Netlogon error codes from the Quick Reference: Troubleshooting Netlogon Error Codes blog and troubleshooting MaxConcurrentApi issues in the Quick Reference: Troubleshooting, Diagnosing, and Tuning MaxConcurrentApi Issues blog, both of which can help guide you to proper troubleshooting and root cause analysis for Netlogon related issues.

I talk about versions a lot when it comes to the Netlogon parser but in reality, as of the date of this post, they are all named Netlogon.config, and the only way for you to truly know the version you have is to open the file and look at the version table at the top. The initial version released to the public, as you can probably tell by the previous blog posts, was v1.0.1. Today we are talking about v1.1.4, which has had many new features added to help you understand and diagnose Netlogon issues. This updated parser is provided in the installation package for Message Analyzer 1.2. If for some reason you are unable to upgrade to Message Analyzer 1.2, then the Netlogon parser v1.1.4 is available to download in this blog.

As with all of our parsers, this is provided as is, however there are a few different feedback mechanisms for you to use. You can use the feedback link/button in Message Analyzer, reach out in the Message Analyzer forum, you can send an email to MANetlogon@microsoft.com where you can submit ideas and any problems or buggy behavior you may run across, or of course you can leave a comment here. I highly recommend reaching out through one of the available methods to provide your suggestions for additional detections to add, problems you encounter, etc.

You can also read up more on Message Analyzer itself at https://blogs.technet.com/MessageAnalyzer

In this new features walkthrough, we will cover the following:

GUI changes in Message Analyzer 1.2

Updates and New Detection Features in the Netlogon Parser v1.1.4

Known issues

How to update the Netlogon parser manually if you are unable to upgrade to Message Analyzer 1.2

Reference links

GUI changes in Message Analyzer 1.2

The first thing you might notice when opening Message Analyzer 1.2 is that the view has changed and now look like the below screenshot. It’s a bit different than you’re used to, but it’s actually a little easier to work with IMO, but you can still customize it to how you like it.

The major components that tie to the Netlogon parser that changed are the view filter and the “hide operations” button. The view filter is now found towards the top right corner, which is much easier than having to drill through the tabs on the bottom right to get to the view filter. It’s there, it’s ready to use, and you’re probably going to use it! In the below screenshot, I’ve highlighted the “View Filter” area in a green square.

The second component that you may use with the Netlogon parser that has gotten a little hidden (ironically) is the “Hide Operations” button. In Message Analyzer 1.2, this is now found in the bottom right corner in the Viewpoints tab. In there, you will see an Operations dropdown button that includes the options: Show, Hide, and Exclusively (all of which are pretty self-explanatory). To make this a bit easier, I highlighted this area in a green square in the below screenshot:

Updates and New Detection Features in the Netlogon Parser v1.1.4

As I mentioned, there are numerous new features and updates added to v1.1.4 of the Netlogon parser. Before I show you the guts of the new features, I want to give you an idea of the updates:

1. Significantly improved performance!

2. Modified wording in the summary to better translate the lines being parsed

3. Added some additional syntaxes for NO_CLIENT_SITE detection

4. Corrected issue with the timestamp being off by minus UTC time

5. Changed the hard coded year in the output to 1601 (I’ll explain why…)

6. Added “sysvol not ready” detection

7. Added detection of specific error codes to detect issues without reviewing individual authentication calls

8. Combined SChannel and Kerberos PAC validation calls into their own groupings

So, let’s go over these additions in a bit more detail:

1. Significantly improved performance!

Performance with the parser has been improved by more than 2x! I won’t get into the nitty-gritty details of how, because that’s more of a development talk and doesn’t really serve a purpose here, but suffice it to say, the performance has been improved dramatically! Stay tuned though because more performance enhancements and features will be coming out in the future. Keep the input coming, so we can keep helping you achieve your goals!

2. Modified wording in the summary to better translate the lines being parsed

There was some confusion in some of the wording around Kerberos PAC validation that external users noticed and brought to my attention (thank you for that!), as well as some other random wording that needed some improvements to be more straightforward. So as a result, this wording was updated to reflect the proper verbiage to ensure consistency. A small change, but it makes a difference when you’re reading your log output!

3. Added some additional syntaxes for NO_CLIENT_SITE detection

There was an issue identified where certain unexpected syntaxes for NO_CLIENT_SITE detection on domain controllers came up. The Netlogon parser v1.1.4 has these syntaxes added to ensure proper detection with all known formats of these lines in the Netlogon log.

4. Corrected issue with the timestamp being off by minus UTC time

Beginning in MA 1.1, the timestamp would show the year, followed by the time. However, the time being shown wasn’t the time actually seen in the Netlogon log. Instead, it was the time in the Netlogon log with the current UTC time difference subtracted (-4 hrs for example). This resulted in a little confusion from time to time when reviewing log files. As a result, the parser has been updated to reflect the timestamp shown in the Netlogon log to eliminate confusion. NOTE: This issue is only resolved in Message Analyzer 1.2. If you are using the manual update method to update the Netlogon parser to v1.1.4 to use in Message Analyzer 1.1, this timestamp issue will still exist!

5. Changed the hard coded year in the output to 1601

In previous versions of the Netlogon parser, the year in the timestamp field always displayed as “2013” (the year of the initial release of Message Analyzer). This was a bit confusing, so we thought it would be better to use something we use in Active Directory as a standard for attributes that require dates but have not yet been set, which is the year 1601 (ok you caught me, we use 1/1/1601 in AD attributes). Remember that the Netlogon log does NOT contain a year in it, so a year was picked to allow the TimeElapsed field to operate properly.

6. Added “sysvol not ready” detection

Recently an issue came across my plate that involved failed authentication and trust establishment. The problem was a massive scale of sysvol not being shared on all/nearly all domain controllers. While fairly easy to spot in the Netlogon log directly, it still involved a lot of review of a lot of logs to find the errors and determine the cause. As a result, the “sysvol not ready” detection was added into the parser to streamline this type of troubleshooting (and besides that, I think you probably want and NEED to know that Sysvol isn’t being shared on your domain controller!).

7. Added detection of specific error codes to detect issues without reviewing individual authentication calls

Long ago when we started down the road to creating a Netlogon parser, I felt it was extremely important to help the general public to resolve their problems outright and not just provide a new way to read the logs. We’ve come a long way in those regards with the Netlogon parser v1.0.1, and have expanded on that significantly in v1.1.4. In v1.0.1, we streamlined identification of things like no client site detection, MaxConcurrentApi issue detection, and RPC port exhaustion…3 very common things that can cause some very major and very expensive problems.

With the Netlogon parser v1.1.4, we’ve expanded on that significantly to include detection of all of the potential issue causing error codes (plus the aforementioned sysvol not ready detection). Note that it does not look for all known error codes, only those that are likely to cause you some headaches. The error code detection evaluates outside of the authentication calls themselves to allow authentication attempts to still be properly grouped together. Here is a list of the new error code detections added into the Netlogon parser v1.1.4 (note that these error detections are grouped together and tell you what the error code means; you will see this in the upcoming screenshots):

8. Combined SChannel and Kerberos PAC validation calls into their own groupings

Since what you can get out of SChannel authentication and Kerberos PAC validation is limited to the domain being called and any return code, it was decided to free up some space in the analysis grid for more useful information. These types of calls are still parsed, however they are each now added to a corresponding larger operational group. If there were errors, they should still be detected by the aforementioned error code detections above.

So now with some of the explanations of the updates out of the way, let’s take a look at the new detections available. The Winsock/RPC port exhaustion and the MaxConcurrentApi detections were discussed in my previous posts for the Netlogon parser v1.0.1, so I will not go into those in this blog. Please review the Introduction blog and the Troubleshooting Basics for the Netlogon Parser for Message Analyzer blog if you want to see details on those particular detection mechanisms. Other than a slight change of wording, the detection has been unchanged in the Netlogon parser v1.1.4.

Normally, I would provide you a breakdown with screenshots of each individual detection mechanism, but since this is a new features post, I am trying to minimize that a bit. Along those lines of thinking, what I’ve done is created a sample Netlogon log file that includes only lines that are applicable to the new features to show those to you. As a heads up, these screenshots are taken with using a custom view layout I created (which in plain English means I removed the columns that aren’t of any use for the Netlogon parser).

It wouldn’t be right if I didn’t expand that view a little more to show you the calls within these groupings…

In this screenshot, we can see the “sysvol not ready”, the MISC and MAILSLOT (which were covered in previous blogs), account lockout, RPC bad stub data, RPC call cancellation, trusted domain failure, no memory status, and finally, the no logon servers available detections. As you can see, the wording doesn’t change *much* under each of these error detection groupings.

In this next screenshot, you can see the invalid server state, insufficient resources, no LSA secret, access denied calls, Netlogon not started, invalid network address (or out of memory), and the RPC server is unavailable detections. There’s also a glance of the SChannel authentication grouping, which I’ll show you in just a second. As with the first screenshot, you can see the wording doesn’t change that much between the operational grouping and the individually parsed line.

In this last screenshot, you can see the changes made to the SChannel and Kerberos PAC validation authentication groups. Again, this was done because the information provided in these calls is minimal, and is basically limited to the user’s calling domain, the proxying server/machine, and the return result. If there is an error on the authentication return call, then that should be detected by the specific error code detection mechanisms put into place. But, when all else fails, you can still filter what you’re looking for as discussed in the Troubleshooting Basics for the Netlogon Parser for Message Analyzer blog.

Known issues

As with all new toys, there’s always some cool new features, but there’s also usually some sort of issue (in this case though, it’s *mostly* not the parser’s fault!). So in this section, I want to recap some of the known issues:

1. Message Analyzer performance

a. There are known issues with using Message Analyzer on single core virtual machines where the CPU can (and will) spike up to 99-100%.

b. Message Analyzer, when used with the Netlogon parser, can have a decent memory footprint. I recommend having at least 4GB of RAM, but as we all know, the more RAM the better!

2. Netlogon parser performance

a. There are known performance issues when the Netlogon log file being reviewed is larger than 100MB.

b. Netlogon parser performance and functionality can be impacted if there are non-contiguous timestamps within the log file being reviewed. Put another way, if you have temporarily enabled Netlogon logging in the past, and then re-enable it 6 months later (as an example), you may impact performance and functionality due to the differing timestamps.

i. In this situation, you can stop the Netlogon service, delete or rename Netlogon.log, then start the Netlogon service once again to start from scratch with your file

3. Authentication groupings

a. There are known issues with authentication groupings that tie back to certain versions of Netlogon.dll. In order for the authentication grouping to work properly, the entered and returned line must contain a timestamp, and must be on their own line.

i. In situations where you see authentication times have a large time elapsed where the lines in the grouping are a distance apart in the log file (ex: entered line on frame 1500 and returned line on frame 98000), then you can check the operational grouping with the summary “The lines grouped here are typically not useful for troubleshooting! Please expand grouping for details.”

ii. In high resource utilization times, such as lsass spiking the CPU, there could be gaps in data that lead to missed lines that can also lead to authentication grouping/TimeElapsed mismatches.

4. Timestamps (only when used with Message Analyzer 1.1)

a. When using the Netlogon parser v1.1.4 with Message Analyzer 1.1, the timestamp -UTC time issue that is corrected when the parser is used with Message Analyzer 1.2 still exists. You still gain the additional functionality.

How to update the Netlogon parser manually if you are unable to upgrade to Message Analyzer 1.2

If for some reason you are unable to upgrade to Message Analyzer 1.2, but still want to take advantage of the new features introduced in the Netlogon parser v1.1.4, then you can follow the below 4 steps to implement the updated Netlogon parser version for Message Analyzer 1.1. Please keep in mind that the Netlogon parser v1.1.4 is written for Message Analyzer 1.2 and beyond, so there may be bugs that were not identified in testing and are not covered in the above known issues list!

NOTE: No version of the Netlogon parser will function on any Message Analyzer version less than Message Analyzer 1.1. It is highly suggested to find a way around your deployment blocker so that you can upgrade to Message Analyzer 1.2 as soon as possible!

With that being said, here’s how you manually update the parser:

1. If Message Analyzer 1.1 is running, please shut it down and ensure the process is no longer listed in Task Manager

2. Download the Netlogon-config.zip file in this blog (this is v1.1.4 of the Netlogon parser)

3. Unzip Netlogon-config.zip to a location of your choosing

4. Copy the Netlogon.config file that you unzipped into %userprofile%\AppData\Local\Microsoft\MessageAnalyzer\OPNAndConfiguration\TextLogConfiguration\AdditionalTextLogConfigurations (when prompted to overwrite the file, select the option to replace the file in the destination)

After following the above 4 steps, the Netlogon parser v1.1.4 should now be implemented and available for use once you reopen Message Analyzer.

Reference links

Message Analyzer v1.2 download (highly recommended!)

https://www.microsoft.com/en-us/download/details.aspx?id=44226

Introducing the Netlogon Parser (v1.0.1) for Message Analyzer 1.1 (By: Brandon Wilson)

https://blogs.technet.com/b/askpfeplat/archive/2014/10/06/introducing-the-netlogon-parser-v1.0.1-for-message-analyzer-1.1.aspx

Troubleshooting Basics for the Netlogon Parser (v1.0.1) for Message Analyzer (By: Brandon Wilson)

https://blogs.technet.com/b/askpfeplat/archive/2014/11/10/troubleshooting-basics-for-the-netlogon-parser-v1-0-1-for-message-analyzer.aspx

Quick Reference: Troubleshooting Netlogon Error Codes (By: Brandon Wilson)

https://blogs.technet.com/b/askpfeplat/archive/2013/01/28/quick-reference-troubleshooting-netlogon-error-codes.aspx

Quick Reference: Troubleshooting, Diagnosing, and Tuning MaxConcurrentApi Issues (By: Brandon Wilson)

https://blogs.technet.com/b/askpfeplat/archive/2014/01/13/quick-reference-troubleshooting-diagnosing-and-tuning-maxconcurrentapi-issues.aspx

Message Analyzer Forum

https://social.technet.microsoft.com/Forums/en-US/home?forum=messageanalyzer

Message Analyzer blog site

https://blogs.technet.com/MessageAnalyzer

Just to recap; please send us any suggestions or problems you identify through the comments below, the Message Analyzer forum, via email to MANetlogon@microsoft.com, or using the integrated feedback button in Message Analyzer as seen below!

Thanks, and talk to you folks next time!

-Brandon Wilson