Troubleshooting account lockout the PSS way

I‘ve been thinking for some time about pulling together the typical approaches we use when troubleshooting account lockout issues.

So... here is the CSS/PSS approach to troubleshooting Account Lockouts.

#1 - Look at the Account Lockout Threshold policy that is defined for the Domain.
Applications commonly do several retries of logons if the first logon fails (in case the first or second failures were because of transient network conditions - which is not unreasonable behavior). This means that a user that enters an incorrect password ONCE can be locked out if the lockout threshold is low – depending on how the application behaves (how many retries it attempts).
Anything below 10 is just asking for helpdesk calls.

I.e. is this simply a result of a too narrow lockout threshold policy or is this a real issue?

#2 Determine how many users are being affected
The easiest way to determine the extent of the problem is to enable Netlogon debug logging on the PDC emulator DC (All other DC‘s will contact it for confirmation in case of an incorrect password).
Once it is enabled, allow it enough time (30-60 minutes or longer) to gather information from users attempting to log on and then copy the netlogon.log and netlogon.bak files to a location where you can examine them closer.

I.e. is this an outbreak of account lockouts that affects all users or is it just affecting a handful of specific users or a single user?
What is the common factor?

I usually run the following against the Netlogon file to filter out the lockouts:
type netlogon.log |find /i "0xC000006A“ >badpassword.txt
type netlogon.log |find /i "0xC0000234“ >lockedout.txt

This should give you a list of users which are supplying bad passwords and users who have been locked out.

Note: you can't do this against a live Netlogon.log file because of the running Netlogon service - simplest solution is to copy it elsewhere and run the commands against the copy.

#3 Look at which machines the bad password attempts are coming from If you see logon attempts for different users coming from the same machine it can mean that it‘s a multi-user machine (like a Terminal Server). It can however also mean that it has some suspect application running on it – in either case you should investigate closer.

#4 Drill down to which applications are sending the bad passwords

A network trace from the client or just examining which applications and service are running on it and stopping each in turn to isolate the issue will usually be enough.

TCPView from Sysinternals or Netstat are also good for this kind of investigation, matching the process ID of a service or application that creates a socket connection with a bad password attempt in the Netlogon log of a DC.

[Footnote: Windows 8 and Windows Server 2012 log the process ID of the application logging the lockout in the Netlogon log - which makes it much easier to troubleshoot this type of problme on W8/W2k12]

Common contributors can be OS components like Credman with stale passwords, services running under a specific domain account, dumb applications with insufficient retry logic, etc.

The Conficker virus was also notorious for attempting brute force password attacks against members of the built-in Administrators group in the Domain.

 

Note also that if you have a mixed environment you may get Account Lockout issues when you change passwords on one OS (client-side or DC-side) and then move to another legacy client that doesn't understand the protocol or algorithm used.

  

Summary:
- If you enable a lockout policy on the domain – you will get accounts locked out at some point.

- The lower the lockout attempt threshold – the more frequent the lockouts obviously.

- Using an account lockout policy is exposing yourself to a potential Denial of Service attack – there‘s nothing that prevents one spiteful user from attempting to log on as another user and entering an incorrect password until they‘ve locked out the account.

The potential value of using a lockout policy in your domain is to prevent brute-force password attacks. At the same time it exposes you to Denial-of-Service attacks if you enable it.

 When I started at Microsoft some years ago I came from a mixed support environment, during that time I had learned to love two extremly powerful string parsing utilities; GAWK and SED.  

I wrote a simple batch file that uses those two commands to parse the Netlogon file into a more visually pleasing format, here is a sample output from it:

 

 Summary of Netlogon log file Netlogon.bak
Netlogon log file starts at 07/14 17:07:40
....................ends at 07/15 09:09:19
-----------------------------------------------
Total successful logons of users: 30271
Total locked out logons of users: 475
Total logons of users with bad password: 16596
Total logons of users with unknown username: 2
Total logons from undefined subnets: 2
Total Access Denied errors: 0
Total trust password errors: 0
Total KDC PAC verifications : 1137

 ......

 #of attempted logons with bad passwords from each machine
----------------------------------------------------------------------------------
11430 DOMAIN\USER9424 from WORKSTATION-43928 (via EXCHANGE09)
2647 DOMAIN\USER9323 from WORKSTATION-44367 (via EXCHANGE14)
756 DOMAIN\USER5254 from WORKSTATION-44791 (via EXCHANGE09)
380 DOMAIN\SVCACCT01 from EXCHANGE09 (via EXCHANGE09)
380 DOMAIN\SVCACCT02 from EXCHANGE01 (via EXCHANGE01) 
  

In this log:

 -the top 3 users had a bad application running on their workstation which attempted to log on to their Exchange mailbox every 5 seconds if an incorrect password had been entered.  
- the passwords had been changed twice for service accounts running for an antivirus agent on EXCHANGE01 and EXCHANGE09, but the password hadn‘t been updated to reflect that on the servers.

 

Account Lockout and Management Tools
http://www.microsoft.com/downloads/details.aspx?familyid=7AF2E69C-91F3-4E63-8629-B999ADDE0B9E&displaylang=en  

Account Lockout Best Practices White Paper
http://www.microsoft.com/downloads/details.aspx?FamilyID=8C8E0D90-A13B-4977-A4FC-3E2B67E3748E&displaylang=en&displaylang=en

 

Troubleshooting Account Lockout
http://technet.microsoft.com/en-us/library/cc773155(WS.10).aspx

Account Lockout Tools
http://technet.microsoft.com/en-us/library/cc738772(WS.10).aspx