The case of the mysterious account lockout coming from Exchange

  • Article

I worked the following case recently:

We have a single user that keeps getting his account locked out every 60 seconds. We've managed to isolate this down to coming from the Exchange server but there isn't anything pointing in the right direction as to what is causing it.
The really strange bit is that if we shut down the users’ workstation the lockouts continue regardless. And no, we’re not using terminal services where a disconnected session could be doing this.

The event logged in the Security log of the authenticating DC was as follows:

Event ID: 529
Source: Security
Type: Failure Audit
Logon Failure:
Reason: Unknown user name or bad password
User Name: User1
Domain:
Logon Type: 3
Logon Process: Advapi
Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
Workstation Name: EXCHSRV01
Caller User Name: EXCHSRV01$
Caller Domain: CONTOSO
Caller Logon ID: (0x0,0x3E7)
Caller Process ID: 1728

The standard approach to Account Lockout troubleshooting of enabling Netlogon Debug Logging unfortunately didn’t help much in this case; the only thing we saw in the resulting Netlogon.log file was a confirmation that the account lockouts were in fact being initiated by the Exchange server.

Looking at the Caller Process ID from the event was however more interesting, process #1728 on the Exchange server was the Inetinfo process. Inetinfo runs the web services that Exchange uses, including Outlook Web Access, SMTP Virtual Server, NNTP, IMAP4 and POP3 services.

Fair enough… that means we’re most likely either looking at a failed Outlook Web Access attempt that’s been cached by a browser or a program that’s resending it every 60 seconds *or* maybe we’re looking at an attempt to download POP3 mail or send authenticated SMTP mail (or even NNTP access).

Looking at the configuration of the Exchange server, we could dismiss the POP3/IMAP4/NNTP bits straight away; they simply weren’t enabled on the server.

The OWA logs in IIS didn’t show the username of the affected user at all, so this wasn’t being caused by an attempt to access the mailbox using Outlook Web Access.

That left the SMTP bit; we looked at the IIS logs for the SMTP Virtual Server but we didn’t see any reference to the user there either. After taking a network trace and analyzing it however we discovered why; the SMTP authentication simply doesn’t log the username into the IIS logs but we did see the IP address of the incoming clients and were able to use that to match the account lockouts in the Netlogon.log to determine where the lockouts were coming from.

Here’s what was happening:

- The affected user was the administrator of a web application

- That web application was able to send SMTP notifications

- To send the notifications, the web app needed credentials

- The web app admin had entered his username/password and the web app was using it when attempting to send authenticated SMTP mail

- After 2 password change cycles had occurred, the lockouts started

To resolve this; the web app admin removed his username/password from the web app configuration and replaced it with the service account that he had been assigned to use for it.

The morale of the story is that the amount of log data available to troubleshoot Account Lockouts resulting from SMTP authentication is very limited.
When only one user is affected it’s also always good to look at the affected user and investigate whether they are doing anything special that could be triggering the problem (as in this case).

Log Formats (IIS 6.0) http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/083c66a1-4860-4b53-9eca-9736cee15138.mspx?mfr=true.

NCSA Common Log File Format (IIS 6.0)
http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/a3ca6f3a-7fc3-4514-9b61-f586d41bd483.mspx?mfr=true.

Setting IP Access Restrictions to Servers (IIS 6.0)
http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/db81ac6a-4014-4189-91a2-12632d21aa64.mspx?mfr=true).