AD Fun Services – Track down the source of ADFS lockouts


Tracking down the devices locking out accounts on an ADFS deployment is quite challenging. From an ADDS perspective, lockouts coming from a WAP server will look like they’re come from an ADFS server:

Lockouts coming from internal client using Form Based authentication also look like they are coming from the ADFS server itself and not the device.

What can I do?

[This article is written for ADFS on Window Server 2012 R2]

You can have fun with the security event logs of the ADFS servers and fish for the right information. Quite perilous eh? First thing to do is to ensure we capture the information. So we need to enable the audit on your ADFS servers. Two things to do to achieve that:

  1. Configure the auditing on the ADFS farm:
  2. Configure the OS of the ADFS server to audit application generated events:

You could do it with a domain group policy and ensure that all your ADFS servers have the same configuration. If you want to go geek, here is PowerShell cmdLet to enable the audit on your ADFS farm: Set-ADFSProperties –LogLevel Information,Errors,Verbose,Warnings,FailureAudits,SuccessAudits and here is the command line you can run locally on the server if you want to enable this kind of audit: auditpol.exe /set /subcategory:”Application Generated” /failure:enable /success:enable

Now we will have security events containing IP addresses when an account gets locked out (we’ll see which one later). Note that because of the load balancing, you cannot predict on which ADFS server the authentication will take place. So all the methods described in this article are looking at event logs on all servers in the farm.

I use multiple devices at home

If the device is behind a NAT, the source IP address of the lockout will just tell us that it is coming from your home, and not tell us if it comes from your tablet, Xbox or fancy Windows Phone 10. Having the source IP isn’t the panacea, you also want the device identity. That, unless you are using Workplace Joined devices, isn’t possible. What we can do though, is getting the UserAgent string of the client and hope that it provides us with enough information to distinguish the device. Could you tell which UserAgent string is my Windows 10 and which one is my Windows Phone?

  • Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko
  • Mozilla/5.0 (Windows Phone 10.0; Android 4.2.1; Microsoft; Lumia 950 Dual SIM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2486.0 Mobile Safari/537.36 Edge/13.10586

Well, you guessed that the first one was my Windows 10 laptop and the second one was my fancy Windows Phone 10 (interestingly, the browser of my Windows Phone also advertises about the fact it could be an Android 4.2.1, a Chrome or a Safari).

Scenario 1: I am using the Extranet Lockout feature

If you are not familiar with this feature, you can read this excellent post. In a nutshell, we are locking the account on the ADFS server before it gets locked on the ADDS infrastructure, preventing potential password discovery attacks. For that we read the badPwdCount attribute from the PDC (note that if the PDC is not reachable during the attempt, it will fail regardless of the password provided by the user and its status -locked out or not, see this article for details). This affects only password based authentication attempts coming from a WAP server (for internal client, the ADDS account lockout policy still applies). The issue with this feature is that if the user gets locked out on the ADFS server only, you will not find a trace of a user being locked out in the ADDS servers. You will find the previous failed attempts but still, the address will show that it is coming from the ADFS server.
When a user is locked out on the ADFS server because of this feature, it generates the following event:


As you can see, the 516 does contain interesting information such as the username, the external IP address of the device, the value of the badPwdCount, the date and time of the lockout and what WAP server it is coming from. However, it does not tell the UserAgent of the device. The event 403 does:

But do you really want to parse your event logs and try to match events manually amongst hundreds of thousands other events? Probably not. If we look at the 516, we also have an activity ID. This activity ID will be included in all other ADFS audit events related to the same activity. So if we take the activity ID of the 516 and look for 403 carrying the same, we’ll match the UserAgent to our lockout.

Here is an example of PowerShell script looking for all user lockout events on all server and match it with the UserAgent. It will should you the time of the lockout, the external IP as well as some information about the device thanks to the UserAgent string.

#list all servers of your ADFS farm
$_all_adfs_servers = "adfs01.ad.piaudonn.com","adfs02.ad.piaudonn.com"
#XML filter that look for the event 516 in the security event logs coming from ADFS
$_xml_lockout_adfs = "<QueryList><Query Id=""0"" Path=""Security""><Select Path=""Security"">*[System[Provider[@Name='AD FS Auditing'] and (EventID=516)]]</Select></Query></QueryList>"
#List all server
$_all_adfs_servers | ForEach-Object `
{
    #for each server query the event logs looking for the last 100 events for lockout
    $_server = $_
    Get-WinEvent -ComputerName $_server -FilterXml $_xml_lockout_adfs -MaxEvents 100 | ForEach-Object `
    {

        #Extract the operation ID
        $_operation_id_adfs = $_.Properties[0].Value
        #Showthe details of the event
        Write-Output "Server:`t$_server"
        Write-Output "Account:`t$($_.Properties[1].Value)"
        Write-Output "ExternalIP:`t$($_.Properties[2].Value)"
        Write-Output "DateTime:`t$($_.Properties[4].Value) $($_.Properties[5].Value)"
        #Craft another XML filter to look for event 403 that have the operation ID matching the one of the 516
        $_xml_lockout_adfs_useragent = "<QueryList><Query Id=""0"" Path=""Security""><Select Path=""Security"">*[System[Provider[@Name='AD FS Auditing'] and (EventID=403)]] and *[ EventData[ Data and (Data='$_operation_id_adfs') ] ]</Select></Query></QueryList>"
       Get-WinEvent -ComputerName $_server -FilterXml $_xml_lockout_adfs_useragent -MaxEvents 1 | ForEach-Object `
        {
               #Display the UserAgent
               Write-Output "UserAgent:`t$($_.Properties[8].Value)"
        }
    }
    Write-Output "--"
}

And here is the output:

Server: adfs01.ad.piaudonn.com
Account: ad\jean
ExternalIP: 63.225.190.60
DateTime: 2/2/2016 7:16:15 PM
UserAgent: Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko

The above PowerShell is quite basic, no error management, no user input, etc. You can go fancy and make a more sophisticated version! Why only the external IP address and not the internal one in the event the lockout comes from an internal connection? The Extranet lockout feature, as the name suggests, only works for extranet connection coming from the WAP.

Scenario 2: I am not using the Extranet Lockout feature

In this case the account is going to be locked out on the ADDS servers. So you will find the event 4740 on your domain controller, but you will not find the event 516 on your ADFS servers. So what will you see in the logs? This:

Great, we can lookup up on the username and will get the Activity ID and thanks to the Activity ID we will track down this to the UserAgent string. The problem is that the username is displayed the same way that the user typed it in. So if the user typed jean@ad.piaudonn.com or AD\jean or aD\Jean or Jean@ad.Piaudonn.Com, these are all different strings… So the first thing to do to is to look up for the actual username typed in by the user. For that we need to extend our previously set audit capabilities. We will need the event 4625 to be logged in the ADFS server. If the user tried to log in with the username AD\JeAn, the event will show it:

If the user typed JeAn@ad.piaudonn.com it will look like this:

This event is keeping the case. To enable this audit on all our ADFS server (not the ADDS servers), we activate the following audit category:

(technically we can enable only the Failure, but Success does not generated noise)

So here is the logic:

  1. Get the actual username input from the event ID 4625
  2. Look for the event 411 that contains that username and retrieve the activity ID
  3. Look for failed authentication related to that activity ID and retrieve the IP address and the UserAgent

How to automate this? Let’s look for all locked out accounts listed in all ADFS server and prompt you to choose what lockout event you wish to see additional information for:

#Define all your ADFS servers
$_all_adfs_servers = "adfs01.ad.piaudonn.com","adfs01.ad.piaudonn.com"
#XML filter to look for the event 4625
$_xml_lockout = "<QueryList><Query Id=""0"" Path=""Security""><Select Path=""Security"">*[System[Provider[@Name='Microsoft-Windows-Security-Auditing'] and Task = 12546 and (EventID=4625)]]</Select></Query></QueryList>"
#Pick one is used to store the user's input
$_pick_one = @()
#List all locked out event on all servers
$_all_adfs_servers | ForEach-Object `
{
    $_server = $_    
    #List all the event 4625
    Get-WinEvent -ComputerName $_server -FilterXml $_xml_lockout -Oldest -MaxEvents 100 | ForEach-Object `
    {
        #We check what is the username input
        If ( $_.Properties[6].Value -ne "" )
        {
            $_target_account = "$($_.Properties[6].Value)\$($_.Properties[5].Value)"
        } Else {
            $_target_account = $_.Properties[5].Value
        }
        $_pick_one += New-Object -TypeName psobject -Property @{
            Server = $_server
            Time = $_.TimeCreated
            Account = $_target_account
        }
    }
}
#Display all the results
$_inc = 0
$_pick_one | ForEach-Object `
{
    $_display_cases = $_pick_one[ $_inc ]
    Write-Host "$_inc`t-`t$($_display_cases.Server)`t$($_display_cases.Time)`t$($_display_cases.Account)"
    $_inc++
}
#Ask the user to chose (here we need to do some parsing of the input, it is not done as today
$_picked_inc = Read-Host "Select a lockout event (from 0 to $($_inc - 1))"
#Once we picked, we look at the info of the lockout using the right username and get the operation ID
$_picked = $_pick_one[ $_picked_inc ]
$_xml_account = "<QueryList><Query Id=""0"" Path=""Security""><Select Path=""Security"">*[ EventData[ Data and (Data='$($_picked.Account)-The referenced account is currently locked out and may not be logged on to') ] ]</Select></Query></QueryList>"
$_get_operation = Get-WinEvent `
    -MaxEvents 1 `
    -ComputerName $_picked.Server `
    -FilterXml $_xml_account 
$_operation_id = $_get_operation.Properties[0].Value
#Look for event 410 and 403 containing the same Activity ID than the lokout event #thanks Renato for helping me out here 🙂
$_xml_operation = "<QueryList><Query Id=""0"" Path=""Security""><Select Path=""Security"">*[ EventData[ Data and (Data='$_operation_id') ] ] and *[System[(EventID=410) or (EventID=403)]]</Select></Query></QueryList>"
$_get_info = Get-WinEvent `
    -ComputerName $_picked.Server `
    -FilterXml $_xml_operation 
#Display the results
$_get_info | ForEach-Object `
{
    If ( $_.ID -eq 410 )
    {
        Write-Output "DateTime: `t$($_picked.Time)"
        Write-Output "Server:   `t$($_picked.Server)"
        Write-Output "Account:  `t$($_picked.Account)"
        Write-Output "ExternalIP:`t$($_.Properties[10].Value)"
        Write-Output "WAPServer: `t$($_.Properties[12].Value)"
    }
    If ( $_.ID -eq 403 )
    {
        Write-Output "UserAgent:`t$($_.Properties[8].Value)"
        Write-Output "InternalIP:`t$($_.Properties[2].Value)"
    }
}

Here is the output:

0 – adfs01.ad.piaudonn.com 02/02/2016 19:07:33 ad\jean
1 – adfs01.ad.piaudonn.com 02/02/2016 19:07:34 ad\jean
2 – localhost 02/02/2016 19:07:33 ad\jean
3 – localhost 02/02/2016 19:07:34 ad\jean

Select a lockout event (from 0 to 3): 0

DateTime:  02/02/2016 19:07:33

Server:    adfs01.ad.piaudonn.com
Account:   ad\jean
ExternalIP: 63.225.190.60
WAPServer:  adfsproxy01
UserAgent: Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko
InternalIP: 10.0.0.7

Again, as previously, the script is doing the minimal work, not even checking if the user’s input is correct. So please, go fancy and improve it 🙂

If the user is connected internally, the script still works.

My External IP is always the IP address of the WAP server

I’ll be brief on that section since it isn’t really an ADFS issue. If there is some NAT in front of your WAP load balancer farm, the incoming connections are actually coming from the WAP server itself and you’ll see the X-MS-Forwarded-Client-IP with the internal IP of the WAP server. In that case, you’ll have to look at a way to get that info right with your load balancer provider. Some of them supports SNAT and will be able to help in that situation. So bing it! 🙂

Comments (16)

  1. yousaidit says:

    thanks
    very helpful

  2. Rini Ghatti says:

    Great Article! Thanks for posting this Pierre.

  3. heythere says:

    very very useful
    thanks

  4. Steve says:

    Great read, is this ADFS 2.0? I’ve enabled auditing on 3.0 and I have no IPs in event code 411 Source AD FS Auditing, or in eventcode 4625, source Microsoft Windows security auditing. It indicates the user name, but no IP. The only host listed is the ADFS
    3.0 server.

  5. Steve says:

    Forgot to mention, no 4740 on the DCs and no failed IP logged on the DCs either… I get a 4741 pre-auth failed on the DCs with the IP of the ADFS server.

  6. There is a well known issue when IP addresses do not get logged. The latest updates fixed that. So I’d say, try to Windows Update your ADFS server and WAP, that should fix it.
    For the AD event missing, it has to do with your Audit Policy in AD, this will get logged only if your current policy is configured to do so. Hope this helps…

  7. Matt says:

    As far as I can see everything in our environment has been enabled to the specs in this article. However, on event 411 the activity ID is "00000000-0000-0000-0000-000000000000 ." Shouldn’t I see something different here? Furthermore, the "Client IP:" address
    of the bad lockout appears to be coming from Microsoft servers. We have Office 365 and use ADFS for authentication. However, I am stuck here as I cannot see anything further than this. Anyone have any ideas?

    1. EAS clients (Active Sync Clients such as native mail apps on Android or iPhone) are using a different profile and EXO (Exchange Online is actually making the call). In that case, you kind a reach the limits of tracking down the device I’m afraid.

      1. Marc says:

        Great article and very informative. I have the same issue. We use O365 / ADFS 3.0 and having extreme difficulties tracking down account lockouts for some users. When reviewing the data in this article, the account being investigated only lists Activity ID: “00000000-0000-0000-0000-000000000000”. Other log entries do provide the Activity ID but many of them do not. Any ideas????

        1. Mauro JK says:

          Any new on troubleshooting Activity ID “00000000-0000-0000-0000-000000000000”.
          We have O365 and ADFS 3.0. We have a lot of entries from so many users with empty Activity ID.

          Other behaviour is that all entries comes with 2 IPs…
          One fixed from our private range and other is public.

          it is very hard to fetch more informations and determine the real source of Log On Attempts.

          1. Those are probably your legacy clients (such as Active Sync clients – like mobile native mail apps). Those clients are using a different authentication flow. They send credentials to Exchange Online and EXO is sending them to ADFS. So the request will have two IP, the one from EXO and the one from the client. However the one from EXO depends whether or not you have routing or nating between the ingress point and the WAP server. This is from my observations though. I’ll try to look for an official statement. But maybe that insight helps out a bit…

          2. Robb says:

            Just an option for those who keep running into the event 411 activity ID is “00000000-0000-0000-0000-000000000000” .. because you can’t do the match-up I wrote it just to parse the 411 and pull the date/server/account/ip of the users. Hope that helps.

            $report = @()
            $_all_adfs_servers = “server1.my.yes”,”server2.my.yes”,”server3.my.yes”
            $_xml_lockout = “*[System[Provider[@Name=’AD FS Auditing’] and (EventID=411) and TimeCreated[timediff(@SystemTime) <= 43200000]]]”
            $_all_adfs_servers | ForEach-Object `
            {
            $_server = $_
            $eventdata = Get-WinEvent -ComputerName $_server -FilterXml $_xml_lockout -Oldest -MaxEvents 1000
            $lockedout = $eventdata | where {($_.properties[2].value -like “*The referenced account is currently locked out*”)}
            $lockedout | ForEach {
            $_target_account = “$($_.Properties[2].Value)”
            $_iplock = “$($_.Properties[4].Value)”
            $t = $_target_account -match “(?.*)-The referenced account is currently locked out and may not be logged on to”
            $x = $matches[‘content’]
            $mObj = New-Object PSObject
            $mObj | Add-Member -MemberType NoteProperty -Name “Server” -Value $_server
            $mObj | Add-Member -MemberType NoteProperty -Name “Time” -Value $_.TimeCreated
            $mObj | Add-Member -MemberType NoteProperty -Name “Account” -Value $x
            $mObj | Add-Member -MemberType NoteProperty -Name “IP” -Value $_iplock
            $report += $mObj
            }}
            $report | sort time

  8. LE2Strat says:

    I turned this on one of our servers (the audit part) and decided to turn it back off, not the security log doesn’t log anything at all.

    1. Make sure you do not have a GPO that will overwrite your local audit configuration.

  9. rjonah says:

    Very helpful…Has anyone figured out how to further troubleshoot Activity ID: 00000000-0000-0000-0000-000000000000 and/or why it is all zeros? All the lockout issues that i have have that in common. They also have the 2 IP addresses. One is my outside address and the other is Microsoft’s.

    Thanks,
    RJ

    1. It is the effect of using legacy clients. They show up as internal calls and do not beneficiate of the Activity ID like the other. The two IPs are: 1. the egress IP address of your device when it connects to Office 365 using a legacy client, 2. the IP address of Office 365 redirecting the auth of the legacy client to your environment (it is doing a proxy sort of authentication then). Somebody suggested a way here on the comments’ sections. basically, you can look up stuff base on the IP you got or/and the time of the day.

Skip to main content