Forensics: Automating Active Directory Account Lockout Search with PowerShell (an example of deep XML filtering of event logs across multiple servers in parallel)

Overview

Today we learn how to efficiently filter event log queries, going beyond simple event ID filtering into the specific values of the XML message data. Then we will run this filter against multiple servers in parallel for faster data collection.

This posts meets the following objectives:

  • Add some efficiencies to my previous popular post for parsing XML event message data.
  • Apply the concept to Active Directory account lockout troubleshooting, formerly posted in the sample code of this post.
  • Provide a working example for your future event log forensic scripts, regardless of which logs or events or data you are mining. Our example today is Active Directory lockouts and bad password attempts.

Someone else may have done this already, but I have not searched for other examples.  This is my own approach to the solution.

“Back in my day…”

During the Windows Server 2003 era the Account Lockout Tools became very popular for tracking down those notorious account lockout scenarios. You would use LockoutStatus.exe to find the DCs targeted with failed password attempts, then you would use EventCombMT.exe to harvest the security log lockout events on those DCs. Eventually you would find the culprit causing the lockout, and sometimes it was your own fault for leaving an RDP session logged in for days. Surprise! Smile

This process was half-automated. You still had to jump through multiple tools with manual effort to get the results. But that was before PowerShell.

Solution Overview

Today’s script mimics these steps entirely with PowerShell:

  1. Get a list of locked out accounts using the AD cmdlet Search-ADAccount –LockedOut
  2. Query the lockout count for each account across all DCs to see where the lockouts are occurring.
  3. Retrieve the related event log entries from the DCs where the lockouts occurred (in parallel).
  4. Review the data in Out-GridView and CSV.

The previous solution using multiple tools would prompt for an account name and ask you to pick DCs to query.  In this case I have automated these steps into one big button to get it all.  It takes the thinking out of the problem solving and makes sure you don’t miss anything.

Locked Out Accounts

Finding the currently locked out accounts is now really easy with the Active Directory cmdlets:

 PS C:\> Search-ADAccount -LockedOut

AccountExpirationDate : 
DistinguishedName     : CN=Jim,CN=Users,DC=CohoVineyard,DC=com
Enabled               : True
LastLogonDate         : 
LockedOut             : True
Name                  : Jim
ObjectClass           : user
ObjectGUID            : d13287cb-5725-4e21-ba75-2acfe383fc46
PasswordExpired       : True
PasswordNeverExpires  : False
SamAccountName        : Jim
SID                   : S-1-5-21-2999376440-943117962-1153441346-7287
UserPrincipalName     : 

We have an array of rich PowerShell data objects coming back for each locked out account.

Lock Out Counts

When bad password attempts occur, you will see the lockout count incremented on the local DC processing the logon attempt and also the PDC Emulator (PDCe). This ensures that in case the logon attempts are tried across different DCs, a grand total bad password count is enforced on the central PDCe. For this reason I added a column to the output to note which DC is the PDCe.

While I was adding value to the DCs in the lockout count query, I also added columns for IPv4Address and Site. This data is helpful to identify cases where clients may be authenticating outside of their intended site. You can also loosely compare the DC IP address with the client IP address that comes up on the later report of bad password attempts.

In the “Lockout Status By Account By DC” report notice that you may see DCs repeated in the list. If you have multiple locked out accounts, then you will get a row for every DC where their logons were attempted. When multiple accounts are locked out you will always see at least the PDC listed for each account.

There are multiple DateTime columns in this output for determining when the account was created, last changed, last logon, last bad password, etc.  DateTime values are not human-friendly sometimes in AD, so I converted them using this trick from a former post:

@{name='badPasswordTimeConverted';expression={[datetime]::fromFileTime($_.badPasswordTime)}}

The fromFileTime() method is your best friend here.

Event Log Collection

This solution builds on the XML event log script from the past post, but this is more efficient by doing deep XML filtering on the message body.  This took some trial-and-error, but I finally crafted the correct XML filter syntax. Please study the code and use this technique across other event log scenarios you may encounter in the general world of Windows Server.

In the event viewer graphical interface we see the lockout event details.

Screenshot (207)

Click the Details tab, and them XML View to reveal the data within the event.

Screenshot (208)

Here you find the username that was locked out, and the computer name where the lockout originated. However, the names of the fields are not very intuitive (TargetUserName, TargetDomainName).

In my Microsoft Virtual Academy series on Active Directory PowerShell, module four covers forensics.  In that segment I demo tracking down account lockouts based in part on some code from an earlier post about parsing XML event log message data.  In that earlier post I offered a couple caveats:  it is terribly inefficient and I couldn’t get it to work remotely using the Get-WinEvent cmdlets directly. There were also some helpful comments worth noting in that previous XML post.

Problem #1: Remote Event Retrieval

PowerShell remoting serializes output before it is returned from remote sessions, and this removes methods like .ToXML() that we need here. I addressed the remote event retrieval issue in two ways:

  1. I incorporate the Get-WinEvent inside Invoke-Command.  Inside a remote session we get the XML event data, append the properties to the event object, and then return it.  We can get the details we need by executing the XML event data collect prior to bringing the data back across the wire. This code is passed as a scriptblock to Invoke-Command.
  2. Another snag with Get-WinEvent –ComputerName is that the Remote Event Log Management firewall rule must be enabled on the server first.  This may not always be practical, so using Invoke-Command uses the standard, already-open remoting port to retrieve the event data without having to open another firewall port.
 # Only get event logs from the DCs that show a lockout count            
$DCs = $report |            
    Where-Object {$_.badPwdCount -gt 0} |            
    Select-Object -ExpandProperty DC -Unique            
            
$Milliseconds = $Hours * 3600000            
# Script block for remote event log filter and XML event data extraction            
# Logon audit failure events            
#   Event 4625 is bad password in client log            
#   Event 4771 is bad password in DC log            
#   Event 4740 is lockout in DC log            
$sb = {            
[xml]$FilterXML = @"
<QueryList><Query Id="0" Path="Security"><Select Path="Security">
*[System[(EventID=4740 or EventID=4771)
and TimeCreated[timediff(@SystemTime) &lt;= $Using:Milliseconds]]]
$Using:UserFilterXML
</Select></Query></QueryList>
"@
     Try {            
        $Events = Get-WinEvent -FilterXml $FilterXML -ErrorAction Stop            
            
        ForEach ($Event in $Events) {            
            # Convert the event to XML            
            $eventXML = [xml]$Event.ToXml()            
            # Iterate through each one of the XML message properties            
            For ($i=0; $i -lt $eventXML.Event.EventData.Data.Count; $i++) {            
                # Append these as object properties            
                Add-Member -InputObject $Event -MemberType NoteProperty -Force `
                    -Name  $eventXML.Event.EventData.Data[$i].name `
                    -Value $eventXML.Event.EventData.Data[$i].'#text'            
            }            
        }            
            
        $Events | Select-Object *            
    }            
    Catch {            
        If ($_.Exception -like "*No events were found that match criteria*") {            
            Write-Warning "[$(hostname)] No events found"            
        } Else {            
            $_            
        }            
    }            
                
}            
            
# Clear out the local job queue            
Get-Job | Remove-Job            
            
# Load up the local job queue with event log queries to each DC            
Write-Verbose "Querying lockout events on DCs [$DCs]."            
Invoke-Command -ScriptBlock $sb -ComputerName $DCs -AsJob | Out-Null            

Note the use of the –AsJob switch with Invoke-Command.  Obviously querying event log data from multiple DCs is going to take some time. The ThrottleLimit parameter defaults to 32, so that should be plenty of parallel threads for most people. While there are multiple methods for parallel execution in PowerShell, this one is the most convenient for our purposes.  If you have not read about Jobs in PowerShell, they have been around since version 2.0.  Check out the help topic about_Jobs.

Problem #2: Deeper XML Filtering Inside the Message Data

In my last attempt at this solution I used this inefficient method:

  • Retrieved all the lockout events
  • Looped through all of them again to extract the XML data
  • Then passed them to Where-Object for filtering for the username

Since that time, my Connect item calling out the lack of MSDN links for XPath in the help for Get-WinEvent has been closed.  The help has been updated with relevant links that are helpful in understanding the nuances of XPath and FilterXML.

However, most helpful to me was a series of forum posts over at StackOverflow by some people who understand XML way better than me. For more background I suggest you go read this handy post by the Scripting Guy on how to use FilterXML by copying from the Windows Event Viewer:  Use Custom Views from Windows Event Viewer in PowerShell.  I love this trick, and I use it all the time, including this script.

Here is the updated FilterXML syntax for deeper message data filtering with the new part highlighted:

<QueryList><Query Id="0" Path="Security"><Select Path="Security">
*[System[(EventID=4740 or EventID=4771) and TimeCreated[timediff(@SystemTime) &lt;= 43200000)]]]
and *[EventData[(Data[@Name= 'TargetUserName' ] = 'alice') or (Data[@Name='TargetUserName'] = 'bob') or (Data[@Name='TargetUserName'] = 'charlie')]]
</Select></Query></QueryList>

I am not going to explain the XML syntax here, but the point is that we are adding another criteria to the filter reaching inside EventData/Data to the property TargetUserName. We are actually repeating the property filter for each username we need to capture for lockouts in the logs. Note that this works well when matching exact values or doing simple greater than or less than comparisons.

After this tweak, our filtering is much more efficient:

  • Query the log and only return the smaller set of entries in scope.
  • Then process the XML properties of each one.

This significantly reduces the operations required to achieve the same end.

While tweaking the FilterXML I added a parameter to my function to specify the number of hours of event log history to search. The highlighted number 43,200,000 above is 12 hours (12 hours * 60 minutes * 60 seconds * 1,000 milliseconds).

View the Results

When the script is finished you will get three reports in Out-GridView. These are also exported as CSV files.

Screenshot (209)

 Directory: C:\Users\administrator.COHOVINEYARD\Documents

Mode         LastWriteTime Length Name                         
----         ------------- ------ ----                         
-a---  8/31/2015   1:02 PM   1121 LockoutEvents_AccountData.csv
-a---  8/31/2015   1:02 PM  12346 LockoutEvents_BadPassword.csv
-a---  8/31/2015   1:02 PM   2105 LockoutEvents_Lockouts.csv   

These reports should contain all the forensic data you need to track down those lockouts in your environment:

  • LockoutEvents_AccountData.csv – the lockout counts by account by DC
  • LockoutEvents_BadPassword.csv – event ID 4771 details, one event for each bad password attempt, IP and attempted reverse lookup of hostname, authentication failure audit event
  • LockoutEvents_Lockouts.csv – event ID 4740 details, lock out success audit event, hostname of lockout machine

Enjoy!

One Big Button

Now you can retrieve your lockout data with one function call like this:

 # Default is events in last 24 hours, no IP to name resolution            
Get-ADAccountLockoutData            
            
# Last 12 hours, IP to name resolution for bad password events            
# Verbose for progress information            
Get-ADAccountLockoutData -ResolveIP -Hours 12 -Verbose            

Assumptions and Prerequisites

Essentially you need to run this from a Windows Server 2012 R2 member server or domain controller. A Windows 8.1 workstation with RSAT installed should also suffice. Here is the full disclaimer on compatibility.

  • This code was labbed on Windows Server 2012 R2 using PowerShell version 4.0.
  • You will need the RSAT for the Active Directory PowerShell module.
  • Currently this makes no provision for specifying an alternate domain name or credentials. You should run this in the domain where the lockouts are occurring using an account with Domain Admin credentials. Feel free to tweak the code for your own purposes to reach across domains.
  • This code will return no results if no accounts are currently locked out.
  • The DNSClient module on Windows Server 2012 R2 includes the Resolve-DNSName cmdlets.
  • Reverse DNS zones are needed for IP to name lookups.

I have not tried running this against Windows Server 2012, but it should work.  Windows Server 2008 R2 should work as long as WMF 3.0 or WMF 4.0 is installed. You must run it from 2012 R2, but it should be able to target these other domain controller operating systems for event log collection.

Make It Your Own

Like all of my published scripts this is intended as sample code for your to use as-is or refactor for other specific needs in your environment. This is a valuable example that reaches far beyond Active Directory applications. I hope that you have learned a little bit more about deep XML event filtering with Get-WinEvent and remote log collection efficiently. Use the comments area below to post questions or share how you have used the code.

Download the Code

You can get today’s script solution over at the TechNet Script Gallery. Please download and study to learn the techniques I discussed here today.