Monitoring what matters - Windows Event Forwarding for everyone (even if you already have a SIEM.)

   Last week at Ignite Australia I presented a session (available here ) on something I don't think gets talked about enough - Windows Event Forwarding, or WEF.  (Edit: I've also since done an depthMicrosoft Virtual Academy sessionon Event Forwarding too! ).

Often when we engage for an Incident Response, we find the customer :

  • Has no centralized logging
  • Are not monitoring endpoints/member servers (often just DCs)
  • Spam logs with extra data
  • Are not logging key events
  • Logs roll too quickly
  • Those with centralized logging still missing data, takes too long for IT admins to get reports

In Internet speak :

WEF's been part of the operating system for a while now, but not many people take advantage of it. Many people appear to not even be aware of it, as evidenced by this highly scientific poll.

WEF is not only free and built-in, it has some nice features when configured appropriately :

  • Works nicely as a compliment to SCOM or SIEM products.
  • Can dramatically increase your overall visibility in the environment. Frequently desktops aren't part of the centralized monitoring due to licensing costs - WEF can be setup (for free) to collect the key events from the desktops and then be forwarding on to the SIEM.
  • Configured via GPO (easy!)
  • Uses Windows Remote Management (Kerberos) to prevent man in the middle
  • Can (and should be) targeted to specific events
  • Native evtx (xml) log format
  • “Push” log mode – less attack surface than adding a monitoring agent or account to a widely privileged group
  • IT admins control their own logging destiny - AD guys looking to track down a certain service account don't have to wait on the security team.

Setting WEF up is really easy too. Prerequisites are essentially a server and a GPO. To collect security events, we'll also need to grant the local Network Service principal rights to read that log. This is just the Network Service on the machine itself, so it's not a wide privilege throughout the domain. The WinRM service will also need to be started on all the clients in the domain - just started though, not configured. This is key, as just starting the WinRM service doesn't leave it in a listening state, versus a quick config of the service would make it listening.

Recommendations for the collector server would be to use 2012R2, although you can even do this on Windows 7 if you have licensing restraints. The log files will be pretty small if you're just collecting targeted/critical events, so you likely won't need much above and beyond your typical VM drive size but you may want to bump up the memory in the box a bit to avoid race conditions.

On your collector server, fire up an administrative command prompt (you do have UAC enabled, right? 🙂 ) and type in winrm qc. You'll then be asked two important questions - do you want the WinRM service to start automatically and do you want to poke a hole in the firewall for WinRM. I'd recommend controlling the firewall with a GPO as well, but go ahead and answer yes to both of these.

When that created the WinRM listener, it also created a Service Principal Name for Kerberos authentication to the service. If something goes flakey with WEF, this is usually where it happens. If the SPNs aren't right, Kerberos authentication can't happen which makes Event Forwarding not work. You may also have noticed the listener is on HTTP ://* - WinRM is at its core HTTP, so while the SPN is WSMAN for WinRM, if you try to install WEF on a server that already has an HTTP SPN Windows Event Forwarding will fail as if it's a duplicate SPN. If you're curious, you can check your SPNs out via attribute editor.

Once WinRM is setup (and hopefully after you've set the firewall via GPO) you can enable Event Forwarding. Open up Event Viewer on the Collector and navigate to the area called "Subscriptions" that you've probably never clicked on before. 🙂 If you haven't clicked on it before, you'll get prompted with this question:

We do need/want the Windows Event Collector to start up automatically with the computer, so go ahead and click yes on that.

And that's it, you've configured a Windows Event Collector! The next trick is to get things report to it.

The best way to do that is via GPO. First things first, we'll need to give the local Network Service principal rights to read the security log. To make sure we don't break anything, run the following command on a server/workstation in your environment:

 wevtutil gl security

That will spit out the information about the Security Event Log, as shown below. The weird "O:BAG:SYD:" line is where the permissions on the log are stored. Copy out that line from the O through the last parenthesis and stick it into Notepad. If yours doesn't have (A;;0x1;;;NS) on the end like mine does, append that in Notepad.

Copy the whole O:BAG:SYD line from Notepad, as we'll need it in the GPO. Create a new GPO in your domain to point the systems you want to monitor at the collector server - the same GPO can be used for both member systems and Domain Controllers in your environment, unless you want to point them at different collectors for delegation reasons.

You'll need to configure two settings :

Computer>Policies>Admin Templates>Windows Components>Event Forwarding>Configure target subscription manager

This will need to be populated with the address of your collector server in this format :

Server=https://fqdnofsubscriptionserver:5985/wsman/SubscriptionManager/WEC,Refresh=60

The refresh interval on the end indicates how often clients should check in to see if new subscriptions are there for them. 60 seconds might be a bit aggressive in production, but it helps out a lot when you're setting things up and testing.

Computer>Policies>Admin Templates>Windows Components>Event Log Service>Security> Configure log access

This is where you'll need to paste your O:BAG:SYD line from Notepad. Remember this is an authoritative setting, so if you had permissions set some other way on a system in your environment this would replace them.

When you're done, your GPO should look like this:

Now any system that has this GPO will know to check into the web service running on the Windows Event Collector to see if there are any subscriptions for it. Computers only send events when they get a subscription telling them to do so.

So what should you monitor?

You absolutely could configure WEF to collect all the security logs in your domain - and maybe if you don't have any other centralized logging in your domain you should do this for forensic reasons - but the real value of WEF is targeted alerts, filtering out what really matters. This is also where WEF is a great compliment to a SIEM you already have in your environment - let the SIEM do the heavy lifting of collecting every single event and use WEF for targeted visibility, and use WEF to get important security events from workstations/member servers in your environment you may not have covered by the SIEM. The SIEM can then collect them from the WEF server, still providing you with the "single pane of glass" view.

The five basic things I think everyone should start with for monitoring in their domain (if they aren't already) are :

  • Security Event Logs being cleared
  • High value groups like Domain Admins being Changed
  • Local administrator groups being changed
  • Local users being created or deleted on member systems
  • New Services being installed, particularly on Domain Controllers (as this is often an indicator of malware or lateral movement behavior.)

Configuring subscriptions can be done via the GUI (easy mode) or via XPath filters (lots more flexibility.) The settings for the subscriptions do matter though, as this is where you configure the logs to be in "push" mode versus "pull."

Let's configure the Security Event log cleared alert via the GUI:

Right click on Subscriptions, then select Create Subscription. We'll need to give the subscription a name and pick "Source Computer Initiated" as that's what makes it "push."

Select Computer Groups next, as this will define which computers send us the events we're interested in. Since we want to know whenever anyone anywhere clears a security log, we're going to use the two built-in/auto populating groups in AD, "Domain Controllers" and "Domain Computers."

Next we click Select Events to define what we're monitoring. This alert is pretty straight forward, we're looking for Event ID 1102 in the Security log, so we can do it all via the GUI. 🙂

If we click on the XML tab we can see what the XPath filter we just created looks like.

Click okay on that and then click Advanced on the main page. We want to set the subscription to Minimize Latency, to make sure we get events as soon as possible and also to help it catch up if we miss any.

You can troubleshoot that things have actually checked in by checking the Source Computers column on the main Subscriptions page :

You can also check the Event Forwarding Plugin Operational log under Applications and Services on the client to make sure everything is happy. This is where you'll see descriptive errors if something has gone awry with Kerberos or Firewalls.

Click okay and you've got a subscription. Find a low-value test VM, clear the Security log and see if you get an alert. 🙂

Configure those 5 events with cut and paste for two commands? Yes you can!

Thanks to Australian PFE Russell Tomkins, you can do just that. Below are two XML files that contain the appropriate subscriptions for Domain Computers and Domain Controllers. The subscriptions here are maybe a bit wider than you want in your production domain to start with, as it's collecting services installing on workstations too, but give it a go. We'll want that data if we ever have to do an IR for you.

To import the XML files, save them to a directory on the server and then run the following commands from the same directory on your Windows Event Collector.

wecutil cs DomainComputers.xml

wecutil cs DomainControllers.xml

DomainComputers.xml

<Subscription xmlns="https://schemas.microsoft.com/2006/03/windows/events/subscription">
<SubscriptionId>Domain Computer Events</SubscriptionId>
<SubscriptionType>SourceInitiated</SubscriptionType>
<Description>Important Domain Controller Events</Description>
<Enabled>True</Enabled>
<Uri>https://schemas.microsoft.com/wbem/wsman/1/windows/EventLog</Uri>
<ConfigurationMode>MinLatency</ConfigurationMode>
<Query>
<![CDATA[<QueryList>
<Query Id="0" Path="Security">
<!-- Local Admins Changed -->
<Select Path="Security">
*[EventData[Data[@Name='TargetUserName'] and (Data='Administrators')]]
and
*[System[(EventID='4732') or (EventID='4733')]]
</Select>
<!-- Local user created or deleted -->
<Select Path="Security">*[System[(EventID='4720') or (EventID='4726')]]</Select>
<!-- New Service Installed -->
<!-- Event Log Cleared -->
<Select Path="Security">*[System[(EventID='1102')]]</Select>
</Query>
</QueryList>]]>
</Query>
<ReadExistingEvents>true</ReadExistingEvents>
<TransportName>http</TransportName>
<ContentFormat>RenderedText</ContentFormat>
<Locale Language="en-US"/>
<LogFile>ForwardedEvents</LogFile>
<AllowedSourceNonDomainComputers></AllowedSourceNonDomainComputers>
<AllowedSourceDomainComputers>O:NSG:NSD:(A;;GA;;;DC)(A;;GA;;;DD)</AllowedSourceDomainComputers>
</Subscription>

DomainControllers.xml

<Subscription xmlns="https://schemas.microsoft.com/2006/03/windows/events/subscription">
<SubscriptionId>Domain Controller Events</SubscriptionId>
<SubscriptionType>SourceInitiated</SubscriptionType>
<Description>Important Domain Controller Events</Description>
<Enabled>True</Enabled>
<Uri>https://schemas.microsoft.com/wbem/wsman/1/windows/EventLog</Uri>
<ConfigurationMode>MinLatency</ConfigurationMode>
<Query>
<![CDATA[<QueryList>
<Query Id="0" Path="Security">
<!-- New Service Installed -->
<Select Path="System">*[System[(EventID='7045')]]</Select>
<!-- Member Added or Removed from an AD Domain Local, Universal or  Global Security Group -->
<Select Path="Security">
(*[EventData[Data[@Name="TargetUserName"] = "Administrators"]]) or
(*[EventData[Data[@Name="TargetUserName"] = "Domain Admins"]]) or
(*[EventData[Data[@Name="TargetUserName"] = "Schema Admins"]]) or
(*[EventData[Data[@Name="TargetUserName"] = "Enterprise Admins"]]) or
(*[EventData[Data[@Name="TargetUserName"] =  "Print Operators"]]) or
(*[EventData[Data[@Name="TargetUserName"] = "Server Operators"]]) or
(*[EventData[Data[@Name="TargetUserName"] = "Backup Operators"]])
and
*[System[(EventID='4732') or (EventID='4733') or (EventID='4756') or (EventID='4757') or (EventID='4728') or (EventID='4729')]]
</Select>
<!-- Event Log Cleared -->
<Select Path="Security">*[System[(EventID='1102')]]</Select>
</Query>
</QueryList>]]>
</Query>
<ReadExistingEvents>True</ReadExistingEvents>
<TransportName>http</TransportName>
<ContentFormat>RenderedText</ContentFormat>
<Locale Language="en-US"/>
<LogFile>ForwardedEvents</LogFile>
<AllowedSourceNonDomainComputers></AllowedSourceNonDomainComputers>
<AllowedSourceDomainComputers>O:NSG:NSD:(A;;GA;;;DD)</AllowedSourceDomainComputers>
</Subscription>

If you have more than one domain, you'll need to specify the SIDs of Domain Computers/Domain Controllers in the <AllowedSourceDomainComputers> line as well as the DC (Domain Computers) and DD (Domain Controlls) SDDLs that are in there.

Hope this helps. Be sure to check out my post on tracking Special Groups with WEF and look forward to more things you can do with WEF in future blog posts! Happy Logging!

-Jessica @jepayneMSFT