Monitoring what matters – Windows Event Forwarding for everyone (even if you already have a SIEM.)


Last week at Ignite Australia I presented a session (available here ) on something I don’t think gets talked about enough – Windows Event Forwarding, or WEF.  (Edit: I’ve also since done an depth Microsoft Virtual Academy session on Event Forwarding too!).

Often when we engage for an Incident Response, we find the customer :

  • Has no centralized logging
  • Are not monitoring endpoints/member servers (often just DCs)
  • Spam logs with extra data
  • Are not logging key events
  • Logs roll too quickly
  • Those with centralized logging still missing data, takes too long for IT admins to get reports

 

In Internet speak : 

 

WEF’s been part of the operating system for a while now, but not many people take advantage of it. Many people appear to not even be aware of it, as evidenced by this highly scientific poll.

 

WEF is not only free and built-in, it has some nice features when configured appropriately :

  • Works nicely as a compliment to SCOM or SIEM products.
  • Can dramatically increase your overall visibility in the environment. Frequently desktops aren’t part of the centralized monitoring due to licensing costs – WEF can be setup (for free) to collect the key events from the desktops and then be forwarding on to the SIEM.
  • Configured via GPO (easy!)
  • Uses Windows Remote Management (Kerberos) to prevent man in the middle
  • Can (and should be) targeted to specific events
  • Native evtx (xml) log format
  • “Push” log mode – less attack surface than adding a monitoring agent or account to a widely privileged group
  • IT admins control their own logging destiny – AD guys looking to track down a certain service account don’t have to wait on the security team.

Setting WEF up is really easy too. Prerequisites are essentially a server and a GPO. To collect security events, we’ll also need to grant the local Network Service principal rights to read that log. This is just the Network Service on the machine itself, so it’s not a wide privilege throughout the domain. The WinRM service will also need to be started on all the clients in the domain – just started though, not configured. This is key, as just starting the WinRM service doesn’t leave it in a listening state, versus a quick config of the service would make it listening.

Recommendations for the collector server would be to use 2012R2, although you can even do this on Windows 7 if you have licensing restraints. The log files will be pretty small if you’re just collecting targeted/critical events, so you likely won’t need much above and beyond your typical VM drive size but you may want to bump up the memory in the box a bit to avoid race conditions.

On your collector server, fire up an administrative command prompt (you do have UAC enabled, right? 🙂 ) and type in winrm qc. You’ll then be asked two important questions – do you want the WinRM service to start automatically and do you want to poke a hole in the firewall for WinRM. I’d recommend controlling the firewall with a GPO as well, but go ahead and answer yes to both of these.

When that created the WinRM listener, it also created a Service Principal Name for Kerberos authentication to the service. If something goes flakey with WEF, this is usually where it happens. If the SPNs aren’t right, Kerberos authentication can’t happen which makes Event Forwarding not work. You may also have noticed the listener is on HTTP ://* – WinRM is at its core HTTP, so while the SPN is WSMAN for WinRM, if you try to install WEF on a server that already has an HTTP SPN Windows Event Forwarding will fail as if it’s a duplicate SPN. If you’re curious, you can check your SPNs out via attribute editor.

Once WinRM is setup (and hopefully after you’ve set the firewall via GPO) you can enable Event Forwarding. Open up Event Viewer on the Collector and navigate to the area called “Subscriptions” that you’ve probably never clicked on before. 🙂 If you haven’t clicked on it before, you’ll get prompted with this question:

 

 

We do need/want the Windows Event Collector to start up automatically with the computer, so go ahead and click yes on that.

And that’s it, you’ve configured a Windows Event Collector! The next trick is to get things report to it.

The best way to do that is via GPO. First things first, we’ll need to give the local Network Service principal rights to read the security log. To make sure we don’t break anything, run the following command on a server/workstation in your environment:

 wevtutil gl security

That will spit out the information about the Security Event Log, as shown below. The weird “O:BAG:SYD:” line is where the permissions on the log are stored. Copy out that line from the O through the last parenthesis and stick it into Notepad. If yours doesn’t have (A;;0x1;;;NS) on the end like mine does, append that in Notepad.

Copy the whole O:BAG:SYD line from Notepad, as we’ll need it in the GPO. Create a new GPO in your domain to point the systems you want to monitor at the collector server – the same GPO can be used for both member systems and Domain Controllers in your environment, unless you want to point them at different collectors for delegation reasons.

You’ll need to configure two settings :

Computer>Policies>Admin Templates>Windows Components>Event Forwarding>Configure target subscription manager

This will need to be populated with the address of your collector server in this format :

Server=http://fqdnofsubscriptionserver:5985/wsman/SubscriptionManager/WEC,Refresh=60

The refresh interval on the end indicates how often clients should check in to see if new subscriptions are there for them. 60 seconds might be a bit aggressive in production, but it helps out a lot when you’re setting things up and testing.

 

Computer>Policies>Admin Templates>Windows Components>Event Log Service>Security> Configure log access

This is where you’ll need to paste your O:BAG:SYD line from Notepad. Remember this is an authoritative setting, so if you had permissions set some other way on a system in your environment this would replace them.

 

When you’re done, your GPO should look like this:

Now any system that has this GPO will know to check into the web service running on the Windows Event Collector to see if there are any subscriptions for it. Computers only send events when they get a subscription telling them to do so.

 

So what should you monitor?

You absolutely could configure WEF to collect all the security logs in your domain – and maybe if you don’t have any other centralized logging in your domain you should do this for forensic reasons – but the real value of WEF is targeted alerts, filtering out what really matters. This is also where WEF is a great compliment to a SIEM you already have in your environment – let the SIEM do the heavy lifting of collecting every single event and use WEF for targeted visibility, and use WEF to get important security events from workstations/member servers in your environment you may not have covered by the SIEM. The SIEM can then collect them from the WEF server, still providing you with the “single pane of glass” view.

The five basic things I think everyone should start with for monitoring in their domain (if they aren’t already) are :

  • Security Event Logs being cleared
  • High value groups like Domain Admins being Changed
  • Local administrator groups being changed
  • Local users being created or deleted on member systems
  • New Services being installed, particularly on Domain Controllers (as this is often an indicator of malware or lateral movement behavior.)

 

Configuring subscriptions can be done via the GUI (easy mode) or via XPath filters (lots more flexibility.) The settings for the subscriptions do matter though, as this is where you configure the logs to be in “push” mode versus “pull.”

Let’s configure the Security Event log cleared alert via the GUI:

Right click on Subscriptions, then select Create Subscription. We’ll need to give the subscription a name and pick “Source Computer Initiated” as that’s what makes it “push.”

Select Computer Groups next, as this will define which computers send us the events we’re interested in. Since we want to know whenever anyone anywhere clears a security log, we’re going to use the two built-in/auto populating groups in AD, “Domain Controllers” and “Domain Computers.”

Next we click Select Events to define what we’re monitoring. This alert is pretty straight forward, we’re looking for Event ID 1102 in the Security log, so we can do it all via the GUI. 🙂

If we click on the XML tab we can see what the XPath filter we just created looks like.

Click okay on that and then click Advanced on the main page. We want to set the subscription to Minimize Latency, to make sure we get events as soon as possible and also to help it catch up if we miss any.

You can troubleshoot that things have actually checked in by checking the Source Computers column on the main Subscriptions page :

 

You can also check the Event Forwarding Plugin Operational log under Applications and Services on the client to make sure everything is happy. This is where you’ll see descriptive errors if something has gone awry with Kerberos or Firewalls.

 

 

Click okay and you’ve got a subscription. Find a low-value test VM, clear the Security log and see if you get an alert. 🙂

 

Configure those 5 events with cut and paste for two commands? Yes you can!

Thanks to Australian PFE Russell Tomkins, you can do just that. Below are two XML files that contain the appropriate subscriptions for Domain Computers and Domain Controllers. The subscriptions here are maybe a bit wider than you want in your production domain to start with, as it’s collecting services installing on workstations too, but give it a go. We’ll want that data if we ever have to do an IR for you.

 

To import the XML files, save them to a directory on the server and then run the following commands from the same directory on your Windows Event Collector.

 

wecutil cs DomainComputers.xml

wecutil cs DomainControllers.xml

DomainComputers.xml


<Subscription xmlns=”http://schemas.microsoft.com/2006/03/windows/events/subscription”>
<SubscriptionId>Domain Computer Events</SubscriptionId>
<SubscriptionType>SourceInitiated</SubscriptionType>
<Description>Important Domain Controller Events</Description>
<Enabled>True</Enabled>
<Uri>http://schemas.microsoft.com/wbem/wsman/1/windows/EventLog</Uri>
<ConfigurationMode>MinLatency</ConfigurationMode>
<Query>
<![CDATA[<QueryList>
<Query Id=”0″ Path=”Security”>
<!– Local Admins Changed –>
<Select Path=”Security”>
*[EventData[Data[@Name=’TargetUserName’] and (Data=’Administrators’)]]
and
*[System[(EventID=’4732′) or (EventID=’4733′)]]
</Select>
<!– Local user created or deleted –>
<Select Path=”Security”>*[System[(EventID=’4720′) or (EventID=’4726′)]]</Select>
<!– New Service Installed –>
<!– Event Log Cleared –>
<Select Path=”Security”>*[System[(EventID=’1102′)]]</Select>
</Query>
</QueryList>]]>
</Query>
<ReadExistingEvents>true</ReadExistingEvents>
<TransportName>http</TransportName>
<ContentFormat>RenderedText</ContentFormat>
<Locale Language=”en-US”/>
<LogFile>ForwardedEvents</LogFile>
<AllowedSourceNonDomainComputers></AllowedSourceNonDomainComputers>
<AllowedSourceDomainComputers>O:NSG:NSD:(A;;GA;;;DC)(A;;GA;;;DD)</AllowedSourceDomainComputers>
</Subscription>

 

DomainControllers.xml

<Subscription xmlns=”http://schemas.microsoft.com/2006/03/windows/events/subscription”>
<SubscriptionId>Domain Controller Events</SubscriptionId>
<SubscriptionType>SourceInitiated</SubscriptionType>
<Description>Important Domain Controller Events</Description>
<Enabled>True</Enabled>
<Uri>http://schemas.microsoft.com/wbem/wsman/1/windows/EventLog</Uri>
<ConfigurationMode>MinLatency</ConfigurationMode>
<Query>
<![CDATA[<QueryList>
<Query Id=”0″ Path=”Security”>
<!– New Service Installed –>
<Select Path=”System”>*[System[(EventID=’7045′)]]</Select>
<!– Member Added or Removed from an AD Domain Local, Universal or  Global Security Group –>
<Select Path=”Security”>
(*[EventData[Data[@Name=”TargetUserName”] = “Administrators”]]) or
(*[EventData[Data[@Name=”TargetUserName”] = “Domain Admins”]]) or
(*[EventData[Data[@Name=”TargetUserName”] = “Schema Admins”]]) or
(*[EventData[Data[@Name=”TargetUserName”] = “Enterprise Admins”]]) or
(*[EventData[Data[@Name=”TargetUserName”] =  “Print Operators”]]) or
(*[EventData[Data[@Name=”TargetUserName”] = “Server Operators”]]) or
(*[EventData[Data[@Name=”TargetUserName”] = “Backup Operators”]])
and
*[System[(EventID=’4732′) or (EventID=’4733′) or (EventID=’4756′) or (EventID=’4757′) or (EventID=’4728′) or (EventID=’4729′)]]
</Select>
<!– Event Log Cleared –>
<Select Path=”Security”>*[System[(EventID=’1102′)]]</Select>
</Query>
</QueryList>]]>
</Query>
<ReadExistingEvents>True</ReadExistingEvents>
<TransportName>http</TransportName>
<ContentFormat>RenderedText</ContentFormat>
<Locale Language=”en-US”/>
<LogFile>ForwardedEvents</LogFile>
<AllowedSourceNonDomainComputers></AllowedSourceNonDomainComputers>
<AllowedSourceDomainComputers>O:NSG:NSD:(A;;GA;;;DD)</AllowedSourceDomainComputers>
</Subscription>

If you have more than one domain, you’ll need to specify the SIDs of Domain Computers/Domain Controllers in the <AllowedSourceDomainComputers> line as well as the DC (Domain Computers) and DD (Domain Controlls) SDDLs that are in there.

Hope this helps. Be sure to check out my post on tracking Special Groups with WEF and look forward to more things you can do with WEF in future blog posts! Happy Logging!

 

-Jessica @jepayneMSFT

 

Comments (42)

  1. adam says:

    Thank you so much Jessica (and Russel). Itching to get this implemented, even if I am the only one that will read the logs 🙂 Again great presentation.
    Didn't you like running on 230V AC through 20cent power adapters? 😛

  2. Ken says:

    The video of the presentation has now been posted, well worth watching.
    https://channel9.msdn.com/Events/Ignite/Australia-2015/INF327

  3. Anonymous says:

    Lateral Movement – the moving of an attacker from one compromised host throughout your domain until they

  4. Gavin says:

    Thanks again Jessica. I watched this at Ignite and watched the recording to refresh.

  5. gareth.douce says:

    Hallo – is this traffic sent plaintext across the network?

  6. Hi Gareth –

    The traffic is sent over WinRM, which is Kerberos (HTTP-Kerberos-session-encrypted) encrypted after an initial HTTP POST. So the data itself is not sent plaintext over the network. It’s really easy to see the encryption in Wireshark/Netmon if you filter for
    TCP Port 5985. You are welcome to set it up for HTTPS to make sure the entire session is negotiated encrypted, but the Kerberos auth prevents man in the middle and eavesdropping on the traffic if the HTTPS configuration would stand in the way of you rolling
    this out. If you use certificate authentication it uses that for the shared secret as well. So you can configure it to be even more robust, but it’s way better than using a UDP paintext connection like many solutions use right out of the box.

    Thanks for the good question, I’ll try to work a packet capture analysis about it into a later post!

    -Jessica

  7. David Howell says:

    Hey Jessica. I've been using this method since you led a POP-SLAM engagement a few months back. Glad to see you blogging!

    I wanted to throw this in for people interested. I started to work on a PowerShell module for Event Subscriptions, but it still needs a lot of work. It's basically a wrapped for wecutil, but was made with automation in mind.

    https://github.com/davidhowell-tx/PS-WinEventSubscriptions

    I also have a script posted that uses the module. It queries AD for Domain Admins membership and updates a subscription with the members' SamAccountName for monitoring.

  8. Anonymous says:

    Hi, Jessica Payne from Microsoft Enterprise Cybersecurity Group’s Global Incident Response and Recovery

  9. Hi David – glad you’ve been able to put the SLAM stuff into practice! You guys were super fun. (Embrace the 500 account!)

    Spoiler alert for the posts ahead, there’s a Powershell logging module at the end of it that ties into the PowerBI dashboards – the fact you made your own is super cool too!

    Keep up the good work and innovation. 🙂

    -Jessica

  10. Tom Bartlett says:

    This is really handy and I'm in the middle of implementing it however, we use group policy preferences to change the local admins group so everytime a machine boots up, it changes the membership of the group and forwards and event, meaning we have 300
    events every morning just from powering up. is there anyway to filter xpath to not monitor the groups added to local admin?

  11. Tony Pellegrini says:

    More great stuff from you, Jessica, thanks so much! I'm wondering if there's a benefit or negative to creating separate subscriptions vs. combining the filters into one event in terms of processing or traffic for either the source or destination servers?
    For example, if I also want to monitor the special groups (I do!) I could create a new subscription or I could modify the existing subscriptions with the additional filter elements. Is one solution empirically "better" or "worse" than another, or is it purely
    dealer's choice?

  12. Tom – there’s a way to filter based on who is doing the change, GPO should be done by SYSTEM so you could "suppress" those (knowing you might miss something.) The dashboard at the end of the series should help too. Are you doing adds or Restricted Groups?
    I’m a bigger fan of Restricted Groups.

    -Jessica

  13. Tony – I like to use different subscriptions so I can track if something goes wrong in the Event Forwarding Plugin operational log, but you can also use one big "security stuff" or "operational stuff" subscription for simplicity too. So sort of your choice?
    🙂
    -Jessica

  14. Tom Bartlett says:

    We're adding a security group of I.T. staff to the local admins group by GPP. Not sure how restricted groups work but if you could point me in the direction of a good article I can certain start researching.

    1. Tom – I use GPP the same as you. Delete all member users = Enabled. Delete all member groups = Enabled. Add members = The domain group I want added to the local PC administrators group. I’m sure sure of the technical differences between this and restricted groups but I actually switched from restricted groups to GPP when we upgraded domain functional level because GPP is easier to look at & manage.
      With this GPP in effect it was generating both Remove and Add events on every computer after every restart just like you described.
      I added this line to the event Subscription events to collect XML to suppress the changes made by local system when running the group policy as Jessica suggested.
      *[EventData[Data[@Name=”SubjectUserSid”] = “S-1-5-18”]]
      Now I only get an event when a human changes the group membership!
      @dconsec

      1. Update: It stripped some of the tags I posted. Lets try again.
        “*[EventData[Data[@Name=”SubjectUserSid”] = “S-1-5-18″]]”

      2. Suppress Path=”Security”>*[EventData[Data[@Name=”SubjectUserSid”] = “S-1-5-18”]]</Suppress

  15. Anonymous says:

    Hi, The Captain here from Microsoft Enterprise Cybersecurity Group’s Global Incident Response and

  16. Paul Bendall says:

    Jessica – great post and glad this topic seems to be getting more coverage. I have a OneNote notebook full of references trying to piece together a best practice approach.

    In all my research it would appear you can create an event log on the collector server other than the Forwarded Events, but I never seen a post on how. The reasoning behind this would be to push domain controller logs at one log with the corresponding file
    on a dedicated and performant disk (we generate around 750GB per day) and create other event logs for member servers and clients (again spreading out the disk IO load). Is this possible and if so how?

    Thanks

    Paul

  17. Paul – yes you can do that! It was hard to get working reliably but we got some help from MSIT on how they do it and have it being tested at some customers now. Should be part of an upcoming post once we make sure it is still reliable at 500k+ endpoints.

    1. Paul Bendall says:

      Jessica – that is great news and something I am keen to implement so as to scale WEF further. I’ll speak with our Microsoft ATS in the UK to hook into the testing and availability.

      1. For those wishing to create custom event logs, Russell Tomkins has blogged about the way we do it inside of Microsoft : https://blogs.technet.microsoft.com/russellt/2016/05/18/creating-custom-windows-event-forwarding-logs/

  18. Hi Jessica, I am looking forward to implementing WEF in my current company after seeing this presentation. We have been reviewing SIEM providers for sometime now and it’s a minefield for sure. I agree with some of your comments from the presentation that one needs to be clear about the type of events that are being logged. I believe there are a key list that everyone should focus as essential but each company should know what’s important to them beyond that. Can you recommend how to create a dashboard for the events?

  19. Thanks for this article Jessica, it is extremely helpful.
    This is simple but I stumbled so I thought I would share to help anybody else doing this.
    I glossed over the whole “WinRM must be started” part so my first attempt did nothing.
    Add this to your GPO to get it up and running:
    Computer > Policies > Windows Settings > Security Settings > System Services > Windows Remote Management (WS-Management) Startup = Automatic

    @dconsec

  20. Hi Jessica,

    I had a similar issue with HTTP SPN.
    And I had concluded that to make a denial of service of Event Forwarding you just have to do a setspn…

    What’s your thoughts on that ?
    How do you monitor that clients are still sending events to collector ?

    Regards
    Greg

  21. Michael W. says:

    Thanks for this post. Very easy to follow. Thank you!

    One thing — The tag in the DomainComputers.xml section looks like it might be incorrect as it mentions domain controllers.

  22. Joel says:

    Thank you so much, Jessica! You just fixed the last issue I was having setting this up.

    The MSDN docs don’t mention the access restrictions on the Security logfile, so I was a bit confused when some of our servers didn’t send their Security logs (but others did, go figure).

    So that other people can Google the error I was having now to find this article:
    Event 101, Eventlog-ForwardingPlugin
    Status SubscribePartialSuccess returned status data which included:

    1. David Kordyban says:

      Anybody else having issues importing the XML files to 2012 R2 collector? I keep getting Invalid config file. Error = 0x80070057

      1. Hi David – check out my reply to Joel for a fix, this is a known issue between versions.

  23. Sean B says:

    I’m copying the DomainControllers.xml but when I try to run the commend to import it I get:
    Invalid config file. Error = 0x80070057.
    The parameter is incorrect.

    I am guessing I’m missing something obvious… any help would be great

    1. There’s an error in the XML schema for 2012R2 since it was made on 2008R2. Best way to ensure it imports is to make a new subscription via GUI and export using :
      wecutil gs “%subscriptionname%” /f:xml >>”C:\Temp\%subscriptionname%.xml”

      You can then use that one as a template and just change the XPath portion.

      1. Pete says:

        This isn’t working for me still Jessica. I’ve created the subscription on a 2012R2 host, exported it, but then can’t import it again (even if i dont change the exported file!). Any ideas?

        1. Does the import display any error?
          -Jessica

  24. Colin says:

    Hi, thanks for the great article. Do we know if there’s a way to set the “Computer>Policies>Admin Templates>Windows Components>Event Forwarding>Configure target subscription manager” by command line?

  25. Joe Gomez says:

    Thanks for the article. Works fine, however occasionally the source computers will turn into an inactive state for apparantly no reason. See my reply to a thread about this here: https://social.technet.microsoft.com/Forums/en-US/3d62d46b-33e7-4db6-b672-8555fd6a9f35/event-log-forwarding-subscription-is-unsubscribed

  26. Ross says:

    What about environments where there are thousands of users forwarding events per day, is there built in options for load balancing between multiple WEC servers?

    1. Load balancing in WEF/WEC (in a supported way) is achieved by putting multiple collectors in the GPO – then it will send a copy of the event to each server, while maintaining the “bookmark” of which was sent last. While network load balancing solutions are possible, due to the bookmark manner or operation it will have the same net result of sending a copy to each server leading to duplicates.
      -Jessica

      1. Andrew says:

        We’re attempting this via network load balancing right now. We also have a single virtual address (VIP + FQDN + SPN) which fronts four event collectors on the backend.
        If the setup is source-initiated, I’m hoping that this wont mean that the backend servers would get duplicate events, since any bookmark being maintained by each client would only point to the virtual address.
        @Jessica – is this correct? is one bookmark maintained by the client for the last event sent, or is it maintained per last event sent to each server?

  27. Pete says:

    Hi Jessica, I decided to roll with WEF after reading this post! Any ideas as to what would give an eventid of 102 with error 5004? I’m testing the collection of sysmon logs from Microsoft-Windows-Sysmon/Operational but it keeps erroring with: The subscription Collect Sysmon Logs can not be created. The error code is 5004.

  28. Micah says:

    I assume there is no way to do wildcards since Microsoft is using XPath 1.0?

  29. kristoff says:

    Hi Everybody
    quick question…Is it possible to implement a WEC server to address multiple forest ? If yes, is there any technical aspects I should take care of ? (We had implemented a WEC servers, we can subscriptions from other forest, but logs are not forwarded..;despite GPO is applied)
    Thanks in advance for your inputs / Cheers / Kristoff

Skip to main content