How to measure Security product(s) overhead using FltMgr minifilter etw tracing on “Windows 7 SP1” and “Windows Server 2008 R2”.
Posts in this blog are provided “AS IS” with no warranties, and confers no rights as specified in the Terms of Use.
Like everything in an Enterprise, they should be tested in a test environment before trying in a production system.
Changed title from:
Windows 7 SP1 and Windows Server 2008 R2 Steps needed to capture Minifilter ETW traces
to:
How to measure Security product(s) overhead using FltMgr minifilter etw tracing on “Windows 7 SP1” and “Windows Server 2008 R2”.
Published 171108, Updated 171113.
Applies to:
Windows Server 2008 R2 SP1
Windows 7 SP1
As more security products are being installed to protect legacy OS’es such as “Windows 7 SP1” and “Windows Server 2008 R2 SP1”, IT Administrators have been troubleshooting slowness introduced by the kernel level filtering of Disk I/O and Network I/O.
Bob Golding (GES EE) wrote the following blog post:
Hotfix to Enable Mini-Filter Performance Diagnostics With XPerf for Windows Server 2008R2
https://blogs.technet.microsoft.com/supportingwindows/2012/05/31/hotfix-to-enable-mini-filter-performance-diagnostics-with-xperf-for-windows-server-2008r2/
Here is a summary of what’s needed.
1) Windows 8.1 ADK or Windows 8.1 SDK
The Windows 10 ADK or Windows 10 SDK do not work with these legacy systems.
For more info:
Windows 10 ADK/SDK WPT is not compatible w/ Windows 7 SP1 or Windows Server 2008 R2 SP1.
https://blogs.technet.microsoft.com/yongrhee/2017/11/13/windows-10-adksdk-wpt-is-not-compatible-w-windows-7-sp1-or-windows-server-2008-r2-sp1/
2) Set the “DisablePagingExecutive” and set it to 1.
For more info:
WPT: WPR/Xperf: Capture high cpu, disk i/o, file, registry, networking, Private bytes, Virtual bytes, Paged Pool/Nonpaged pool and/or application slowness
https://blogs.technet.microsoft.com/yongrhee/2012/11/23/wpt-wprxperf-capture-high-cpu-disk-io-file-registry-networking-private-bytes-virtual-bytes-paged-poolnonpaged-pool-andor-application-slowness/
Note: You might as well include into the image (WIM).
3) You need to install:
2666390 A hotfix that lets you diagnose mini-filter-based performance issues in Windows Server 2008 R2 is available
https://support.microsoft.com/?id=2666390
Note 2: This is newer (2015):
3027577 "0x0000007F" Stop error when you run both data encryption and antivirus software in Windows 7 or Windows Server 2008 R2
https://support.microsoft.com/?id=3027577
Supersede(s):
2965917 A computer freezes during startup after filter drivers are installed in Windows 7 or Windows Server 2008 R2
https://support.microsoft.com/?id=2965917
2979933 Sharp increase in CPU usage occurs when a folder is renamed in Windows
https://support.microsoft.com/?id=2979933
2885616 Filter Manager Verifier unexpectedly flags a violation after a mini-filter performs a reparse operation in Windows
https://support.microsoft.com/?id=2885616
2710594 Memory leak when a file system minifilter driver calls the FltCreateSystemVolumeInformationFolder function in Windows 7 or in Windows Server 2008 R2
https://support.microsoft.com/?id=2710594
2666390 A hotfix that lets you diagnose mini-filter-based performance issues in Windows Server 2008 R2 is available
https://support.microsoft.com/?id=2666390
Note 2: You might as well include into the image (WIM).
Note 3: Steps 2 and 3 require a reboot.
4) You cannot use WPRUI.exe or WPR.exe to collect the MiniFilter data in the legacy OS’es such as Windows 7 SP1 or Windows Server 2008 R2 SP1.
Minimum to gather Mini-Filter information:
xperf -start -on PROC_THREAD+LOADER+FLT_IO_INIT+FLT_IO+FLT_FASTIO+FLT_IO_FAILURE+FILENAME -stackwalk MiniFilterPreOpInit+MiniFilterPostOpInit -BufferSize 1024 -MaxBuffers 512 -MaxFile 4096 -FileMode Circular -f c:\temp\kernel.etl
timeout.exe /t 60
xperf -stop -d c:\temp\%computername%_MiniFilterETWTraceOutput.etl
Medium to gather Mini-Filter information:
xperf -on PROC_THREAD+LOADER+CSWITCH+FILENAME+FILE_IO+FILE_IO_INIT+DRIVERS+FLT_IO_INIT+FLT_IO+FLT_FASTIO+FLT_IO_FAILURE -stackwalk CSwitch+DiskReadInit+DiskWriteInit+DiskFlushInit+FileCreate+FileCleanup+FileClose+FileRead+FileWrite+MiniFilterPreOpInit+MiniFilterPostOpInit -BufferSize 1024 -MaxBuffers 512 -MaxFile 4096 -FileMode Circular -f c:\temp\kernel.etl
timeout.exe /t 60
xperf -stop -d c:\temp\%computername%_MiniFilterETWTraceOutput.etl
Note: It’s missing “CPU Sampled” and “Disk I/O”.
To get a full (holistic) mini-filter etw trace, you will use:
:: Start a local Perfmon
logman.exe create counter %ComputerName%_short_interval -f bincirc -v mmddhhmm -max 350 -c "\Cache\*" "\IPv4\*" "\LogicalDisk(*)\*" "\Memory\*" "\Netlogon\*" "\Network Interface(*)\*" "\Paging File(*)\*" "\Per Processor Network Activity Cycles(*)\*" "\Per Processor Network Interface Card Activity(*)\*" "\Processor(*)\*" "\Processor Information(*)\*" "\PhysicalDisk(*)\*" "\Process(*)\*" "\Physical Network Interface Card Activity(*)\*" "\Redirector\*" "\SMB Client Shares\*" "\SMB Server Shares(*)\*" "\SMB Server Sessions\*" "\Server\*" "\Server Work Queues(*)\*" "\System\*" "\VM Memory(*)\*" "\VM Processor(*)\*" "\TCPv4\*" -si 00:00:02
logman.exe start %ComputerName%_short_interval
xperf -on PROC_THREAD+LOADER+CSWITCH+FILENAME+Profile+Interrupt+DPC+Hard_Faults+Disk_IO+Disk_IO_Init+Split_IO+Perf_Counter+Power+MemInfo+MemInfo_WS+FILE_IO+FILE_IO_INIT+DRIVERS+FLT_IO_INIT+FLT_IO+FLT_FASTIO+FLT_IO_FAILURE -stackwalk CSwitch+DiskFlushInit+DiskReadInit+DiskWriteInit+KernelQueueEnqueue+KernelQueueDequeue+Profile+ReadyThread+FileCreate+FileCleanup+FileClose+FileRead+FileWrite+MiniFilterPreOpInit+MiniFilterPostOpInit -BufferSize 1024 -MaxBuffers 512 -MaxFile 4096 -FileMode Circular -f c:\temp\kernel.etl
Xperf -start User -on 8c493695-3df4-40cb-b11d-9edc41d5d2ab:0x0:0x4:'stack'+9580d7dd-0379-4658-9870-d5be7d52d6de:0x0000000000000200:0xff:'stack'+0a002690-3839-4e3a-b3b6-96d8df868d99:0x0:0x5:'stack'+d8975f88-7ddb-4ed0-91bf-3adf48c48e0c:0x0:0x4:'stack'+bf406804-6afa-46e7-8a48-6c357e1d6d61:0x0000000000000003:0xff:'stack'+49c2c27c-fe2d-40bf-8c4e-c3fb518037e7:0x0:0xff:'stack'+e7ef96be-969f-414f-97d7-3ddb7b558ccc:0x0000000000002000:0xff:'stack'+315a8872-923e-4ea2-9889-33cd4754bf64:0x0000000000100000:0x4:'stack'+36b6f488-aad7-48c2-afe3-d4ec2c8b46fa:0x0000000000010000:0xff:'stack'+d49918cf-9489-4bf1-9d7b-014d864cf71f:0x0:0xff:'stack'+331c3b3a-2005-44c2-ac5e-77220c37d6b4:0x0000000000000004:0xff:'stack'+57277741-3638-4a4b-bdba-0ac6e45da56c:0x0000000000000001:0xff:'stack'+e13c0d23-ccbc-4e12-931b-d9cc2eee27e4:0x0000000000000098:0x5:'stack'+751ef305-6c6e-4fed-b847-02ef79d26aef:0x0:0xff:'stack'+cfeb0608-330e-4410-b00d-56d8da9986e6:0x0:0xff:'stack'+8e92deef-5e17-413b-b927-59b2f06a3cfc:0x0:0xff:'stack'+e4b70372-261f-4c54-8fa6-a5a7914d73da:0x0:0xff:'stack'+83ed54f0-4d48-4e45-b16e-726ffd1fa4af:0x0:0xff:'stack'+6ad52b32-d609-4be9-ae07-ce8dae937e39:0x0:0x4:'stack'+8c416c79-d49b-4f01-a467-e56d3aa8234c:0x0000000000402000:0xff:'stack'+e6835967-e0d2-41fb-bcec-58387404e25a:0x0000000000000001:0xff:'stack'+a669021c-c450-4609-a035-5af59af4df18:0x0000000000000118:0x0:'stack'+8c416c79-d49b-4f01-a467-e56d3aa8234c:0x0000000000080000:0xff:'stack'+e7ef96be-969f-414f-97d7-3ddb7b558ccc:0x0000000000080000:0xff:'stack'+0f67e49f-fe51-4e9f-b490-6f2948cc6027:0xffffffffffffffff:0xff:'stack'+2a274310-42d5-4019-b816-e4b8c7abe95c:0x0000000000000010:0xff:'stack'+a6ad76e3-867a-4635-91b3-4904ba6374d7:0x0000000000000010:0xff:'stack'+ce8dee0b-d539-4000-b0f8-77bed049c590:0xffffffffffffffff:0xff:'stack'+c4636a1e-7986-4646-bf10-7bc3b4a76e8e:0xffffffffffffffff:0xff:'stack'+1babefb4-59cb-49e5-9698-fd38ac830a91:0xffffffffffffffff:0xffffffff:'stack'+8E9AC05F-13FD-4507-85CD-B47ADC105FF6:0xffffffffffffffff:0xffffffff:'stack'+DEDADFF5-F99F-4600-B8C9-2D4D9B806B5B:0xffffffffffffffff:0xffffffff:'stack'+A103CABD-8242-4A93-8DF5-1CDF3B3F26A6:0xffffffffffffffff:0xffffffff:'stack'+F5D05B38-80A6-4653-825D-C414E4AB3C68:0xffffffffffffffff:0xffffffff:'stack'+6B4DB0BC-9A3D-467D-81B9-A84C6F2F3D40:0xffffffffffffffff:0xffffffff:'stack'+C4636A1E-7986-4646-BF10-7BC3B4A76E8E:0xffffffffffffffff:0xffffffff:'stack'+CB587AD1-CC35-4EF1-AD93-36CC82A2D319:0xffffffffffffffff:0xffffffff:'stack'+cc7b00d3-75c9-42cc-ae56-bf6d66a9d15d:0xffffffffffffffff:0xffffffff:'stack'+90742496-54a9-4075-8079-425399080068:0xffffffffffffffff:0xffffffff:'stack'+Microsoft-Windows-TCPIP:0xffffffffffffffff:0xffffffff:'stack' -BufferSize 1024 -MinBuffers 320 -MaxBuffers 320 -MaxFile 4096 -f c:\temp\User.etl
timeout.exe /t 60
xperf -stop -stop User -d c:\temp\%computername%_MiniFilterETWTraceOutput.etl
::Stop perfmon
logman.exe stop %ComputerName%_short_interval
Yong
More information:
816071 How to temporarily deactivate the kernel mode filter driver in Windows
https://support.microsoft.com/?id=816071