IMPORTANT ANNOUNCEMENT FOR OUR READERS!
AskPFEPlat is in the process of a transformation to the new Core Infrastructure and Security TechCommunity, and will be moving by the end of March 2019 to our new home at https://aka.ms/CISTechComm (hosted at https://techcommunity.microsoft.com). Please bear with us while we are still under construction!
We will continue bringing you the same great content, from the same great contributors, on our new platform. Until then, you can access our new content on either https://aka.ms/askpfeplat as you do today, or at our new site https://aka.ms/CISTechComm. Please feel free to update your bookmarks accordingly!
Why are we doing this? Simple really; we are looking to expand our team internally in order to provide you even more great content, as well as take on a more proactive role in the future with our readers (more to come on that later)! Since our team encompasses many more roles than Premier Field Engineers these days, we felt it was also time we reflected that initial expansion.
If you have never visited the TechCommunity site, it can be found at https://techcommunity.microsoft.com. On the TechCommunity site, you will find numerous technical communities across many topics, which include discussion areas, along with blog content.
NOTE: In addition to the AskPFEPlat-to-Core Infrastructure and Security transformation, Premier Field Engineers from all technology areas will be working together to expand the TechCommunity site even further, joining together in the technology agnostic Premier Field Engineering TechCommunity (along with Core Infrastructure and Security), which can be found at https://aka.ms/PFETechComm!
As always, thank you for continuing to read the Core Infrastructure and Security (AskPFEPlat) blog, and we look forward to providing you more great content well into the future!
Hello all. Jacob Lavender here again for the Ask PFE Platforms team to share with you a little sample tool that I’ve put together to help with performing network captures. This all started when I was attempting to develop an effective method to perform network traces within an air gapped network. My solution had to allow me to use all native functionality of Windows without access to any network capture tools such as Message Analyzer, NETMON, or Wireshark. In addition, I’d need to be able collect the trace files into a single location and move them to another network for analysis.
Well, I know the commands. The challenge is building a solution that junior admins can use easily. Several weeks later I found the need for it again with another customer supporting Office 365. This process resulted in the tool discussed in this post.
Time and time again, it seems that we’ve spent a great deal of effort on the subject of network captures. Why? Because one of the first questions a PFE is going to ask you when you troubleshoot an issue is whether you have network captures. Same is true when you go through support via other channels. We always want them, seem to never get enough of them, and often they are not fun to get, especially when dealing with multiple end points.
So, let’s briefly outline what we’re going to cover in this discussion:
Topic #1: How to get the tool.
Topic #2: Purpose of the tool.
Topic #3: Requirements of the tool.
Topic #4: How to use the tool.
Topic #5: Limitations of the tool.
Topic #6: How can I customize the tool?
Topic #7: References and recommendations for additional reading.
Compatible Operating Systems:
- Windows 7 SP1
- Windows 8
- Windows 10
- Windows Server 2008 R2
- Windows Server 2012 R2
- Windows Server 2016
Topic #1: Where can I get this tool?
Topic #2: What is the purpose of this tool as opposed to other tools available?
This certainly should be the first question. This tool is focused toward delivering an easy to understand approach to obtaining network captures on remote machines utilizing PowerShell and PowerShell Remoting.
I often encounter scenarios where utilizing an application such as Message Analyzer, NETMON, or Wireshark to conduct network captures is not an option. Much of the time this is due to security restrictions which make it very difficult to get approval to utilize these tools on the network. Alternatively, it could be due to the fact that the issue is with an end user workstation who might be located thousands of miles from you and loading a network capture utility on that end point makes ZERO sense, much less trying to walk an end user through using it. Now before we go too much further, both Message Analyzer and Wireshark can help on these fronts. So if those are available to you, I’d recommend you look into them, but of course only after you’ve read my entire post.
Due to this, it is ideal to have an effective method to execute the built-in utilities of Windows. Therein lies NetEventSession and NETSH TRACE. Both of these have been well documented. I’ll point out some items within Topic #7.
The specific target gaps this tool is focused toward:
- A simple, easy to utilize tool which can be executed easily by junior staff up to principle staff.
- A means by which security staff can see and know the underlying code thereby establishing confidence in its intent.
- A lite weight utility which can be moved in the form of a text file.
With that said, this tool is not meant to replace functionality which is found in any established tool. Rather it is intended to provide support in scenarios where those tools are not available to the administrator.
Topic #3: What are the requirements to utilize this tool?
- An account with administrator rights on the target machine(s).
An established file share on the network which is accessible by both
- The workstation the tool is executed from, and
- The target machine where the trace is conducted
Microsoft Message Analyzer to open and view the ETL file(s) generated during the trace process.
- Message Analyzer does not have to be within the environment the traces were conducted in. Instead, the trace files can be moved to a workstation with Message Analyzer installed.
Remote Management Enabled:
- winrm quickconfig
- winrm quickconfig
Note: Technically, we don’t have to have Message Analyzer or any other tool to search within the ETL file and find data. However, to do so, you must have an advanced understanding of what you’re looking for. Take a better look at Ed Wilson’s great post
from the Hey, Scripting Guy! Blog:
Topic #4: How do I use this tool?
Fortunately, this is not too difficult. First, ensure that the requirements to execute this tool have been met. Once you have the tool placed on the machine you plan to execute from (not the target computer), execute the PS1 file.
PFE Pro Tip: I prefer to load the file with Windows PowerShell ISE (or your preferred scripting environment).
Note: You do not have to run the tool as an administrator. Rather, the credentials supplied when you execute the tool must be an administrator on the target computer.
Additional Note: The tool is built utilizing functions as opposed to a long script. This was intentional as to allow the samples within the tool to be transported to other scripts for further use – just easier for me. While I present the use of the tool, I’ll also discuss the underlying functions.
Now, that I have the tool loaded with ISE, let’s see what it looks like.
- The first screen we will see is the legal disclaimer. These are always the best. I look forward to executing tools and programs just for the legal disclaimers. In my case, I’m going to accept. I will warn you that if you don’t accept, then the tool will exit. I’m sure you’re shocked.
Ok, now to the good stuff. Behind the scenes the tool is going to clear any stored credentials within the variable $credentials. If you have anything stored in that variable within the same run space as this script, buckle up. You’re going loose it. Just FYI.
- Next, the tool is now going to ask you for the credentials you wish to use against the target computer. Once you supply the credentials, the tool is going to validate that the credentials provided are not null, and if they are not, it will test their validity with a simple Get-ADDomain query. If these tests fail, the tool will wag the finger of shame at you.
- After supplying the credentials, we will be asked to supply a file share to move the capture files.
Note: The file share must be accessible from both the local client and the target computers. Here is why:
The tool is going to validate that the path you provided is available on the network. I’m assuming that after the capture is complete you will want to have access to the files. However, if the local machine is unable to validate the path, it will give you the option to force the use of the path.
- Second, the tool is going to attempt to validate the file share path on the target computer. If the path is not accessible by that computer, it will give you the option to update the path. If you do not update the path it will leave a copy of the trace files on the target computer.
- Next, we will specify the target machine. Once you specify the machine, the tool will validate this machine with DNS by performing a query. If the query fails, you will have to correct the machine. The assumption is that if the query fails, the machine won’t be accessible by FQDN (probably a safe assumption unless you’re using a hosts file, which is outside the scope of this guide).
- Next, we will specify for how long we want the network capture to run. The value is in seconds.
Note: As stated by the tool, capture files can take up a great deal of space. However, the defaults within the tool are not very large.
You can customize the values of the network captures. The commands are located within the Start-NETSH and Start-Event functions.
For the purpose of this tool, I utilized the defaults with NO customization.
Now, once we hit enter here, the tool is going to setup a PowerShell session with the target machine. In the background, there are a few functions its doing:
- It establishes a PSSession.
- It establishes the boot volume drive letter.
- It sets a working path of <bootvolume>:\TEMP\Tracefiles. If this path does not exist, it creates it.
Next, we must specify a drive letter to use for mounting the network share (from Step 4). State any drive letter you want that isn’t already in use.
Now, you might be asking why are we mounting a drive letter instead of using the Copy-Item command to the network path. Yeah, I tried that without thinking about it and got a big giant ACCESS DENIED. This is due to the fact that we can’t double-hop with our credentials. Kerberos steps in and screams HALT! HALT WITH YOUR DOUBLE-HOP COMMAND!
Great article discussing this problem:
If you read the article, you’ll see there are multiple ways to address this. I opted for the simple path of just mounting the network share as a drive letter. Simple. Easy. Can be used again without special configuration of computers, servers, or objects in AD. Keep it simple, right? Additionally, we want to minimize any special configuration of systems to accomplish this.
Now, again in the background the tool is performing a little extra logic:
- It first validates that a drive is not already mounted with the network path provided from Step 4. That would be silly to do it twice.
- Next, once you provide a drive letter, it validates that you didn’t select one already in use.
Great. Now to the really good stuff.
- Our next screen presents us with the option to select the capture method we wish to use. Both have advantages and disadvantages. See the references section for details on these. Really, you should read those articles before selecting which capture method if you are not already familiar with them.
For this example, I’m selecting N for NETSH TRACE. NETSH TRACE provides a CAB file by default which I’ll show you in Step 10.
Again, we have some behind the scenes logic happening.
Windows 7 and Windows Server 2008 R2 do not have the NetEventSession option available. So, the utility is going to establish what version of Windows the target computer is. If the computer is either Win7 or W2K8R2 it will not allow you to use NetEventSession. It will force the use of NETSH TRACE upon you.
NOTE: Also note that the utility is going to provide a report to you at the end of execution. Within that report it includes the running processes on the target computer.
Well, one of my favorite features of NETMON and Message Analyzer is the conversation tree. I like to know which of my applications are talking and to who. This is performed on the backend by the application to map PIDS to executables. Well, the capture file might not tell me the executable, but it does give me the PID. So, by looking at the report I can identify which PID to focus on and then use that when looking at the network trace file in Message Analyzer. Yay.
- Ok, as soon as we selected which capture method we were going to use, the tool executes the capture on the remote computer and it runs the capture for the length of time previously specified.
As you can see, it states the location. On the target computer we can even see the temporary files which are put in place for the capture:
Once the specified time is reached, the utility sends a stop command to the target computer to end the network capture:
NOTE: In the event that the utility is disconnected from the target computer prior to the stop command being issued, you can issue the commands locally at the target computer itself:
- NETSH TRACE: netsh trace stop
- PowerShell NetEventSession: Get-NetEventSession | Stop-NetEventSession
Finally, the tool will move the files used for the trace to the specified network share, and then remove them from the target computer.
- Next, we see that the tool completed its network trace and has placed a report for us in the C:\Temp directory on the local machine we ran the tool from.
If we open that report file, we’re going to be presented with this (there are more than two processes within the actual report) :
- Finally, we are now set to utilize the ETL files as necessary. In my case, I’ve opened an ETL that was generated on a Windows Server 2008 R2 computer using NETSH TRACE, and I’m looking at the LSASS.EXE process. 100 extra points if you can identify what this process is responsible for.
And finally, what’s in that CAB file? Lots of goodies. You’re going to want to explore that to better understand all the extra information which is provided about the system from this file.
Topic #5: What are the limitation of the tool?
The tool, at present, can only target a single computer at a time. If you need to target multiple machines, you will need to run a separate instance for each. (Multiple PowerShell Sessions)
I would recommend getting each instance to the point of executing the trace, and then do them all at the same time if you are attempting to coordinate a trace amongst several machines.
- I’m hoping to release a new version in the future which has the correct arrays and foreach loops built. We’re just not there yet.
- The variables within the script utilize memory space within the script. They are not set to global. However, I haven’t tested this scenario in depth so I would recommend giving that a test prior to trying against production machines.
Again, the tool is not meant to replace any other well-established application. Instead, this tool is meant only to fill a niche. You will have to evaluate the best suitable option for your purposes.
- The NETSH TRACE and NetEventSession have not been customized. This was intentional. I highly recommend that you read some of the additional content found in Topic #6 regarding the scenarios and advanced configuration options available within these commands.
Topic #6: How can I customize the tool?
Well, we do need to address some customization options. To do so, you simply need to modify the command invoked against the target computer within the trace type’s respective function. The function names are called out below.
NETSH TRACE Customization
First, let’s start with NETSH TRACE. Yong Rhee has a great article discussing some of the functionality within NETSH TRACE, specifically he uses scenarios:
Using NETSH to Manages Traces:
Let’s look at some of the built-in scenarios. To do so, execute netsh trace show scenarios:
Next, we can view some of the configuration of the providers within the scenarios using netsh trace show scenario <scenario name>, such as netsh trace show scenario LAN:
From this, we can see that one of the providers is Microsoft-Windows-L2NACP, which is currently configured to event logging level (4), Informational. Well, what if I wanted to configure that to be higher or lower. I can customize the NETSH TRACE command to accommodate this:
netsh trace start Scenario=Lan Provider=Microsoft-Windows-L2NACP Level=5 Capture=Yes TraceFile=$tracefile
This would increase the logging level to (5), Verbose:
Note: This is just one sample of how the NETSH TRACE option within the tool can be customized. There are plenty of other options as well. I strongly recommend that you review Netsh Commands for Network Trace:
Fundamentally, this is going to be the same as customizing NETSH TRACE. You simply have to know what you’re looking for. In this case, we are going to focus on two aspects.
Configuring the NetEventSession: This overall is simple. As a whole we’re not going to change too much on this. I’d recommend reviewing the New-NetEventSession documentation:
Now, the real meat of the capture. The NetEventProvider. The default used natively within the tool is the Microsoft-Windows-TCPIP provider. However, there are quite a few others available. You may want to output to a file as there will be several.
From PowerShell, execute:
What you should notice is that the providers are all set with a default configuration. You can adjust these as necessary as well using:
By adding an additional Invoke-Command line within the Start-NetEvent function, you can easily customize the provider(s) which you wish to use within the network capture session.
Customization Conclusion: For both NETSH TRACE and NetEventSession, I would recommend making adjustments to the commands locally on a test machine and validating the results prior to executing against a remote machine. Once you know the command syntax is correct and the output is what you desire then incorporate that customization back into the tool as necessary.
Topic #7: References and Recommendations for Additional Reading:
Learning how to use Message Analyzer:
- These two posts by Lakshman Hariharan and his fellow co-conspirators will help you greatly understand what you can now do once you’ve acquired the trace:
Introduction to Network Trace Analysis Using Microsoft Message Analyzer:
Part 1: https://blogs.technet.microsoft.com/askpfeplat/2014/08/18/introduction-to-network-trace-analysis-using-microsoft-message-analyzer-part-1/
- Part 2: https://blogs.technet.microsoft.com/askpfeplat/2014/10/12/introduction-to-network-trace-analysis-using-microsoft-message-analyzer-part-2/
Michael Rendino’s two posts:
- Basic Network Capture Methods: https://blogs.technet.microsoft.com/askpfeplat/2016/12/27/basic-network-capture-methods/
- Network Capture Best Practices: https://blogs.technet.microsoft.com/askpfeplat/2017/04/04/network-capture-best-practices/
Victor Zapata’s post on Leveraging Windows Native Functionality to Capture Network Traces Remotely:
- A note on this post. It includes some sample material on running traces against multiple machines at once. I’d recommend exploring this a little.
Yong Rhee’s two posts:
Network Tracing built-in to Windows and Windows Server: https://blogs.technet.microsoft.com/yongrhee/2012/12/01/network-tracing-packet-sniffing-built-in-to-windows-server-2008-r2-and-windows-server-2012/
So you want to use Wireshark to read the NETSH TRACE output.etl : https://blogs.technet.microsoft.com/yongrhee/2013/08/16/so-you-want-to-use-wireshark-to-read-the-netsh-trace-output-etl/
Microsoft Message Analyzer Tutorial:
PowerShell Remoting Kerberos Double Hop Solved Securely:
Packet Sniffing with PowerShell: Looking at Messages: