From time to time, we come across issues where capturing a network trace is critical to determining root cause for a problem and even more important… a solution. Simple right? Just start Network Monitor or WireShark and reproduce the problem. Things can’t get much easier than that!
The reality is that sometimes it’s just not that simple. It’s not uncommon to hit an issue where unexpected failures occur at random, unpredictable times. This poses a real challenge when it comes to collecting the right data at the right time.
There’s good news! If the problem happens to log something to the Event Log, things just got a whole lot easier for us. Beginning in Windows Vista/Server 2008, the Event Viewer now includes functionality that allows us to “attach a task” to any event. This comes in super-handy when capturing an open-ended network trace because we can now programmatically stop the network trace when a specific Event is logged in any of the Event Logs.
The setup to programmatically stop a network trace consists of a two-part process that utilizes the following components:
– Network Monitor
– Event Viewer
– Task Scheduler
The two-part process looks like this:
Part A: Performed by Network Monitor
– Capture network data and watch for a specific “pattern” on the network
– Stop the network trace when the “pattern” is detected
Part B: Performed by Event Viewer & Task Scheduler
– Watch the Event Logs for a specific Event ID
– When the Event ID is logged, “do something” that will generate the “pattern” that Network Monitor is waiting for to stop tracing.
In this example, the “do something” will be to query for non-existent host name “stopthetrace”.
Now that we have an idea of what we need to do… let’s get started.
We’ll need to setup the following 3 items in order to get this going:
1. Start the network capture with appropriate triggers to stop the trace automatically
2. Create a batch file that will query DNS for “stopthetrace”
3. Configure a “task” in Event Viewer that will execute the batch file when a specific Event ID is logged.
STEP 1: DOWNLOAD & INSTALL NETWORK MONITOR
This is simple. Just make sure to right-click the installer and “run as administrator”
This ensures that the Network Monitor driver successfully binds to all network interfaces.
Network Monitor 3.4
STEP 2: CREATE A BATCH FILE THAT WILL GENERATE A DNS QUERY FOR “STOPTHETRACE”
In order to verify that we configured the appropriate filter in the NMCAP command (we’ll do this in step 4, below), we can simply run “Nslookup StopTheTrace” from a command prompt while running a network trace. Once we’ve captured the DNS name resolution for “stopthetrace”, we can simply expand all of the DNS header in the Frame Details and right-click on the “QuestionName” field and select “Add Selected Value to Display Filter”. This results in the following filter:
DNS.QRecord.QuestionName == “stopthetrace.CORP.CONTOSO.MSFT” (Note that the primary DNS suffix is automatically appended to the name)
Now that we have the filter syntax, we have 2 options here:
A. We can use the filter “as is” if we specify the Fully Qualified Domain Name (FQDN) in the DNS query
DNS.QRecord.QuestionName == “stopthetrace.CORP.CONTOSO.MSFT”
B. We can use the Contains() function built in to Network Monitor to look for any DNS query “containing” string matching “StopTheTrace”
For the purposes of this example, we’ll create a batch file in C:\Scripts and name it STOPTHETRACE.BAT
The batch file will simply contain the following command:
STEP 3: ATTACH A TASK TO A SPECIFIC EVENT ID
Next, we’ll attach a task to a specific event that can be logged in any one of the Event Logs (e.g. System, Application, Security, etc.)
In this example, we want to capture network data until Event ID 5719 is logged in the System Event log.
With Event ID 5719 highlighted, select “Attach a Task to this Log”
In the task wizard, we’ll specify a name and description for the task that we’re creating… click “Next”
The next dialog will default to the appropriate required settings. We’ll keep the defaults here… click “Next”
On the Action dialog, we’ll select “Start a program”… click “Next”
We’ll specify the location of the batch file that we created on the “Start a Program” window… click “Next”
On the Summary window, select the option “Open the Properties dialog for this task when I click Finish”… click “Finish”
Notice that the wizard simply creates a Scheduled Task.
The properties dialog for the task looks just like any other Scheduled Task property window.
In this example, we’ve specified that the batch file should run as SYSTEM… click “OK” to close.
By default, the Event Viewer task will automatically stop after 3 days and will not execute additional instances.
These options are configurable. We can simply uncheck the “auto stop” option to run the task indefinitely if needed.
Note: We should now see a new task under “Event Viewer Tasks” within Task Scheduler
Check out Michael Hildebrand’s “PFE Troubleshooting Series” blog which outlines more Event Log goodies.
PFE Troubleshooting Series
STEP 4: START THE NETWORK CAPTURE USING NMCAP.EXE
NMCAP.exe is simply the command-line version of NetMon. In this example, we’re going to start a network trace using a circular buffer of 50 Mb. The data collected will be saved to a file named NetworkCapture1.cap on the root of the C:\ drive. The capture will continue to run until the specified pattern match is detected.
nmcap /Network * /Capture /File c:\NetworkCapture1.cap:50M /StopWhen /Frame dns.qrecord.questionname.Contains(‘stopthetrace’)
Note: Make sure to use single quotes when specifying a string in the Contains() function.
Run “nmcap /?” to get more detail related to each switch used here.
In order to be able to configure NMCAP.EXE to continue running after we log off, we’ll need to run it as a Scheduled Task.
We’ll create a batch file named NETTRACE.BAT that contains the NMCAP syntax that we’ve come up with.
Note: Unless we create the batch file in the same location as NMCAP.EXE, we’ll need to specify the full path to NMCAP.EXE.
Next, create a basic Scheduled Task that runs NETTRACE.BAT
Configure the task to run as SYSTEM
Right-click the task and select “Run”
Since the trace is running under the SYSTEM context, we won’t see it interactively on our desktop.
We can use Task Manager to verify that NMCAP.EXE is running.
To verify that the trace will actually stop when the batch file is executed, we can manually run the batch file named STOPTHETRACE.BAT.
We’ll need to confirm that we no longer see NMCAP.EXE running in Task Manager and that a network trace file was successfully created.
At this point, simply start NMCAP.EXE again using the Task Scheduler, log off… and go have dinner!