Needle in a Hayfield of Haystacks

Your customer is down and your task is to understand why. You’ve been graciously supplied 8 gigs of traces, text logs, sniffs, captures, syslogs, configurations, ad nauseam. Even these have been curated from terabytes of available trace data. The names and formats are different, but they all correlate to the time period during the problem time period. The problem is vague; “it gets slow at night”, “some web pages are slow”, and “Netflix looks grainy”, and so on. Like a haystack, each file, contains volumes of data about your system. Some of it important, most of it not. Like stacks in a hayfield, each log file might be relevant, or it might not be.

Which Hayfield?

Haystacks can be very different from each other. But I imagine that Hayfields are like force fields. They have dimensions we can quickly measure and use to organize your haystacks and limit which piece of each haystack we need to comprehend. The phrase “at night”, means you pick out a certain time to focus on, which can be accomplished quickly in most cases. Further, “web pages are slow” means that you can filter on some ports to limit the data you will inspect more closely.

One Haystack

Once you decide to focus on a time period and/or filter, you limit the data. The goal is to optimize your view of the 8 gigs of data based on some specified criterion that makes your data manageable. You can use Message Analyzer’s Browse Select View (BSV) features to organize and focus the data that you want to import to simplify your analysis process. Some methods that Message Analyzer provides to focus your data view are as follows:

1. Time range – If you know the time at which a problem occurred, you can limit the data you need to review. This speeds up the import process and shrinks the amount of data held in memory, which in turn, makes analysis faster and more targeted.

2. Selection filter – You can also use a port, IP address, or any other filter to narrow down the data that you import. While a Selection Filter may not reduce import time, the end result is still less data in memory and a subsequent improvement in your analysis process.

3. User defined parsers – In some cases you only need to troubleshoot the network. By limiting parsing to the networking protocol TCP, you can simplify and speed up your analysis process.

It’s Still a Haystack

The next step is multi-faceted. You can use a domain-specific system of visualizations and pattern recognition to identify issues, for example, a set of known patterns for network issues such as TCP Exponential Backoff or Blocked ports. Alternatively, you can use a chart to visualize TCP performance, for example, by utilizing a chart that is similar to the following experimental Stevens graph:


Now you can drill down further by zooming into various time slots or by selecting a suspect result in greater detail using the grid. Each domain has its different techniques and to accommodate that, we will continue to provide helpful views such as the one above, as we move forward.

For Networking, the graph above provides a lot of information, however, a graph of a thousand lines might not be too helpful and it also could be slow. There is still a haystack of information you need to sift through. So, what else can you do? In most cases, you will need to filter/group your data even more. Alternatively, you might need to correlate a piece of information from a different log and then look at which messages surround it. Again, this is very domain specific, however the process is often the same across disciplines.

There’s Haystacks in the Haystacks?!?!?!

This is insane! Even a single network stack is a layering of different protocols, each one to some extent autonomous from each other. With Message Analyzer, only the top-level messages are exposed, while the lower layers are hidden to create a top-down approach that makes analysis easier and limits noise. So, how can you navigate this wee-haystack? Well, that’s where Viewpoints come in to facilitate a layered approach to analyzing the network stack. Viewpoints let you switch which protocol layers you want to focus on.

Visual Tour of Viewpoints

I’ve been trying to find ways to explain Viewpoints. I’ve written the following related blogs:

In the section that follows, I provide what I hope is the best explanation so far as it also provides a point of reference for Network Monitor and Wireshark users.

The list of messages on the left in black represents the lines you see in your protocol analyzer. We start with a timestamp, Conversation ID based on the TCP/IP connection, and then a short summary. I add (INC) to show that the message is incomplete.

Next to the list of messages, we see a visualization of the data. Each stacked column represents a network stack. Each column is of different conversations that are represented by an IP or Ethernet address pair with a TCP port pair, if present. Each row of network stacks in the z-axis occur at a different interval of time, starting with 1s.

We’ll start with the Network Monitor view where we have sixteen messages, because we have a 4×4 matrix. Each message is represented in the message listing. The HTTP Response is in two HTTP Chunks, and the SMB Read is actually contained in the 2 TCP fragments that follow, since Network Monitor doesn’t reassemble messages by default.


Next is the Wireshark view. The primary difference is that reassembly is done for you in some situations, but the last fragment contains the full payload. We still see a total of 16 frames listed.


For Network Monitor, we provided a button to reassemble data. This results in inserting new reassembled frames, we call DecryptedPayloadHeader. In this case I’ve removed all the ARP traffic to simplify things, but as you can see, two new complete and reassembled messages are now inserted.


On the other hand, Message Analyzer tries to summarize as much as possible. Even with the noise from TLS and ARP, you can see the HTTP and SMB request and response pairs encapsulated as two separate high-level operation messages (bars) in Message Analyzer. We can also measure the length of these bars to understand how long it took to complete an SMB or HTTP Request/Response. This is a powerful feature by itself, which can help you discover performance issues.


Next you can expand operations by clicking the Hide Operations button in the Viewpoints group on the Ribbon of the Message Analyzer Home tab. This removes the encapsulation of request and response pairs from the top-level operation nodes to expose the actual time orientation of the request and response messages. Now you will see the HTTP and SMB messages as intermixed.


Next you can explore what it means to set a TCP Viewpoint. Forming a plane, it cuts though the stack like mountain top removal and exposes the messages below it.


After you apply the Viewpoint, you are left with only TCP messages, which makes it easier to focus on those messages during analysis. By using Message Analyzer’s Message Stack tool window, you can still see how TCP messages relate to the upper layers, however, other protocols above the TCP level are invisible.


By contrast, take a look at what happens if you apply a View Filter for TCP instead, with no Viewpoint applied. In this case, we see the top most layer while at the same time the TCP fragments are abstracted. The list of messages is brief, which is nice because unwanted fragment noise is removed.


Viewpoints are not necessarily limited to a single protocol. As a fictitious example, our team could define a Viewpoint to expose both SMB and HTTP traffic. This would allow traffic for either to come through, but messages from protocols above this level, such as RPC, would be removed.


Behold the Needle

Problems can be very difficult to troubleshoot, especially in a world where continuing faster speeds and more complex systems mean that more data will have to be considered. By using the BSV features as your first step, followed by filtering, Viewpoints, Sequence Expressions, grouping, correlation, and other tools as a second step, you will be able to get close enough that the elusive shinny needle is discoverable in the haystack field.

More Information

For additional details about some of the concepts described in this article, see the following topics in the Message Analyzer Operating Guide on TechNet:

Comments (2)

  1. Anonymous says:

    Pingback from Interesting things that i see on the internet, may 2014. | 503 5.0.0 polite people say HELO

  2. Anonymous says:

    Needle in a Hayfield of Haystacks – MessageAnalyzer – Site Home – TechNet Blogs