Retrieving Packets Received Discarded Perfmon Counter From Multiple Servers

Whilst working on a customer’s Exchange 2010 DAG issue, I wrote a quick script to quickly grab some performance monitor counters from all of their Exchange servers.  The issue that we were investigating was related to discarded packets when the VM was running on a certain hypervisor host.  The customer had moved their Exchange VMs to a new host and after doing so they were experiencing cluster issues.  Randomly nodes would be dropped from cluster membership which would impact the Exchange 2010 DAG as any active copies of those Exchange databases would then have to be mounted on another server.  The activation was happening automatically (as expected) but it is still not a desired state. 

On the Exchange servers we observed EventID 1135 – Cluster node was removed from the active failover cluster membership. 

At this point we did not do the typical knee jerk reaction that normally happens -- which is to simply rack up the cluster timeout values.  Why you ask? Well that does not address the root cause, and only masks the symptom.

Update 18-11-2014: Please see this post for a script to retrieve the number of 1135 EventId errors on multiple servers.

Update 19-4-2015: Adding link to KB 2634907 "TCP/IP packets that are received out of sequence are discarded in Windows 7 or in Windows Server 2008 R2' for reference.  This is an issue with Windows Server 2008 R2 and may cause discarded packets.  At the time of adding comment, servers should already be patched beyond these builds. 


Oh Issue, Where Art Thou?

We quickly checked the basics, and made sure that the Exchange 2010 recommended DAG update (it’s a cluster update but Exchange recommends it strongly) was installed, and also the generic updates recommended for the version of the OS Exchange was installed onto.  They are discussed in this post along with other Exchange 2010 deployment tweaks. 

None of this made a difference.  The cluster still experienced EventID 1135 cluster disconnects.  Since this only started after the VMs were moved to the new host, known issues for those hosts were then reviewed.   In VMware KB 1010071  and 2039495  these symptoms are discussed and Exchange 2010 is specifically tagged in the second article. 

While the hypervisor admins have their own tools to report and investigate such issues, we can use Performance Monitor to see how Windows perceives the lay of the land. 

The counter that we were looking at was “Packets Received Discarded”.  The sample image below is from Hyper-V and shows the location:

From the Perfmon description: Packets Received Discarded is the number of inbound packets that were chosen to be discarded even though no errors had been detected to prevent their delivery to a higher-layer protocol.  One possible reason for discarding packets could be to free up buffer space. 

This is great – we can use this counter to look at the issue, but how to do it easily across multiple servers?  And then potentially across every single VM that the customer has since if we are hitting the issue on one set of VMs what other VMs are affected?  We could:

  • Open Perfmon locally on each box and then (hell no, let’s stop the crazy talk right there)
  • Open Perfmon and add the counter in for each server – ugg…
  • Create an input file specifying server\perfcounter and feed this to logman.exe
  • Some PowerShell scripting to remotely retrieve the perf data – sold!



PowerShell To The Rescue

To see if we were experiencing the issue across multiple Exchange servers, and to gauge the severity I wrote a quick PowerShell script that would pull in the required performance counters from multiple servers quickly and easily.    This uses the Get-Counter cmdlet as shown here:

Get-Counter "\Network Interface(*)\Packets Received Discarded"  -ComputerName $Server

The script will get a collection of NICs from the specified server, and then loop through them and remove the pseudo ones.  For example do not want to see Teredo, ISATAP or 6to4 interfaces.  For the purposes of this script we are concerned with the physical ones, and that includes the "physical" NICs that are made visible in virtual guest Operating Systems.  NIC names are not hardcoded into the script else it would not be portable across physical server types and hypervisors. 

You can obtain this from the TechNet Gallery under the Get Perfmon Counter Packets Received Discarded On Multiple Servers

Update 22-10-2014:  Updated script to also include OS uptime and OS installation date

Get Perfmon Counter Packets Received Discarded On Multiple Servers

Using the script, we were able to quickly check all of the customer’s servers and quickly pinpoint trends in the environment.  One half of the DAG servers were experiencing discarded packets many times higher that the others, and the trends were noted on both MAPI and REPL networks.  This allowed us to focus on particular hypervisor hosts. 

Armed with this data, we then could ask why specific VMs were more impacted than others and prove it.  It turns out that the Exchange VMs had 64GB of RAM assigned and had been placed onto hosts which had 64GB of physical memory.  Since there was no free memory for the hypervisor, this was placing pressure onto the hypervisor and exacerbating the issue. 



This is an issue that has received attention in the past from the Exchange community.  In addition to the other great posts out there on this topic, I posted this to set the context around the script as we all want an easy to check lots of servers and potentially monitor for this issue. 




Comments (18)
  1. anonymouscommenter says:

    When resolving issues with on-premises Exchange sometimes the issue may be directly within Exchange,

  2. anonymouscommenter says:


  3. Very cool! We see this issue quite often from our own support customers. Great post!

  4. Thanks folks!

    Please share this with the team if that can help them Andrew!


  5. anonymouscommenter says:

    In some of the recent posts you will have noted that there have been some issues with VMware , and also

  6. anonymouscommenter says:

    Same issue.. I have checked the Mem and CPU utilization, not a problem.. the buffers have been set to max, did not help

  7. Hi Mudasser,

    What feedback did the hypervisor team have for you? Have they looked into updating the host?


  8. anonymouscommenter says:

    Hi Rhoderick,

    It is running on ESXi 5.5 GA and VMware do suggest an upgrade, but there is no documentation/identified known issues mentioned.
    I have seen an increase in "Receive packets dropped" count vethernet of VM, no increase in "received packets discarded" in perfmon

  9. Good to hear that you have a case open and are working it!

    In terms of issues are you see the one in here:


  10. anonymouscommenter says:

    I use both of your scripts Check-EventID-1135 and Show-Discarded-Packets which has helped in identifying the dropped packets on our Exchange 2013 DB servers. It also helped us to identify that only the VmxNic3 was having dropped packets since I included
    other servers such as DC/WEB/Print and SQL 2012 Clusters.

    We have contracted some specialists to deep dive into VMWare and the Network infra to help resolve the issue. They have request a report on the actual servers which are having dropped packages.

    Since I am not that good in PowerShell, can you help me modify tour script to generate a report which includes all the information including uptime and host name? Export-CSV only exports the NIC and Cooked values and >.Discarded.log requires too much cleaning
    up for 200 servers.


  11. Hi Tim,

    Great to hear that you find the scripts useful. I’d rather challenge you to do this mod yourself 🙂

    Look at this template file I’ve posted:

    "Borrow" the custom PSObject concept, and that will allow you to easily export to CSV.

    LMK how you get on.


  12. anonymouscommenter says:

    On the Retrieving Packets Received Discarded Perfmon Counter From Multiple Servers post, Tim commented

  13. anonymouscommenter says:

    This is link throw down for items that we discussed in a recent Exchange 2013 workshop. Well, actually

  14. anonymouscommenter says:

    This is link throw down for items that we discussed in a Exchange 2013 workshop which I delivered in

    1. Narayanan says:

      Hi how to generate perfmon counter alert for packet drop in perfmon

Comments are closed.

Skip to main content