Issues to join a node to a Cluster between sites? Problem solved!

Introduction

Here is Dante Stancato, Support Engineer from Windows Networking team, and in this article, we will explore an interesting topic, based in several cases we've received recently, where you may experience communication failures between two Failover Cluster Nodes. This behavior usually happens on the process of joining nodes to a Failover Cluster in an environment where you have third party network equipment performing packet inspection/packet forwarding between the machines involved.

In this scenario, there are other cluster node nodes in both sites which are already joined and do not experience the issue. However, when you try to join new node in cluster, you receive messages that state the operation time out, or that is not able to contact the cluster. During those type of cases, the initial course is start the work with of Cluster Support Engineer who was not able to identify anything abnormal on reviewing cluster logs, or any relevant information from usual data sources. Networking support engineer gets involved reviewing if there's issues outside cluster which maybe affecting the node joining process and that was the case as you can see more details below.

Environment

This Scenario is configured as follows:

• Node A, located at Site A (With IP Address 192.168.0.100)
• WAN Link with third party network equipment performing packet inspection / packet forwarding
• Node B, located at Site B (With IP Address 192.168.1.100)

Data Collection

In order to troubleshoot this issue is necessary to get a regular network capture trace plus a trace of the NetFT using the following command:

netsh trace start capture=yes overwrite=yes maxsize=1024 tracefile=c:\base_cluster.etl provider={7E66368D-B895-45F2-811F-FB37940421A6} keywords=0xffffffffffffffff level=0xff

During the process of Node B to the Cluster fails and we need to stop the capture by running:

Netsh trace stop

Data Analysis

Reviewing the trace taken and focusing on NetFT component you can observe the following events along the trace:

[26] 0000.0000::12/15/16-12:38:00.2645164 [rm] rmadp_c797 RmHeartbeatTimerDpc() - Monitor FFFFE0007B4522A0 (192.168.0.100:3343 - 192.168.1.100:3343) sending RCP request FFFFE0007B4C7570 (sequence 7). Ticks=5969997, last=5969741, delay=256.
[26] 0000.0000::12/15/16-12:38:00.2645404 [rm] rmadp_c844 RmHeartbeatTimerDpc() - Monitor FFFFE0007B4522A0 (192.168.0.100:3343 - 192.168.1.100:3343) missed HB response, gap=6
[8] 0000.0000::12/15/16-12:38:00.2645566 [rm] rmadp_c1035 RmSendComplete() - Monitor FFFFE0007B4522A0 (192.168.0.100:3343 - 192.168.1.100:3343) completed RCP message FFFFE0007B4C7570 with 0.
[12] 0000.0000::12/15/16-12:38:00.2734304 [rm] rmsup_c1412 RmFilterRcpMessage() - Validation of RCP packet failed - Route monitor (192.168.0.100:3343 - 192.168.1.100:13429) FT IPs (fe80::d5e4:bf14:708f:b3db - fe80::ec3f:16be:f9a4:3cf3) not found.
[12] 0000.0000::12/15/16-12:38:00.2734317 [ta] tnsup_c113 TaFilterPacket() - Dropping received RCP messages from 192.168.1.100:13429.

What needs to be checked in here is TCP the source and destination port on each packet. As stated in the document Technet documents Networking in Failover Clusters, source and destination port of Failover Cluster's Heartbeat must be UDP 3343. When this port is changed, even in source or destination, by default, the Cluster will discard the packets, causing communication failures.

Filtering the network capture for the conversation between Node B and Node A as well as UDP 3343 port, we will notice the following behavior on Heartbeat packets:

If we look at the network packet on the source side, we can see that source and destination ports are the expected one: UDP 3343:

Frame: Number = 78, Captured Frame Length = 146, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[10-8C-CF-56-01-01],SourceAddress:[5C-B4-01-C8-70-52]
+ Ipv4: Src = 192.168.0.100, Dest = 192.168.1.100, Next Protocol = UDP, Packet ID = 11776, Total IP Length = 132
- Udp: SrcPort = 3343, DstPort = 3343, Length = 112
SrcPort: 3343
DstPort: 3343 TotalLength: 112 (0x70)
Checksum: 27094 (0x69D6)
+ UDPPayload: SourcePort = 3343, DestinationPort = 3343

Now, if we look at that same package but at destination side, we will notice a slightly different behavior in the packet, which in a quick look may seem harmless, but it's what really is causing us all our headaches:

Frame: Number = 79, Captured Frame Length = 146, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[5C-B4-01-C8-70-52],SourceAddress:[10-8C-CF-56-01-01]
+ Ipv4: Src = 192.168.1.100, Dest = 192.168.0.100, Next Protocol = UDP, Packet ID = 11261, Total IP Length = 132
- Udp: SrcPort = 13429, DstPort = 3343, Length = 112
SrcPort: 13429DstPort: 3343
TotalLength: 112 (0x70)
Checksum: 52191 (0xCBDF)
+ UDPPayload: SourcePort = 13429, DestinationPort = 3343

As you can see, the source port of the packet, while going through the WAN link, comes different or modified to destination. This exact same behavior is noticed in both sides of the communication in a network capture, where all the packets leave the server with the correct source port but arrives with a different one at destination.

Solution

Considering that, from a Windows perspective, the behavior while sending packets is the expected (I mean, Windows always send the packets with source and destination ports as UDP 3343) and that, while going through the WAN link between sites the source port gets changed, it is recommend reviewing the following actions:

• Review the network infrastructure to be able to identify:
  • How many hops are between the affected nodes?
  • What is the purpose of each of the hops between the nodes?
• Review the presence of devices doing packet forwarding or packet inspection
• Take network traces from each one of the hops to be able to identify which one of them is changing the Source port in the communication
• Contact Network Equipment's vendor to make the necessary corrective actions

In case you have a device manufactured by F5 between the Cluster Nodes communication, there's a specific recommendation to follow here for those specific cases where source port has to be preserved. As described in F5 documentation, for Windows Failover Clusters, the next best option is to perform a correct configuration:

The recommended option made by F5 is to configure the Virtual Servers to use option "Source Port" as "Preserve Strict". By default, when configuring a Virtual Server, the option for "Source Port" is configured as "Preserve", in which if the source port is already in use by any other device in the environment, it will use a new port number, causing the issues mentioned above.

It is always a recommend that you get in touch with the Vendor, in this case F5, to verify accordingly about these configurations, because Microsoft does not provide support for F5 equipment. We are simply sharing this documentation as reference to accelerate the solution of the issue.

I hope this entry helped you to understand this behavior and (hopefully) to keep your environment up and running.

As usual, thank you for reading and of course for keep trusting Microsoft.

See you soon!!

Special Thanks: This blog note wouldn't be possible without the help of my colleagues, so I would like to publicly thank them: Felicio Da Silva (LATAM Technical Advisor - Core / Platforms), Danilo Gonçalves (LATAM Support Escalation Engineer - Core / Platforms), Nicolas Rojas (LATAM Support Engineer - Core / Platforms) and Martin Kirtchayan (LATAM Technical Advisor - Networking / Identity).