Hi! My name is Sai Balaji and I work with Microsoft Networking Support in Bangalore. I primarily help Premier Customers resolve their real time Windows Networking issues. The ‘push’ that I got to write this first post of mine goes back to my own experiences with this simple and elegant component called ARP (Address Resolution Protocol). Cutting out the allusive and clichéd introductory part, let us get ourselves aligned with the agenda.
Things that you can look forward to read on this post:
DAD (Duplicate Address Detection), gratuitous ARP and Windows machines:
- Gratuitous ARP Send and Receive behaviors – Windows XP, Vista/7 and Windows 8.
- Why so many changes in gratuitous ARP behavior from Windows XP till Win 8?
- The new change in Windows 8 / Windows Server 2012’s gratuitous ARP behavior.
A case study of Windows Vista/7’s gratuitous ARP Receive and Send behaviors:
- Windows Vista/7’s gratuitous ARP Receive behavior in a ‘Failover Cluster’ scenario and the ‘hotfix’.
- Lab demonstration for illustrating the gratuitous ARP Receive behavior.
- Windows Vista/7’s gratuitous ARP Send behavior in an ‘across subnet’ scenario.
- Lab demonstration for illustrating the gratuitous ARP Send behavior. (3 and 4 are clubbed together)
Before diving into these pointers, you might want to understand about how DAD and gratuitous ARP works on a surface level.
You can make use of this neatly written TechNet blog that talks about this: http://blogs.technet.com/b/networking/archive/2009/03/30/tcp-ip-networking-from-the-wire-up.aspx
Here is a snippet of that blog to give you an overview:
Address conflict detection (Gratuitous ARP)
ARP is also used to detect IP address conflicts. Address conflict detection is used to insure that a system that is brought up on the network or that is assigned a new IP address does not have an address that conflicts with a system already on the network.
In address conflict detection, we use what is known as a Gratuitous ARP. When a system is configured with an IP address either manually or by DHCP it will send a Gratuitous ARP to insure that another node on the network is not already configured with this IP address. In the case of a conflict the two nodes are defined as follows. The Offending Node is the node that is sending the gratuitous ARP, and the Defending Node is a system already configured with the IP Address in question. The contents of this request and how this affects the ARP cache on other systems on the network differs depending on the OS.
First let me introduce some of the terms you might see further ahead. I just want to make sure that you get acquainted to them beforehand. The important ones, I suppose, are SHA and SPA.
In an ARP Request the fields are filled in as follows:
- Source Hardware Address (SHA) – MAC address of the requesting system.
- Source Protocol Address (SPA) – Protocol Address, this is the IP Address of the sending system.
- Target Hardware Address (THA) – MAC address of the system with the Target IP, for ARP request this will be 0.0.0.0 since this is the address we are trying to discover.
- Target Protocol Address (TPA) – Protocol Address, will be the destination IP address that we are trying to discover the MAC address for.
Gratuitous ARP Send behaviors:
When a new IP address is added to an Interface, this is how the various clients send gratuitous ARP packets for DAD.
Windows XP – Sends six gratuitous ARP packets at intervals of 1s with the IP address that it is trying to add in the SPA field and its MAC address in the SHA field.
Windows Vista/7 – Sends three gratuitous ARP packets at intervals of 1s with ‘0.0.0.0’ in SPA field and its MAC address in the SHA field. (I will explain why this change was made in a later section)
Windows 8 – Sends three gratuitous ARP packets at intervals of 1s with ‘0.0.0.0’ in SPA field and its MAC address in the SHA field, waits for 1 more second and then sends another gratuitous ARP packet with the ‘IP address’ that it is trying to add in the SPA field and its MAC address in the SHA Field. (Again, I will explain why this change was made in a later section)
Gratuitous ARP Receive behaviors:
When a gratuitous ARP packet, which contains ‘0.0.0.0’ in SPA, is received by a Windows Client, nobody is going to make any change to the ARP/Neighbor Cache entry because it is 0.0.0.0. The whole purpose of sending out such a gratuitous ARP packet was to prevent ARP poisoning and this technique fights it tooth and nail and emerges successful.
When a gratuitous ARP packet, which contains an ‘IP address’ in SPA, is received by a Windows Client, this is how the Clients react to it.
Windows XP – If the SPA matches with an IP in its ARP cache table entries, it will go ahead and replace the old entry with the new one i.e. the old MAC will be replaced by the new MAC that is present in the SHA field of the gratuitous ARP packet.
Windows Vista/7 – If the SPA matches with an IP in its Neighbor cache table entries, it will ‘not’ go ahead and replace the old entry with the new one but changes the state of the “Neighbor Cache entry” to ‘Stale’.
Windows 8 – If the SPA matches with an IP in its Neighbor cache table entries, it will go ahead and replace the old MAC with the new MAC.
All three of these behaviors hold valid for their Server equivalents as well – Windows Server 2003, 2008/R2 and Server 2012 respectively.
- Why are there so many changes in this sending and receiving gratuitous ARP behaviors?
- Why all this switching back and forth in gratuitous ARP behavior?
Keep reading for answers..
The reason for the Windows XP to Vista/7 Change:
The reason why gratuitous ARP Send and Receive behaviors were changed in Windows Vista/7 was to eliminate the shortcomings of the previous Windows XP gratuitous ARP behavior which encourages ARP poisoning.
With the new ‘0.0.0.0’ SPA technique, gratuitous ARP works pretty well for a normal DAD scenario as the Client sends a gratuitous ARP broadcast that does not poison the ARP tables of the other machines or the ARP table of the Router interface in its same subnet. Now, if there is a defending node claiming to own this IP already, it can directly respond to this offending/problem node by sending a Unicast ARP response.
This means that there is no need for the defending node to send a new gratuitous ARP broadcast over the Network to update the clients’ ARP tables, unlike XP, as the 0.0.0.0 SPA technique would not let that happen (sending side prevention) and if the receivers are Windows Vista/7, they do not replace the old MAC with the new MAC even if the SPA contains an ‘IP address’ (Receiving side prevention).
The reason for Windows Vista/7 to Windows 8 Change:
When I got a chance to observe the new gratuitous ARP behavior in Windows 8 /2012, I realized that it was not actually a ‘new behavior’ as something told me that I have seen this same behavior somewhere else in my past experience with other Microsoft products. That is when I realized that this new gratuitous ARP behavior was introduced way before Windows 8/2012.
For comprehending what I am going to say further here in this post, it is important for all of us to understand that DAD /gratuitous ARP is not something that is used only while adding an IP address to an interface from the Adapter properties or using net shell or after getting an IP address from DHCP.
There is one more component that uses DAD/gratuitous ARP. This component is very much used in almost all production environments nowadays, mainly, for making resources highly available. It is called ‘Failover Clustering’. Note, I am not just referring to ‘Windows Failover Clusters’ but also all other products in the market that offer the same ‘resource failover’ features.
A case study of a Windows Vista/7’s gratuitous ARP Receive and Send behaviors
A ‘failover cluster’ case study for Windows Vista/7’s gratuitous ARP Receive behavior:
I am not a Cluster expert but with my limited understanding about how a ‘failover’ operation works and on a very surface-level explanation, this is what happens:
When an IP resource fails over from one node (Node 1) to another (Node2), it is important that the Clients and the Routers on that subnet realize that the MAC address for that IP resource has changed from MAC1 (Node1’s MAC) to MAC2 (Node2’s MAC). Any delay in this process will add to the overall failover delay.
Total Failover time = Time taken when (the IP resource unbinds itself from Node1’s network stack + actual resource failover time + IP resource binds itself to Node 2’s Network stack +Clients ARP/Neighbor Cache tables replace MAC1 with MAC2+ Time taken by the Client Application to connect back in case the application time-out period has already elapsed).
This operation is, more or less, like adding a new IP to an interface and for all that we know, the same DAD process should kick in and it obviously does.
Windows 2008/2008 R2 Windows Failover Cluster follows a different gratuitous ARP approach for DAD as opposed to a Windows Vista/7 and Windows Server 2008/2008 R2 hosts that are ‘not’ Cluster nodes.
Moving further, on a failover incident from node 1 to node2, we can get a better understanding of gratuitous ARP behavior by elucidating it with an example and assuming a few key parameters:
- MAC Address of node 1 : MAC1
- MAC address of node 2: MAC2.
- IP of the Cluster resource – 10.0.0.6.
- Initially, the Clients’ ARP/Neighbor Cache tables on the same subnet have got MAC1 for that Cluster Resource IP.
- The Router interface’s ARP table contains MAC1 for that Cluster Resource IP.
- Clients are able to use that resource say, File Server, by using its IP 10.0.0.6 and its MAC (MAC1).
Let us get into the core part of this scenario – The File Share resource fails and is failed over from Node1 to Node2. (Boom!)
Gratuitous ARP Send behavior in a 2008/R2 Cluster failover scenario:
- Three gratuitous ARP Broadcast requests with 0.0.0.0 in SPA and MAC2 against SPA are sent out from Node1 at 1s intervals.
- Then, it sends the 4th gratuitous ARP packet with the 10.0.0.6 against the SPA field and MAC2 against the SHA field.
Anecdotally, it is very clear that this new behavior has taken up a new synergistic approach by combining the Windows Vista/7’s 0.0.0.0 SPA technique and the previous Windows XP’s ‘IP address in SPA’ technique.
Now, looking through this new behavior from a normal DAD’s perspective, it is a clear indication that the Windows 8/2012 (non-Cluster hosts) follows this same technique to achieve both DAD and MAC address replacement (For a Cluster failover scenario) intelligently.
- It uses the 0.0.0.0 SPA technique for the first 3 packets and makes sure that there are no defending nodes for that IP address in a total time span of 3 seconds (roughly).
- Then, it sends the 4th gratuitous ARP packet with the 10.0.0.6 in the SHA field.
- Now, if there is a defending node in that ARP broadcast domain, Windows 8/2012 gives 3 seconds (first 3 ARP broadcast requests)for letting the defending node notify it about a possible duplicate address threat before sending out the 4th gratuitous ARP which would update the ARP tables. (In 99.9% of the cases, the defending node replies back to the first gratuitous ARP request itself)
Wow! Justice has been served by this new gratuitous ARP behavior for both DAD and ‘Cluster failover’ scenarios! Wait a minute, I think now comes the tricky part. What about the gratuitous ARP Receive behavior?
This new 4th gratuitous ARP packet technique would
- replace the old MAC with the new MAC in a Windows XP/2003 Client’s ARP table (Scroll up to check the gratuitous ARP Receive behavior section).
- replace the old MAC with the new MAC in a Windows 8/2012’s Neighbor cache table (New behavior)
- ‘not’ replace the old MAC with the new MAC in a Windows Vista/7/2008/2008R2’s Neighbor cache table.
Windows Vista/7/2008/2008R2’s gratuitous ARP Receive behavior and the hotfix:
Windows Vista/7/2008/2008R2 ‘s TCP/IP stack is designed not to update the Neighbor Cache table when it receives the 4th gratuitous ARP packet as it would contain an ‘IP address’ against the SHA field. What it does is it marks the old entry as ‘Stale’. Therefore, it would wait for 5 more seconds (Worst case) to probe the old MAC by sending out Unicast ARP requests and later discover the new MAC by sending out Broadcast ARP requests for that IP (Cluster resource IP) which would apparently contribute to the overall failover delay. An important thing to know about this ‘probe’ behavior is that it happens only when the Application layer is constantly trying to use that old MAC address and failing.
More detailed information about different states of a neighbor cache entry would be overkill for this post which you might as well have thought from the beginning.
Okay, if you are still with me and with your head is up, you would probably be thinking, “How on earth would these ‘5 seconds’ actually pose a major threat whatsoever?”
Let us look through it. Consider this situation where we have 2 Windows Server 2008 R2 nodes that are a part of a Windows Failover Cluster. These nodes are connected to a Disk Array (Storage Cluster). Now, when the Storage controller fails over from one node to the other, it sends out gratuitous ARP packet broadcasts like any other failover operation. When the Cluster nodes receive this gratuitous ARP packet, after already going through a delay due to the failover operation from the Storage controller end, the 2008 R2 nodes would further delay by waiting for 5 more seconds (roughly). In this scenario, we are actually talking about ‘High available Cluster nodes’ acting as Clients. This small delay would apparently contribute to the whole threat posed against the ‘High availability’ feature of a Cluster.
This gratuitous ARP receive problem/issue in Windows Vista/7/Server 2008/2008 R2 has already been addressed and fixed by Microsoft with a hotfix described in the following article: http://support.microsoft.com/kb/2582281
After installing this hotfix and rebooting the Vista/7 Client, it would be able to replace the old MAC with the new MAC whenever it receives the gratuitous ARP broadcast packet on the fly.
Lab setup for ‘Cluster Failover’ for understanding Windows Vista/7 gratuitous ARP Receive behavior
Windows Server 2008 or Windows Server 2008 R2 Cluster (2 Nodes) :
- IP address of Node 1 – 10.1.1.1
- IP address of Node 2 – 10.2.2.2
- MAC address of Node 1- 00-15-5D-50-AD-09
- MAC address of Node 2- 00-15-5D-50-AD-0A
- IP address of the Cluster’s File Server Resource – 10.0.0.6
- Name of the File Server resource – FILESERVER
- Windows XP sp3
- Windows 7 sp1 – Without hotfix 2582281
- Windows 7 sp1 – With hotfix 2582281
- Windows 8/2012 RTM.
We are going to simulate a failover from Node 1 to Node2 and check when the ARP/neighbor cache entries are getting changed for the resource IP 10.0.0.6 from 00-15-5D-50-AD-09 to 00-15-5D-50-AD-0A
For doing this on a finer detail, we can use a PowerShell script to continuously monitor the Clients’ ARP and Neighbor cache tables every second. In this way, we will be able to see the exact point in time when the entry gets changed from the old MAC to the new one and then compare it with the exact time when the gratuitous ARP packets hit those Clients.
- Install PowerShell on your Windows XP machine. You need to install Microsoft .NET Framework 2.0 SP1 for running PowerShell in XP. In Windows 7 and 8 Clients, PowerShell comes installed by default.
- Make sure that current owner of the File Share resource is Node 1.
- Install Microsoft Network Monitor 3.4 on all the 4 Client machines. Use this link to download : http://www.microsoft.com/en-us/download/details.aspx?id=4865
- Run a command prompt as administrator and ping 10.0.0.6 so that the Client caches Node1’s MAC address in its ARP/Neighbor cache table.
- Start the Network captures on all the 4 Clients.
- Run the ARP_cache.ps1 script on the Windows XP Client and Neighbor_cache.ps1 script on both the Windows 7 machines and also on the Windows 8 machine.
- You can demonstrate this problem either by using the ‘Simulate failure of this resource option’ or ‘Move this service or application to another node’ option from the Failover Cluster Manager console.
- The resource/s will go offline for a moment and come back online quickly. After when they come back online, you will be able to see that the ‘Current owner’ of the resource has been changed from node1 to node2.
- Once when you see that the resource has come back online with a new owner, you need to stop the network captures on all the Clients and wait till the Script stops by itself after 100 seconds.
For Windows 7 and Windows 8 Clients:
For the Windows XP Client:
Analyzing the captures and the outputs:
When the Cluster resource fails over from Node1 to Node2, it will release the IP from Node 1’s network stack and adds it to Node 2’s network stack.
When it binds the new IP to node 2’s stack, it sends out gratuitous ARP broadcast packets as an implementation of DAD and new MAC announcement. From the network capture, we see that from 11:01:30 to 11:01:32, there are 3 gratuitous ARP packets that reach all the Clients at the same time. The 4th gratuitous ARP broadcast packet which has the SHA as 10.0.0.6 reaches the Clients at 11:01:33.
Illustration of gratuitous ARP Receive behavior of Windows XP:
The gratuitous ARP packet with SPA field set to 10.0.0.6 reaches the Client at 11:01:33.
The ARP entry is changed to the MAC address of node 2 exactly at 11:01:33.
Illustration of gratuitous ARP Receive behavior of Windows 7 (without KB 2582281)
The 4th gratuitous ARP packet reaches the Windows 7 Client at 11:01:33
The Neighbor cache entry does not change even after receiving the 4th gratuitous ARP packet at 11:01:33 and still contains Node 1’ MAC. It is just marked ‘Stale’. Since we did not have any application running in the background and using this Neighbor Cache entry, the entry turned to a ‘Stale’ state. If you like to see the state change from ‘reachable’ to ‘stale’, you need to follow this same procedure with a continuous ping in the background from this Windows 7 Client.
Illustration of gratuitous ARP Receive behavior of Windows 7 (with KB 2582281)
The 4th gratuitous ARP packet reaches the Windows 7 Client at 11:01:33
The Neighbor cache entry changes from node1’s MAC to node 2’s MAC exactly at 11:01:33.
Illustration of gratuitous ARP Receive behavior of Windows 8 / 2012
The 4th gratuitous ARP packet reaches the Windows 7 Client at 11:01:33.
Windows Vista/7/2008/2008 R2’s gratuitous ARP Send behavior
The new Windows 8/2012 gratuitous ARP behavior and the Windows Vista/7/2008/2008R2 hotfix for gratuitous ARP Receive behavior tackle the DAD and the ‘Cluster failover’ scenarios efficiently.
Pushing further, let us take a completely different scenario highlighting the problem with “Windows Vista/7’s gratuitous ARP Send behavior”.
This is the setup:
I have a machine A in 10.x.x.x subnet which has a subnet mask of 255.0.0.0. There is another machine X in 11.x.x.x subnet which has a subnet mask of 255.0.0.0. There is a Gateway which acts an internetwork for 2 subnets – 10.x.x.x and 11.x.x.x
IP address of A – 10.1.1.1
Gateway’s IP for 10.x.x.x interface – 10.10.10.10
Gateway’s IP for 11.x.x.x interface – 126.96.36.199
Machine X pings Machine A for the first time:
Machine X would know that 10.1.1.1 does not belong to its subnet range and therefore would send the ICMP Echo Request packet to its Gateway. The Gateway would then send an ARP broadcast for that IP (if it does not have its MAC in its ARP table already) and when it gets a response from Machine A, it would cache that MAC (MAC1) for that 10.1.1.1 in its ARP table.
Then, the Gateway would send the ICMP Request packet to Machine A’s MAC (MAC1) which would respond by sending out an ICMP Reply packet. The Gateway forwards that packet to Machine X’s MAC and thus, ping is successful here.
Removing 10.1.1.1 from Machine A and assigning it to Machine B on the same subnet:
Now, if I remove this IP (10.1.1.1) from machine A’s interface and assign it to Machine B on the same 10.x.x.x subnet, the Windows Vista/7’s gratuitous ARP mechanism would not let its Neighbors, including the Router’s 10.x.x.x interface, know about the fact that 10.1.1.1 is now an IP address that should point to MAC2 and not MAC1.
Windows Clients within the same subnet would be able to recover from this state and contact the new MAC after sometime depending upon the ARP cache timeout in Windows XP Clients (2-10 minutes) and the ‘5 second Stale-Probe-New ARP broadcast’ phase in Windows Vista/7 Clients.
However, in most of the Customer environments, the Gateways (Routers) would be configured with an ARP cache time out specified in ‘hours’ (The maximum that I have seen is 2 hours).
The 0.0.0.0 SPA technique for gratuitous ARP/DAD that is adopted by Vista/7/2008/2008 R2 would not let the Router know about this change. This impacts cross –subnet clients as they would not be able to contact 10.1.1.1 for at least 2 hours unless we delete that old ARP entry from the Router.
If you are assigning an IP address that was used by a different machine to a Windows Vista/7/2008/R2 machine, you need to make sure that you delete the old ARP entry manually from the Gateway/Router.
Thanks for reading patiently! I hope that this post has answered some of your questions related to differences in gratuitous ARP/DAD behavior between Windows XP, Vista/7 and Windows 8 and other ‘failover delay’ woes. Happy gratuitous ARPing!
– Sai Balaji