Windows Server 2012 R2 DHCP Expired Leases Not Getting Scavenged

Scenario

 I encountered an interesting case with a customer recently where expired leases were not getting deleted from the database. The customer had recently migrated from Windows Server 2008 R2 to a Windows Server 2012 R2 DHCP HA cluster configured for Failover.  The symptoms were that some of the scopes were full, however exporting leases that were visible in the GUI showed that plenty of IP addresses should have been available.

A quick search revealed that there are practically no tools to troubleshoot the DHCP database and most articles that were available said DHCP scavenging "just works" and that DNS scavenging was where problems typically occur vs. DHCP scavenging. Fortunately this was Windows Server 2012 R2 and as with almost anything Windows Server 2012 R2, PowerShell came to the rescue.

Identifying The Problem

  The easiest way to identify the problem is to dump the full DHCP scope and all of its leases, to include the ones that are not visible through the GUI. The following command will dump the entire database. Replace x.x.x.x with the IP address of the target scope: 

         get-dhcpv4serverlease -scopeid x.x.x.x -allleases | out-file leases.txt | notepad leases.txt

As shown in the following screen shot, the resulting file will show all leases, including expired leases. The telltale lease is the one circled in red. The lease expired months ago yet it is still in the database.

Additional research and filtering showed that over 300 such records were in the 192.168.0.0/22 scope alone, which is a significant number of IP addresses that were unavailable for DHCP clients.

Resolution

The following steps will fix the problem while minimizing downtime and preventing IP conflicts from occurring.

Remove All Scopes From The Standby DHCP Server

  1. Disable the NIC on the standby DHCP server
  2. On the standby DHCP server open an elevated PowerShell command prompt and type the following commands:  Get-DHCPServerv4Failover | Remove-DHCPServerv4Failover -Force
  3. Get-DHCPServerv4Scope | Remove-DHCPServerv4Scope -Force

Fix The Database On The Active Server

  1. Open the DHCP Console on the active DHCP Server > Right click the DHCP Server Object > Backup to create a backup of the database
  2. Open an elevated PowerShell command prompt and type the following commands: Get-DHCPServerv4Failover | Remove-DHCPServerv4Failover -Force
  3. Open the Registry and locate the following key: HKLM\SYSTEM\CurrentControlSet\Services\DHCPServer\Parameters\DatabaseCleanupInterval
  4. Select Decimal and change the interval to 5 then click OK
  5. Restart the DHCP Server service (PowerShell: Restart-Service DHCPServer)
  6. Wait approximately 5-10 minutes then run the following PowerShell Command: get-dhcpv4serverlease -scopeid x.x.x.x -allleases | out-file leases.txt | notepad leases.txt (replace x.x.x.x)
  7. Search for Expired leases. If none are found the proceed to the following step. If expired leases are still found then wait 5 or 10 more minutes. Keep in mind that by default the DHCP server waits 4 hours to delete expired leases, so only leases that exist well after their expiration date + 4hrs are leases that should have been scavenged.
  8. Once all bad expired leases have been properly scavenged open the Registry and locate the following key: HKLM\SYSTEM\CurrentControlSet\Services\DHCPServer\Parameters\DatabaseCleanupInterval
  9. Select Decimal and change the interval to 60 then click OK
  10. Restart the DHCP Server service (PowerShell: Restart-Service DHCPServer)

Recreate the DHCP HA Failover Cluster

  1. Enable the NIC on the standby DHCP server
  2. Open the DHCP Console on the Active DHCP server > Right click IPv4 > Configure Failover > Next > Add Server > select the failover DHCP server
  3. Click OK > Next > Select Hot Standby for the failover mode and enter the Shared Secret password (if Message Authentication is desired) then click Next > Finish
  4. Check the standby DHCP server to ensure the scopes properly replicated and compare the standby DHCP server's Server Options to the primary DHCP server's Server Options to ensure they are correct since server options do not replicate between DHCP servers.

Possible Root Cause

I am not completely certain as to why the problem occurred. However, the dates on all of the expired records were from the day the database was migrated from Windows Server 2008 R2 to Windows Server 2012 R2. All reservations created since the migration date were being properly scavenged. What I believe happened is that the Windows Server 2012 R2 HA failover partnership was created right after the database was migrated and any expired DHCP reservation entries were then unable to be scavenged during the DHCP service's normal scavenging process. After seeing this issue, if I am ever asked to migrate a DHCP database to a Windows Server 2012 R2 HA cluster, I will recommend dumping the database and verifying there are no expired DHCP leases prior to enabling the failover partnership.

Conclusion

This may have been a one off completely odd scenario that is unlikely to occur again, or it could be something more common that is due to a bug in Windows Server 2012 R2. If this post resolves anyone else's Windows Server 2012 R2 DHCP issues let me know via feedback and I will file a bug report if it appears to be more common than currently thought.