Automatic syncing of scope configuration changes between 2 DHCP failover servers


DHCP Failover is a new feature in Windows Server 2012 which provides for high availability of DHCP service.  Two DHCP servers in a failover relationship synchronize the IP address lease information on a continual basis there by keeping their respective databases up-to-date with client information and in sync with each other. However, if the user makes any changes in any property/configuration (e.g. add/remove option values, reservation) of a failover scope, he/she needs to ensure that it is replicated to the failover server. Windows Server 2012 provides functionality for performing this replication using DHCP MMC as well as PowerShell. But these require initiation by the user. This requirement for explicitly initiating replication of scope configuration can be avoided by using a tool which automates this task of replicating configuration changes on the DHCP failover server. DHCP Failover Auto Config Sync is a PowerShell based tool which automates the synchronization of configuration changes. You can download the tool and usage guide from this post on TechNet Script Center.

Please let us know your feedback on this tool!

Comments (56)

  1. Anonymous says:

    Thanks for the quick reply…so to be on the safe side, besides monitoring
    it will be ok to change the MCLT to lets say 8 hours, and enable proper monitor
    for the relationship "Get-DhcpServerv4Failover | select state" so…while its on COMMUNICATION INTERRUPTED state
    we are ok for 8 hours and will give the admins proper time to check issues, but if they stop talking to each other for longer than 8 hours, it will go into PARTNER DOWN and at least the DHCP service on any of the 2 boxes should be DISABLED, to avoid conflicts with IP assignments ?

  2. Anonymous says:

    Hi DHCPTeam
    if im using the ps tool to do the sync between Server 1 and Server 2
    Server 1 being the one the tool where it runs from
    what happens if the 2 boxes stop talking to each other ? but the clients still see the DHCP service ?
    meaning the sync between each other will break, but the clients will still see the DHCP servers as available
    im guessing they will go as partner down after 1 hour, so they will try to take over each other's IP pool ?
    reading this KB http://technet.microsoft.com/en-us/library/dn338983.aspx, it says "If two DHCP servers configured as failover partners are unable to communicate, precautions are taken to avoid the same IP address lease being issued to two different DHCP clients." but im not sure how we can prevent or detect this besides adding proper monitoring ?
    Thanks
    Martin

  3. Anonymous says:

    DHCP Team
    yes, had both servers configured as switch over enabled to 1 hour (after creating the failover relationship), then disabling the blocking the 647 TCP port between themselves so simulate site link failure, so the expected was both going into PARTNER DOWN, and own 100% of the scopes, but only first box went into PARTNER DOWN, while second one stayed on COMMUNICATION INTERRUPTED, but not PARTNER DOWN…so for now I disabled the automatic switch over, and added extra monitoring so the admins will have to fail the box accordingly, meaning depending on outage.

    any ideas why the second box is not going into PARTNER DOWN on its own ? the first one goes to this state as it should, but not the second one

    Thanks
    Martin

  4. Anonymous says:

    hi, should this sync tool also sync reservations. i have hot standby mode. the script runs but a new reservation i am am adding is not syncing to the 2nd server. if i do it through the mmc works no probs

  5. teamdhcp says:

    Hi Philg, yes – you can use Invoke-DhcpServerv4FailoverReplication cmdlet to sync scope configuration instead of the automatic sync script tool. In fact, the automatic sync script uses the same cmdlet to syn scopr configuration whenever there is a scope
    configuration change.

  6. teamdhcp says:

    Error id 5 is ERROR_ACCESS_DENIED. The script needs to run as a user who has admin permissions.

  7. teamdhcp says:

    Yes, the tool should sync reservations as well. Can you send the log output of the script to teamdhcp@hotmail.com

  8. Anonymous says:

    Thanks a bunch for the info, by the way I tested this config as well…to have switch over at 60 minutes, then blocking port 647 between each other, after 60 minutes they should go into PARTNER DOWN if I understand correctly….well…the first box goes into this state, serving 100% of scopes…but the second box goes into COMM INTERRUPTED state, not into PARTNER DOWN…even after the 60 minutes….im not sure why the second box doesn't go into PARTNER DOWN ? Thanks !

  9. teamdhcp says:

    Hi Emmnauel, the error you mentioned is an outcome of the PowerShell cmdlet: Invoke-DhcpServerv4FailoverReplication. This error seems to have occurred when periodic sync was called for one of the failover relationships on the server. We verified that this
    cmdlet works for the super scope. We will add more loggint to the tool and publish a new one which will help understand the root cause of the failure. In the interim, could you send details on the superscope configuration that you have – for example number
    of scopes in the superscope, any disabled scopes etc. Can you please send the same to teamdhcp_at_hotmail.com

  10. teamdhcp says:

    Martin, can you bring down the first server and see if second server goes into partner down (after communication interrupted). BTW, in our tests, we block 647 using the windows firewall on the server – it should not make any difference though.

  11. teamdhcp says:

    Based on feedback received from customers, the tool provided on TechNet Script Center (link is in the blog above) has been updated on 20 Jun 2013 to include a fix for periodic synchronization of scope configurations. Also while running, in its default mode, it can now automatically include any failover relationships that were created after it was started. A restart of the tool, for including new relationships, is required only if the user is running the tool in selective replication mode.

  12. teamdhcp says:

    You can also use IPAM in Windows Server 2012 R2 for DHCP Failover management which will perform any configuration changes including reservations on both the DHCP failover servers.

  13. teamdhcp says:

    Tamas, the script can be run on only one of the DHCP failover servers and sync changes made on that server to the other DHCP failover server. So, the configuration changes (option values, reservations) need to be made on only one of the servers – where
    you are running the script. Currently the script does not support two way synchronization of configuration changes. Thanks for the feedback, we will consider revising the script to support two way sync.

  14. teamdhcp says:

    Is automatic state switchover enabled on both the DHCP servers. A server will continue to stay in COMMUNICATION INTERRUPTED state if automatic state switchover is not enabled.

  15. teamdhcp says:

    Hi Val,

    Lets say you have created the failover relationship from server 1 to server 2 and now want to decommission server 1. You can do so by deleting the failover relationship from server 2. This will remove the scopes from server 1 and retain the same on server 2. You can later create new failover relationship for the scopes from server 2 to the newly commissioned server.

  16. teamdhcp says:

    Hi Lee, can you please clarify under what user account you are running the script.

  17. teamdhcp says:

    We tried this again (its anyway a part of our tests), blocking TCP port 647. This caused both servers to move into Communication Interrupted and on expiry of state switchover interval, to PARTNER DOWN state. What you have observed is not the expected behavior. How are you blocking port 647 – we do this using firewall.
    Also, if you bring down the first server, does the second server go into communication interrupted and then to partner down ?

  18. teamdhcp says:

    Saket, I am not sure I understand the question. Can you please elaborate. If primary server is down, secondary will be able to sync the leases to primary after the primary comes up. Was your question related to the auto sync script ?

  19. teamdhcp says:

    Hi Philg, the server running in partner down will take over the entire IP address pool but the pool statistics are not reflecting to indicate that change. This is only related to display and does not affect the "failover" behavior of the server.

  20. teamdhcp says:

    Hi Martin,
    When the 2 DHCP servers stop talking to each other, they will both transition to COMMUNICATION INTERRUPTED state. Its fine for 2 DHCP failover servers to run in COMMUNICATION INTERRUPTED state since they will be giving new leases from their ownership of the free IP pool. If you have enabled “automatic state switchover”, they will automatically transition from COMMUNICATION INTERRUPTED state to PARTNER DOWN state after expiry of state switchover interval (default 1 hour). An admin needs to avoid having both servers operating in PARTNER DOWN state since a server in PARTNER DOWN state will take over the entire free IP pool assuming the other server is down. This takeover of free IP pool occurs after a period of MCLT since moving into PARTNER DOWN.
    If you have enabled automatic state switchover in DHCP failover, you should monitor events on the DHCP server which indicate PARTNER DOWN state transition and take appropriate action.

  21. teamdhcp says:

    That would be appropriate. The DHCP server logs failover state change events in the DHCP server admins channel – you can use those events to monitor.

  22. teamdhcp says:

    IP reservation does not follow the same sync method as the leases. Once you create an IP reservation on one of the servers, you need to sync it to the other DHCP server using MMC or PowerShell cmdlet. 3300 scopes on two servers with load balancing is not a problem.

    You can use the script shared at the below location to achieve automatic sync of reservations and other configuration changes-

    gallery.technet.microsoft.com/…/Auto-syncing-of-configurati-6eb54fb0

  23. teamdhcp says:

    Andy, Any time the state of an IP address on a DHCP failover server changes i.e.

    – an IP address is leased to a client,

    – the existing lease on an IP address is renewed

    – an IP address lease is released or expired

    the updated IP address record post this state changes is almost immediate communicated to the partner DHCP server using a lease synchronization message (called BINDING UPDATE). This happens almost instantaneously any time the state of any IP address in a DHCP scope changes. The only delay is what may be introduced by the underlying network between the 2 DHCP failover servers.

  24. teamdhcp says:

    Hi Lee, this is not the expected behavior unless there are configuration changes happening that quickly. Is that the case ? We will look into this and get back.

  25. That's great – thanks for the heads up!

  26. Emmanuel Arko Sam says:

    I have been using this tool for sometime now and its been working perfectly until recently I began to see these errors in the log file, Any ideas on how to fix this? ‘VFGHGBVMDHCPW2P’ is the hostname of my secondary DHCP Server ‘Periodic Sync TimeOut Happened:
    Syncing Relation:VFGHDHCPCLUSTER01 Error: Failed to get superscope information on DHCP server VFGHGBVMDHCPW2P. ————————————————————————————————– Scope not synced.Please sync it manually. If it
    does not belong to any relation please create a failover relation for it to ensure safety.’ ————————————————————————————————–

  27. Andy Wendel says:

    you mentioned: "Two DHCP servers in a failover relationship synchronize the IP address lease information on a continual basis " – can you provide me with further Infos about the time-interval, they are communicating? As an MCT – would be great to hear from you!

    Andy

  28. Flemming Skjoett says:

    @TeamDHCP

    We find that new and changed ip-reservations tend to take a little while to replicate from one server to the other, Can you confirm this to done differently than leases?

    Usually we have to manually click replicate to get it on the partner server in time when a tech is standing there waiting for it.

  29. Flemming Skjoett says:

    @TeamDHCP

    Will an IP-Reservation follow the same sync method when created on server one and then automatically replicating to server two instantaneously?

    In reality we find that we need to do a manual replication via the gui or powershell to get the new reservations copied immediately. If left to its own, there replication will occur at some point, but its nowhere instant.

    Also, could 3300 scopes on two servers with 50/50 load balancing be the problem?

  30. Martin says:

    correct, relationship was created to switchover 60mins enabled, then replicated to confirm failover was working, both nodes were in normal state, then tcp647 blocked thru juniper fw, both boxes are going into INTERRUPTED, after 60 mins, the first box goes into DOWN mode, second one stays in INTERRUPTED, but not DOWN
    i will try to replicate the same case and let you know if i can re-create
    Thanks
    Martin

  31. Martin says:

    Hi DHCP Guys,

    i might be wrong (i have been wrong before) but this seems related to the way the fw is blocking the communications
    on the 647 tcp port, meaning the juniper is not sending the "reject" packet back to sender
    hence, i think the failover keeps on waiting for ever to hear back for serverdown…which never happens

    i configured the stateswitchinterval to 60 mins, then the fw was configured not to send reject, and i waited 3 hours…the state is not
    going into partnerdown, after going thru communicationinterrupted, meaning it says there on that state

    PS C:Windowssystem32> Get-DhcpServerv4Failover

    Name :
    PartnerServer :
    Mode : LoadBalance
    LoadBalancePercent : 50
    ServerRole :
    ReservePercent :
    MaxClientLeadTime : 01:00:00
    StateSwitchInterval : 01:00:00
    State : CommunicationInterrupted
    ScopeId :
    AutoStateTransition : True
    EnableAuth : True

    telnet with no reject back to sender takes about 30 seconds to fail…

    Welcome to Microsoft Telnet Client

    Escape Character is 'CTRL+]'

    Microsoft Telnet> set localecho
    Local echo on
    Microsoft Telnet> o 647
    Connecting To …Could not open connection to the host, on port 647: Connect failed

    telnet with reject back to sender takes about 2 seconds to fail…

    Microsoft Telnet> o 647
    Connecting To …Could not open connection to the host, on port 647: Connect failed
    Microsoft Telnet>

    i then bounced the DHCP service while it was on CommunicationInterrupted, once the fw was re-configured
    to sent the reject back to sender, status goes into startup, and then communications interrupted, so i left them for 3 days
    even though "StateSwitchInterval : 01:00:00" they are not going into PARTNERDOWN when the fw is blocking

    you can test from your side if you have a hardware fw to test this…for now i will disable "StateSwitchInterval" and manually manage this

    once the fw rule is removed, and server is bounced, if second server is down,
    the partner down state kicks in as it should, sending first box into partnerdown state, after going thru communicationinterrupted

    Thanks

    Martin

  32. val says:

    Hi folks,

    I've setup DHCP failover between 2 servers in my environment following this article:
    http://technet.microsoft.com/en-us/library/hh831385.aspx

    However, I need to decommission the initial the initial server from which the scope was initially replicated. How do I go about breaking the synch relationship between them? I'm afraid that if I just unauthorize the server, and take it offline that there will be replication objects left in the background.

    Any ideas?

  33. Lee DAmore says:

    Is the log file supposed to post entries 3 to 4 times per second ? Causing large expansion of the log file. Entries coninutously are : Sync process complete at Will automatically sync again when new configuration changes are made. These repeat at least 3 to 4 times per second

  34. philg says:

    Hi, I have two dhcp 2012 r2 servers with loadbalance 50/50, state switchover 60minutes, MCLT 60minutes. When I shutdown the server1, the server 2 shows then partner down, but the ip pool doesn't change from 50% to 100%?
    Why doesn't this change?

  35. philg says:

    Hi, I have two dhcp 2012 r2 servers with loadbalance 50/50, state switchover 60minutes, MCLT 60minutes. When I shutdown the server1, the server 2 shows then partner down, but the ip pool doesn't change from 50% to 100%?
    Why doesn't this change?

  36. Philg says:

    Hi, ok thanks for your quick response. For scope replications, it's also possible to start twice a day the "Invoke-DhcpServerv4FailoverReplication -Force" command instead of the your tool? What do you mean?

  37. darrenk says:

    Can someone please explain to me the logic employed for requiring manual syncronization of DHCP reservations and then providing a tool that effectively makes a scheduled task to maintain syncronization? Could you imagine the havoc that would ensue if Active
    Directory worked the same way? I just lost a few dozen reservations because my DHCP failed over and I had no idea that reservations weren't synchronized.

  38. Saket says:

    Hi,

    If primary server is down, and secondary doesn't sync with primary. what could be the issue.

  39. Tamas says:

    Hi TeamDHCP,

    I would like to confirm my experience. I have 2 Windows Server 2012 DHCP configured in Hot standby mode. I'm using your powershell script to keep up-to-date the 2 servers, but i noticed the sync is working only if i made the changes on the active DHCP server!
    Is it correct?
    Is there any way to make the sync happen automatically if i make any changes on the Stand-by server?
    Would you like to change your script to support this scenario or it is absolutely impossible?

    Thanks in advance.

    Tamas

  40. Tamas says:

    Hi TeamDHCP,

    Thanks for your reply.
    I think it would be really appreciated if you could implement the two way sync into your script.
    It's a really helpful stuff now but it could be more powerfull with these changes! 🙂

    Thanks.
    Tamas

  41. Victor says:

    hi, am getting this error when I try running the DFACS

    – Get-DhcpServerv4Failover : Failed to enumerate failover relationships on DHCP server OJTDHCP01.
    At C:DhcpFailoverAutoConfigSyncToolDhcpFailoverAutoConfigSyncTool.ps1:165 char:35
    + $script:includeRelations=(Get-DhcpServerv4Failover).Name
    + ~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : PermissionDenied: (PS_DhcpServerv4Failover:root/Microsoft/…erverv4Failover) [Get-DhcpS
    erverv4Failover], CimException
    + FullyQualifiedErrorId : WIN32 5,Get-DhcpServerv4Failover

  42. Great article. Thank you!

  43. Great article. Thank you!

  44. KatTer says:

    We got configuration with Clustered DHCP’s. For propper relationship replication I’ve added serverName (clustered) to scipt parameters and all powershell dhcp commands. Because we want multimaster synhronization I’ve added event filtering by event source
    account. This allow not sync back deactivate scope changes made by "Invoke" command. Script need also change for verification of replication partner if synchronization is running. I did not change it because of firewall rules in our environment verification
    is unsuccessuf so all is working. That why script need more improvement. If this kind of functionality could be usefull please contact me any1 from dhcp team for verification changes made by me in script and make rest improvement.

  45. Subhra says:

    My DHCP lease address are not replicating at all after configuration of DHCP server 2012 R2. Scope details and option has been force replicated but we need the lease to replicate which is not happening.

  46. teamdhcp says:

    Subhra, if the DHCP failover relationship has been configured successfully and the failover relationship is in NORMAL state, the leases should replicate automatically. How are you verifying that the lease is not replicated to the second DHCP server.

  47. rajk says:

    Hi TeamDHCP,
    It is very informative. Here is my question… I have two DHCP server with hotstandby mode enabled, MCLT:1 hour and automatic failover enabled, and we use IP relay agent therefore the two DHCP servers IP has been added into our switch like primary helper and
    secondary helper. What if our primary server goes down , will our primary IP helper get the IP from partner server? Thanks.

  48. Cornel Kaufmann says:

    I just modified a bunch of my DHCP scopes using a small PowerShell script. To my surprise, this created NOT a single event log entry and as a result DFACS did not detect the modifications and sync never happened. Doing the same changes with MMC does create
    event log entries and all works as expected. I’d expect DHCP scope modifications show up in the event log anyway – if applied in GUI or using PowerShell should not make a difference?

    is this a bug, a feature or just me missing something?

  49. teamdhcp says:

    Hello Cornel, the DHCP server events should get logged regardless of whether a scope modification is made through PowerShell or MMC. Could you please share the script or the snippet of the script which performs the scope modification.

  50. teamdhcp says:

    Hello RajK, IP helpers in the switch need to be configured with IP addresses of both DHCP servers such that each DHCP request is forwarded to both DHCP servers. I am not sure if the primary/secondary IP helper configuration would do that. I am not an expert
    on switch configuration but I am inclined to think that you need to have both IP addresses in primary IP helper configuration.

  51. who's name? says:

    @teamdhcp: the relevant code is pretty simple:

    $scopes = Get-DhcpServerv4Scope
    foreach ($scope in $scopes) {
    $end = $scope.EndRange.IPAddressToString
    $newend = $end -replace ".254$", ".239"
    Set-DhcpServerv4Scope -EndRange $newend -ScopeId $scope.ScopeId -StartRange $scope.StartRange
    }

    the above correctly lowers the EndRange on all my maxed scopes, but does not create a single event log entry 🙁

    what I just found out: the problem does not seem to be related to GUI vs PowerShell, it's the attribute I'm setting. same code as above but changing LeaseDuration instead of EndRange works as expected. and same results using GUI or PS – my previous message
    was incorrect about this.

  52. teamdhcp says:

    The DHCP server does not log an event for change of IP address range of a scope. All other parameter changes are logged.

  53. cornel says:

    oh well – that's pretty close to the worst case I was worried about. any explanation why it got implemented like this?

    any chance to have this fixed – or it's a "documented" feature now?

    thanks a lot for your quick reply

  54. teamdhcp says:

    Hi Cornel, please open a support case towards fixing this. By the way, the DHCP server events are documented here
    https://technet.microsoft.com/en-us/library/dn800668.aspx

  55. belpad says:

    Hello teamdhcp, the documentation for Automatic syncing of scope configuration changes between 2 DHCP failover servers pertains to Server 2012. Does it still apply to Server 2012 R2 or has this functionality been built into the OS now?

    Thanks.

  56. teamdhcp says:

    Hello Belpad, you can use IPAM 2012R2 to manager DHCP scopes which are configured for failover. IPAM 2012R2 makes any changes done by admin to failover scopes to both the DHCP servers in the failover relationship.