VMM 2008 R2: Hyper-V Host is in Perpetual “Pending” State & VMM Service Crashes

Recently, I was doing some Hyper-V physical hosts moving and I found that one Hyper-V host getting managed by VMM got in a state whereby it found itself “sick.”  I describe sick meaning that Virtual Machine Manager 2008 R2 (VMM) could no longer manage the host (who was actually getting managed by another VMM instance) but as administrator I was unable to remove the host.  Instead, the outcome was a physical host was shown in the legacy VMM instance as in Pending state like below -

image

If you attempt to remove the host, the VMM Service will crash within seconds of the job starting.  You will receive the following information in the jobs detail after restarting VMM -

image

In short, the removal of this physical host isn’t possible using any technique within the VMM Administrators console or via a PowerShell command-let.

Background:  How did I put myself in this spot?

To help many of you understand how I put this server in the particular spot where it ended up being un-manageable yet un-capable of getting removed.  I felt it important to explain how this server, called out as “Server Name” got into this state -

  • Server was moved into a unique Host Group in VMM
  • I moved the physical host to the other domain where this VMM instance was running
  • I added the service account for this VMM instance to the administrators group on this host
  • I then went to another VMM console that manage other hosts and use Add Host and selected Associate Host with this server optionimage

After this, I assumed that the “existing” VMM host would get removed from the old VMM instance or that I could manually remove it.  This wasn’t the case though as when I attempted to remove the host removal process would start and within a few seconds it would crash VMM.

When using DebugView to do advanced debugging, the following was shown when this action was taking place -

image

Who Cares – Just help me Rid Myself of this Physical Host from <#@$>

For many of you, the root cause is less important and the resolution is the key to your happiness.  Besides, this is the very reason you are reading this blog.  I took sometime today to dig into how I could rid myself of this host and clean up my VMM environment.  The following steps are used to remove yourself of the host.

NOTE: The first step you should do is backup your Virtual Machine DB before moving forward with this process. This will ensure that, in case of an undesirable outcome, you can restore your VMM instance. Please proceed at your own caution.

In this process, we will bounce around as we solve this complex problem but the end-result is you will now have the un-managable host out of VMM.  To do this, we will alter the VMM database by doing the following -

  • Remove objects from dbo.tbl_ADHC_VirtualNetwork
  • Remove objects from dbo.tbl_ADHC_Host
  • Remove objects from dbo.tbl_ADHC_HostNetworkAdapter
  • Remove objects from dbo.tbl_WLC_VObject

This is an ordered approach as there are constraints all through the VMM VirtualManagerDB (as their should be to maintain data integrity) so it isn’t an easy 1..2..3 process.

The data you will need to collect from you broken host are the following -

  • All MAC Addresses for the NICs

Let’s get on with it…

The following steps must be followed exactly or you will receive errors that constraints have been violated. Start Here.

Locate your broken Host’s HostId in the database

  1. Open SQL Management Studio
  2. Connect to the VirtualManagerDB using Administrator credentials
  3. Execute this SQL statement -
    1: Select HostID,ComputerName from dbo.tbl_ADHC_Host

You should now locate the server name and the copy the HostID (you will use it a lot!).

image

Delete all VM objects associated with this Host in the VObject Table

  1. In SQL Management Studio, use the following command to delete any objects associated with this HostId

Delete from dbo.tbl_WLC_VObject where HostID = 'e32a0641-0268-4ba5-b8fb-8249cce01642'

This should clean up any virtual machine objects that are attached to this Host.

Delete all Host Volume objects from the Host Volume Table

  1. In SQL Management Studio, use the following command to delete any objects associated with this HostId

Delete from dbo.tbl_ADHC_HostVolume where HostID = 'e32a0641-0268-4ba5-b8fb-8249cce01642'

This will remove all references to the volumes that are available on this host, in my case it was the C:\ & D:\ drives.

Delete References for Host Adapters in the HostNetworkAdapter Table

This is where you will need to have the NIC MAC addresses from the host that is currently unable to be managed. This can be obtained using the ipconfig /all command in Windows.

  1. In SQL Management Studio, use the following command to delete the NIC adapters associated with this host using MAC Addresses
 Delete from dbo.tbl_ADHC_HostNetworkAdapter where MacAddress = '00:11:0A:5D:9C:06'

You should do this for each NIC that this server has in the table.

Delete References to Virtual Network in the VirtualNetwork Table

  1. In SQL Management Studio, use the following command to delete any objects associated with this HostId

Delete from dbo.tbl_ADHC_VirtualNetwork where HostID = 'e32a0641-0268-4ba5-b8fb-8249cce01642'

This removes any reference for this HostId for the Network (think Network Tags, etc.).

Finally, Delete the Host from the Host Table

After all this, you’ve found your way to the last step and you should now be able to successfully delete the record related to the Host.  To do this, go back to the first step we did earlier where you determined the HostId and now you can easily delete using the following SQL query -

 Delete from dbo.tbl_ADHC_Host where HostId = 'e32a0641-0268-4ba5-b8fb-8249cce01642'

The command should execute completely and you’ve now cleaned up this rogue Hyper-V host.  I mentioned, I hope, that this isn’t a supported method to do this as you shouldn’t be accessing the database directly but sometimes drastic times means drastic measures.  The end result for my situation is that VMM is now clean and life is good!

Summary

Breathe.  It’s over.  This server is now history.  You can now move on with life.  In today’s post, I spent some time helping you understand how to forcibly remove a host from VMM 2008 R2.  This is linear process that isn’t for the faint of heart and requires you to follow your detective-like leads and use them to obliterate this server from the VirtualManagerDB.  Breathe.  It’s over.  This server is now history.

Enjoy!

Thanks,

-Chris