Change Block Tracking needs to be reset if another backup product has protected a VMWare VM prior to DPM

In this scenario, the VMWare Host\VCenter server has successfully been connected in DPM. However, Replica creation, Consistency Checks, Recovery points fail with the noted error below (Some other similar errors but with same resolution).

DPMUI Error:

DPM encounterd error from VMware while trying to get ChangeTracking information. Server – 10.10.10.10 ErrorCode – FileFaultFault (ID 33621 Details: Internal error code: 0x80990EF3)

DPMRA Log on DPM server:

vmwarehelpers.h(25)                    WARNING           VMWareErrorCode = FileFaultFault, Message = Error caused by file /vmfs/volumes/53dc5a20-ce9ee1ed-635b-288023aad568/Servername/Servername.vmdk

NOTE: If you are getting these errors, but have NOT protected a VMWare VM with any other product prior to DPM, this post may be helpful.

Review the “Steps to reset Change Block Tracking” section below if the VMWare VM has been protected prior to DPM backup. The article below has details. However, the steps I confirmed are slightly different, as step for step in the KB did not resolve the issue.

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2139574

Environment Setup

  1. DPM must meet the prerequisites for protecting VMWare servers. (Link Below)https://technet.microsoft.com/en-us/library/mt742097(v=sc.12).aspx
  2.  Add the VMWare ESXI host or vCenter server in DPM. This can be done via the Agents tab under Management in the DPM console. The TechNet article Manage VMware with DPM has step by step instructions.
  3. To protect VMWare VMs “change block tracking” must be enabled on the desired VM within VMWare. Both the VM and the disks need this feature enabled. This needs to be performed on all VMs needing protection. See https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1031873 .

 Steps to reset Change Block Tracking

  1. Log in to vSphere\vCenter
  2. Shut down the desired VM
  3. If you are not sure about the VM being protected before, steps 3-5 will confirm. In vSphere locate the desired VM. On the Summary tab under resources you’ll see Storage. The items listed under storage are the VMWare Data Stores. Right click on the Data Store and “Browse Data Store”

60144-1

4. The list on the left will contain all servers associated with this particular data store. Locate the desired VM in the list and double click to open the Data Store file.

60144-2

WARNING:

Step 5 can cause outages for the VM if the correct VMDK is not identified.

Brief description of VMWare VMDK files:

In VMware a VMDK serves the same purpose as a Windows VHD. The VMDK is the VMs virtual disk. In the data store you’ll see several VMDK files and this is why we need to be cautious as renaming the running virtual disk can cause issues.

5. In the Data Store, locate the CTK.VMDK (checkpoint VMDK). The article says to delete this file. To be safe it can be renamed until backups are working. Then you may want to consider deleting this file as it will continue to consume space in the Data Store. Exit the Data Store

6. In the vSphere console a list of servers will populate under your host. Locate the desired VM, right click, Edit Settings, Options Tab, Configuration Parameters.

60144-3

7. Within the Configuration Parameters locate the two below rows and set the value to false.

scsi0:0.ctkEnabled

ctkEnabled

Apply, and exit VM Settings.

60144-4

NOTE: The following steps are not mentioned in the VMWare article. I was not able to get a successful Change Block Tracking reset without the next set of reboots.

8. Power on the VM

9. After the VM boots, perform another shut down

10. Once the VM has been shut down, Right click on the VM, Edit Settings, Options Tab, Configuration Parameters. Within the Configuration Parameters locate the two below rows and set the value to True.

scsi0:0.ctkEnabled

ctkEnabled

Apply, and exit VM Settings.

60144-5

11.Enable protection for the VM in DPM.

Links to VMWare and DPM documentation:

Reset CBT: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2139574

Enable CBT: https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1031873

DPM Prerequisites:

https://technet.microsoft.com/en-us/library/mt742097(v=sc.12).aspx

DPM Manage VMWare:

https://technet.microsoft.com/en-us/library/mt742101(v=sc.12).aspx

-- Josh Gulick, Support Engineer, Microsoft