Troubleshooting VSS and Backup

Article
04/23/2008

[Today’s post comes to us courtesy Rituraj C.]

Some of the most interesting issues that we come across in SBS are those related to Volume Shadow Copy (VSS) and Backup. Today, we are going to discuss the procedure we generally follow to troubleshoot issues related to Backup failures because of problems with VSS.

For any issue that deals with VSS or with backup, the first thing we look for on the server is, how patched it is. Windows Server 2003 Service Pack 2 addresses some critical problems affecting the performance of VSS. If you don’t want to install the service pack, you can look at KB 914962, which provides you a list of all the updates on the service pack. You can then install just the updates for VSS. However, we would recommend you install the full service pack.

Once we are sure we have all the required updates, we need to check the health of VSS on the server. This can be done using the command line.

vssadmin list writers

The output of this command should provide all writers registered with VSS. We need to check the State of the Writers. If the state is anything other than Stable, it might be a cause of concern; however, it does not necessarily have to be a cause of backup failure. Below is an output for the command on one of our test servers:

C:\>vssadmin list writers

Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {449e12cf-6b8f-4d75-a821-bc869d6612c4}
State: [1] Stable
Last error: No error

Writer name: 'SqlServerWriter'
Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
Writer Instance Id: {de3b2861-77f6-43f8-8619-64dc9ea2fbcf}
State: [1] Stable
Last error: No error

Writer name: 'MSDEWriter'
Writer Id: {f8544ac1-0611-4fa5-b04b-f7ee00b03277}
Writer Instance Id: {9036fdfa-05ef-4f7d-9c2f-84fa2f06560a}
State: [1] Stable
Last error: No error

vssadmin list providers

This command lists the registered volume shadow copy providers. It should ideally be 'Microsoft Software Shadow Copy provider 1.0'. You can change it through the registry, if you need to. Here is the sample output obtained by running the command on a working server.

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001 Microsoft Corp.
Provider name: 'Microsoft Software Shadow Copy provider 1.0'
Provider type: System
Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
Version: 1.0.0.7

Provider name: 'ABC DEF VSS Provider'
Provider type: Hardware
Provider Id: {d1da7f61-d080-4ce9-92a1-2c4b819b2787}
Version: 1.5.0.3.5

By default, we would see only the Microsoft Software Shadow Copy Provider. However, some Backup software use their own providers and we may see an additional provider if such Backup software is installed.

A good result on both commands is extremely necessary. For more information on the use of VSSADMIN, try running vssadmin /? on the command prompt for a list of other tests available.

So, let us assume that the problem server is fully patched and we still have problems with VSS. If you have manually switched Volume Shadow Copy providers, or if you have installed a Backup Solution that uses its own Volume shadow Copy provider, switch back to the default provider. To do this, open the Registry Editor, and find and modify the following registry key:

HKLM\SYSTEM\CurrentControlSet\Control\BackupRestore

Value Name: UseMicrosoftProvider
Value Type: REG_DWORD
Value Data: 1

You might need to reboot the server for these changes to come into effect.

If we see that we have a problem with the writers, let us verify that the following keys and values are accurate:

HKCR\CLSID\{FAF53CC4-BD73-4E36-83F1-2B23F46E513E\InProcServer32

Value Name: Default
Value Type: REG_SZ
Value Data: C:\Windows\System32\ES.dll

HKLM\SOFTWARE\Microsoft\EventSystem\{26c409cc-ae86-11d1-b616-00805fc79216}\EventClasses\{FAF53CC4-BD73-4E36-83F1-2B23F46E513E}-{00000000-0000-0000-0000-000000000000}-{00000000-0000-0000-0000-000000000000}

Value Name: TypeLib
Value Type: REG_SZ
Value Data: C:\Windows\System32\eventcls.dll

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\EventSystem\{26c409cc-ae86-11d1-b616-00805fc79216}\EventClasses\{ECABB0C6-7F19-11D2-978E-0000F8757E2A}-{00000000-0000-0000-0000-000000000000}-{00000000-0000-0000-0000-000000000000}

Value Name: TypeLib
Value Type: REG_SZ
Value Data: comsvcs.dll

In case we have a discrepancy on any of these values, change them to reflect the path to the files above.

Re-register the libraries used by VSS as well as the service. To do so, run the commands below:

cd /d %windir%\system32
net stop vss
net stop swprv
regsvr32 ole32.dll
regsvr32 oleaut32.dll
regsvr32 vss_ps.dll
vssvc /register
regsvr32 /i swprv.dll
regsvr32 /i eventcls.dll
regsvr32 es.dll
regsvr32 stdprov.dll
regsvr32 vssui.dll
regsvr32 msxml.dll
regsvr32 msxml3.dll
regsvr32 msxml4.dll
regsvr32 "%programfiles%\Microsoft SQL Server\80\Tools\Binn\sqlvdi.dll"
vssvc /register
net start swprv
net start vss

You can copy these into notepad, save the file as say “vss.bat” and run the same, if you don’t want to type the commands manually.

NOTE: You may find additional versions of the MSXML library. You may put them to the registration list as well.

NOTE: The SQLVDI.DLL might be present at a different path as well. Run a search for it and point to the right path to register it.

If you are unable to register any of these libraries/services, check if there is a known issue and an alternate way to register the same.

In case you find that the output of the vssadmin list writers command is blank, there might be registry corruption that is preventing the system from picking up and displaying the correct information. This is documented on KB 940184. It explains how you can recreate the list of VSS writers.

If we still have problems, we can enable VSS tracing. To do so, copy the commands below into notepad, and save the file as “tracing.bat”. Run the same to make the registry changes required to configure VSS tracing.

net stop swprv
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceFile /t REG_SZ /d C:\trace.txt /f
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceLevel /t REG_DWORD /d 0xffffffff /f
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceEnterExit /t REG_DWORD /d 1 /f
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceToFile /t REG_DWORD /d 1 /f
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceToDebugger /t REG_DWORD /d 1 /f
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceFileLineInfo /t REG_DWORD /d 1 /f
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceForceFlush /t REG_DWORD /d 0 /f

After running the batch file that enables tracing, we need to reproduce the issue. Once we have received the errors, stop the VSS services. You can do this by running the following commands on the command prompt.

net stop VSS
net stop SWPRV

Then, disable VSS tracing by running the following command:

reg delete HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /f

The trace file is created on the root of the C drive. You can also look at the backup logs, Mounted Volumes/Devices, SQL setup Logs, COM Setup, SetupAct, SetupAPI logging, DTC Install logs along with the trace log to gather information on the backup failure. You can also compare the VSS registry keys with a working system.

For problems with SQL or MSDE writers, please make sure that ‘SQL Server Client Network Utility’ (Run cliconfg) shows TCP/IP and Named Pipes enabled in addition to the ‘Enable shared memory protocol’ which should have a check mark against it. To do this:

Click on Start > Run > cliconfg > Enter

You may follow the KB 919023 which explains SQL Server 2005 connectivity and Volume Shadow Copy Service.

When we talk of SBS Backup, it is nothing but a NTBACKUP schedule with the verify switch. When we run SBS Backup, we would see the following information in the backup log file.

Launching NTBackup: ntbackup.exe backup "@C:\Program Files\Microsoft Windows Small Business Server\Backup\Small Business Backup Script.bks" /d "SBS Backup created on 06/09/2007 at 08:23" /v:yes /r:no /rs:no /hc:on /m normal /j "Small Business Server Backup Job" /l:s /p "DLT" /UM

NTBACKUP LOG FILE: C:\Documents and Settings\SBS Backup User\Local Settings\Application Data\Microsoft\Windows NT\NTBackup\data\backup08.log

If the backup fails at the verify process, you may get the following events:

Event Type: Error
Event Source: SmallBusinessServer
Event Category: SBS Backup
Event ID: 5634
Date: 7/25/2006
Time: 5:18:13 AM
User: N/A
Computer: SERVERNAME
Description:
One or more components of Small Business Server Backup failed. For more information, click Backup in Server Management, and view the log files.
For more information, see Help and Support Center at https://go.microsoft.com/fwlink/events.asp.
Event Type: Error
Event Source: NTBackup
Event Category: None
Event ID: 8009
Date: 7/25/2006
Time: 5:18:07 AM
User: N/A
Computer: SERVERNAME
Description:
End Verify of 'E:' 'Failed'
Consult the backup report for more detail. For more information, see Help and Support Center at https://go.microsoft.com/fwlink/events.asp.

If we go back to the backup logs, it may show that the backup completed fine but failed at the Verify. We then need to check the hardware. There is always a conflict with the driver version, hardware etc. So, act accordingly. FYI, The Filter driver drive class for the Tape Drive has a GUID of {6D807884-7D21-11CF-801C-08002BE10318}.

As far as we know, you should be able to resolve your VSS issues with the procedures discussed above. If you would like to read more on VSS, here are some references:

Volume Shadow Copy Service on Technet
https://msdn2.microsoft.com/en-us/library/bb968832(VS.85).aspx

VSS Tracing
https://support.microsoft.com/kb/887013: How to enable the Volume Shadow Copy service's debug tracing features in Microsoft Windows Server 2003

SQL / MSDE Writer connectivity
https://support.microsoft.com/kb/919023: SQL Server 2005 connectivity and Volume Shadow Copy Service (VSS)

COM+ hotfix
https://support.microsoft.com/kb/934016: Availability of Windows Server 2003 Post-Service Pack 2 COM+ 1.5 Hotfix Rollup Package 12

https://support.microsoft.com/kb/936296: FIX: Error message when a COM+ 1.5 application tries to marshal the COM+ object context in Windows Server 2003 Service Pack 2: "An error occurred during a seek operation"

VSS hotfix (current)
https://support.microsoft.com/kb/940349: Availability of a Volume Shadow Copy Service (VSS) update rollup package for Windows Server 2003 to resolve some VSS snapshot issues

We want to hear from you.

Troubleshooting VSS and Backup

Additional resources