You may find that recovery operations from online backup fail when the target of a restore is a Recovery Storage Group that exists on an External USB storage
device. Most commonly External USB storage is implemented as a temporary measure due to lack of physical disk space on the server. Thus, the storage is added, an RSG is
configured and linked, and a restore is attempted. However, there are a number of “gotchas” you need to be made aware of to allow these operations to complete
The “general inspiration” for this post was taken from a real case I worked within Customer Support Services.
Primary Issue & Symptoms:
As mentioned above, I recently worked with a customer who was attempting to
restore an Exchange 2003 Mailbox Database to a Recovery Storage Group that had been configured on an External USB Hard-Drive. This was done because the available disk space on
the production server was not sufficient for restoring the complete mailbox database from backup. Once restored, the customer was going to perform any number of additional
recovery operations (single item recovery, complete mailbox, complete database, etc.). What we found was that the restore job would initialize successfully but the byte count
never increased above 238 bytes. The restore operation would eventually timeout and fail.
A review of the restore log contained the following
Job Completion Status
Job Ended: <time stamp>
Completed Status: Failed
Final Error: 0xa0008488 — Access is Denied
glance the “Access is Denied” error would suggest a permissions related problem. I initially began the call with a review of the Recovery Storage Group’s configuration. The
configuration clearly showed that the linked mailbox store was being written to the USB drive, which was being provided to the system as the f:\ volume. I performed a compare
and contrast of the permissions that had been written to the volume with the other drives and did not notice any specific NTFS permissions anomalies.
size="2">I then proceeded to perform the following physical tests:
size="2">I then proceeded to perform the following physical tests:
- Verified that I could traverse the f:\ via Windows Explorer as the
user performing the restore.
- Verified that I could manually create files and folders on the f:\ as this user.
- size="2">I tested building a new mailbox store on the f:\, built a single mailbox, sent a test message and then verified I could backup the mail store on the f:\.
All of these operations completed successfully. I then proceeded to add the “Test Mailbox Store” to the RSG as a linked database (I actually built
“Test Mailbox Store” within the same production storage group that he was trying to restore), and attempted to restore it to the f:\ (e.g. the USB Storage Device). This failed
with the identical errors as before. I then reviewed the application log which contained the following Error:
Event Source: ESE Backup
Event ID: 950
Description: Unexpected file system error 53 encountered while opening the restore
This event clearly shows that the nature of the failure was during the creation of the Exchange Restore
Environment file (restore.env). The System Error Code of 53 equates to “ERROR_BAD_NETPATH” or more commonly referred to as “the network path was not found”.
I decided to try testing whether or not my “Test Mailbox Store” could be restored to a physical (non-USB device). I did this by removing the RSG from the USB
device, recreating it, and testing the restore. The result was that the restore was successful.
1. The fact that the “Test Mailbox Store” could be restored to an RSG that did not exist on USB storage, immediately made
the USB drive suspect.
2. The fact that the restore could complete successfully suggested no problems with the backup/restore agent or with the
files actually being restored.
So outside of the f:\ simply being a USB device, what made it unique?
and Restore Agents:
When any valid Backup/Restore agent (e.g. one that abides to the rules and methods of the APIs responsible for
Exchange aware backup/restore), actually restores an Exchange Database (regardless of whether or not it resides within a Recovery Storage Group or a production directory), the
Extensible Storage Engine needs to be able to physical map the path specified as the Temporary Location for the restore. It does this via a UNC path that includes within the
path the Admin Share for the target drive.
\\Server Name\<AdminShare of Drive>\Temp Directory Specified For Log Files
So to verify that the shares were configured properly, I had the customer open a command line and
enter: NET SHARE
NET SHARE will list all configured shares on a system. In my case, a review of the output clearly showed that the default Admin
Share had not been created for the f:\ yet was present for all other drives. This explained why a restore to a separate physical disk worked in my previous
troubleshooting step. The fact that share was not present also provided a logically sound reason for why System Error 53 was being documented in the 950 Event.
Once, I had made this observation, I initially tried to create the share “manually”. To do so, I shared the f:\ volume as f$ and manually applied all the
permissions via a compare and contrast of a properly initialized Admin Share. Once done, I re-ran NET SHARE. Although the share now showed as present, the share itself did
have a few noticeable differences (e.g. was not a Default Admin share), etc. (which in this instance would be expected as the share itself had not been actually created by the
I proceeded to retry a restore, which failed with the identical errors.
The fact that the Default Admin share for the f:\ was missing suggested the following:
size="2">An Administrator had manually removed the Admin Share (seemed very unlikely).
- The USB drive was not fully initialized when the
SERVER service had completed startup (seemed very likely).
- “Manually” attempting to create the share has no net effect on providing a
resolution to the issue.
As previously noted, Admin Shares (drive$) are created during startup when the SERVER service fully initializes.
My suspicion was that when the SERVER service created the shares the USB device itself had not been fully initialized by the Disk Management System. Thus, no Admin
Share would have been configured for the f:\ because it would not have been technically valid at the time of assignment by the SERVER service.
tested this hypothesis on my own system and witnessed the identical behavior. If you simply attach an external USB drive you will notice that a Default Admin Share is not
created because the SERVER service is already started.
Taking this a step further, I restarted the SERVER service on my own workstation then
re-ran NET SHARE once the service was back online. The result was that a Default Share had been created for the USB drive on my system.
logical step was to restart the SERVER service on the Exchange Server. There are two primary Exchange Services that maintain dependencies on the SERVER service (as well as the
Computer Browser service which is also dependent on the SERVER service):
- The Exchange Information Store service (MSExchangeIS)
- The Exchange System Attendant service (MSExchangeSA)
Prior to restarting the SERVER service, I had the
customer manually dismount all production stores which was done to force all pending writes into the store. Once done, I manually stopped the Information Store and System
Attendant services, then restarted the SERVER service:
From command line (cmd):
net stop srv
net start srv
With the SERVER service back online, I proceeded to bring the Computer Browser, Information Store and System
Attendant services back online. Once done, I re-ran NET SHARE to verify the Default Admin share had been created successfully:
We then rebuilt our Recovery Storage Group, linked the original database back to the RSG on the f:\, placed our Database, Log and
TEMP paths onto the f:\ and restored successfully!
It would appear that when
External USB Storage is used, Windows does not automatically create the Admin Shares for the USB drive. As previously discussed, Admin Shares are created during system
startup when the SERVER service fully initializes. My “professional opinion” (I am after all an Exchange Specialist not a Windows specialist), is that most likely this
occurs because the External Storage isn’t fully initialized by the Disk Manager System when the SERVER service has completed startup. This is why a “reboot” of the
server wouldn’t correct the issue (it would simply recreate the issue). By restarting the SERVER service with the USB Drive fully initialized, you effectively force the
shares to be re-evaluated and recreated.
So what is important to remember here is that if you ever have to perform a similar operation
(restoring an Exchange Database to an External Drive), you need to ensure that the Admin Share for the Drive is configured properly prior to attempting the restore (as
Exchange needs to directly map to the UNC path specified via the Admin Share). The Admin Share itself is not needed for a backup, but it should go without saying that no
production Exchange Database should ever by physically located on a USB drive. Otherwise, you and I will probably be working on an Exchange Performance related issue in
the near to distant future (hello RPC pop-up).
Hope you find this interesting.
size="2">– Eric Norberg
size="2">– Eric Norberg