This post is to clarify and answer some questions raised by my post last week on recovering when your Exchange log drive fills up.
Several readers wrote in with additional workarounds, and I also had quite a few questions offline. So I’ll address all of them here.
First, with regard to clearing out excess log files by doing an online backup. This works great–if you can start the databases. After running out of log drive space, sometimes you will be able to re-start the databases, and sometimes you won’t. It depends on how much of the reserve log Exchange had to use (I’ll explain more about the reserve log below).
Assuming the database mounts again before clearing log files, you will need to keep additional data from coming in to the database. All it will take is one more 5 MB message, and that will almost certainly fill up the last reserve log and force a shutdown right in the middle of your backup.
Second, with regard to turning on NTFS file compression on the log file drive:
Using NTFS file compression with Exchange has always been a risky thing to do, but mostly with regard to database files. Over the years, we’ve seen administrators compress Exchange database files, and it seems to work OK–for a while, or until the database gets bigger than 4GB in size. NTFS compression really should be avoided with very large files of any kind, and with Exchange databases regardless of size.
In Exchange 2003 SP3, we decided to build some prevention into Exchange instead of just scolding people about compressing their database. Nowadays, we will either automatically decompress an Exchange database, or make you do it before the database can mount. You can read more about that here:
We also decompress log files on-the-fly if you put them in a compressed folder. That’s going to cause some performance overhead, if nothing else. If you run out of space on a compressed log drive, and we can’t decompress the *temp.log file or the reserve log files before we try to use them, Exchange is going to shut down dirty.
So compressing a log folder doesn’t buy you much extra space in normal operation: every new log file you create will be uncompressed before it’s used.
As a best practice, we strongly advise you not to compress Exchange logs. We have seen cases in the past where compressed log files ended up corrupted. Whether that was coincidence or not, I can’t say. Log file replay times are also likely to be extended by compressing log files. However, the data corruption risk from compressing log files is certainly not as great as the risk from compressing database files.
If you do compress logs to get out of a disk full situation, don’t do it to the current log, the reserve logs or a *temp.log (if one exists). In any case, compressing Exchange database or log files is not supported by Microsoft. Unless you literally have no other drive to move the log files too, and your only other choice is deleting them permanently, I would recommend moving log files instead of compressing them.
Another possibility for recovery that I didn’t mention in the previous article is to change the log file drive. You can do this in Exchange System Manager on the properties for the storage group–all you have to do is point the transaction log location to a different drive. Exchange will automatically move all the logs for you, regardless of the shutdown state of the databases.
(Moving the logs will not prevent you from replaying transaction logs into the databases. However, moving the databases will prevent you from doing transaction log replay. Exchange System Manager will not let you move a database until it is in Clean Shutdown state.)
Changing the log drive is a pretty simple solution, but I don’t consider it a preferred solution, for several reasons.
In a large production environment, the log drive has often been chosen for very specific characteristics of performance and fault tolerance. Dropping the logs off in some place that happens to have enough space at the moment might have big performance and recovery implications. Also, this might put you out of compliance with corporate standards, or lead to other bad things (like filling up your operating system drive).
Finally, I’ll say a little about the reserve log files. I didn’t mention them before because the previous blog entry was already getting way too long. Understanding reserve log files is not essential to being able to recover from a drive full condition, but people are obviously curious about them.
If you look on your log drive, you will see the res1.log and res2.log files. These files are the same on every Exchange server and storage group–they are placeholder logs, filled with a simple repeating pattern, intended to be used if Exchange runs out of disk space. In any case where the next temp log can’t be built, Exchange will suspend new changes to the database and convert res2.log into the current log. Transactions already in progress in the previous log will be secured to the new log file and the database will be prepared for an orderly shutdown. If necessary, res1.log will also be used (although usually this isn’t necessary).
In Exchange 5.5, you could only have two databases on a server, and one of them was a public database. 10 MB of reserve log space was usually more than enough to allow clean shutdown of the database. In Exchange 2000, with up to 5 mailbox databases running against a single set of logs, and with the databases tending to be much larger and more active, we discovered that it was becoming more frequent for databases to end up in Dirty Shutdown state after a log drive filled up.
Service packs for Exchange 2000 have made significant efficiency improvements in this area, and now it should be rare to see an Exchange 2000 server shut down dirty after a disk full condition. Even if the database does shut down dirty, you’re not at any risk of data loss as long as you don’t lose any of the required logs.
Still, we much prefer to shut down cleanly because then you don’t need any log files in order to re-mount the database. If Exchange shuts down cleanly, there’s no chance for an administrator to be stuck with unmountable databases from accidentally deleting required log files.
You may still on occasion see Exchange shut down dirty after a disk full error. Most of the time that will be because Exchange has been denied access to the reserve log files, or the log drive has become completely unreachable. Remember, also, if you compress the reserve log files, and then run out of disk space, Exchange will shut down dirty because there won’t be room to uncompress the reserve logs before using them.
For all these reasons, you shouldn’t count on Exchange shutting down cleanly after a problem on the log drive. Knowing how to check database shutdown states, and how to tell which log files are still needed, are essential troubleshooting skills for an Exchange administrator.
– Mike Lee