How to fix/repair broken Exchange 2007 counters

Article
10/21/2009

I commonly get calls on the inability to see performance counters in Performance Monitor (perfmon) and the inability to query them through WMI. I thought I would take some time to write about how to look for any problems with Exchange Performance Counters and then provide some high level insight on how to possibly fix them. Most of this information applies to Windows 2003 servers.

If the counters are not being shown at all, the first place to check is the registry to see if the counters are not disabled. Here is a snippet of what one of the registry keys would look like

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ESE\Performance]
"Close"="ClosePerformanceData"
"Collect"="CollectPerformanceData"
"Library"="C:\\Program Files\\Microsoft\\Exchange Server\\bin\\perf\\%PROCESSOR_ARCHITECTURE%\\eseperf.dll"
"Open"="OpenPerformanceData"
"PerfIniFile"="eseperf.ini"

If you also see a value of Disable Performance Counters in addition to the above default entries and is set to a nonzero value, the counter at one point had a problem loading and the Operating System disabled them for whatever reason. Set the value to 0 and then close and open Perfmon again to see if you can see the counters again. More information on this Disable Performance Counters setting can be found here . If this works for you, then whew, that was an easy one….

If the Performance key is missing for a particular service, then we have bigger problems. I am not sure what causes this key to get removed, but if the key is not there, Perfmon or WMI does not know how to load the counters. There are a couple of key required parts that you need to understand before we can load any Performance counter, not just Exchange. The key pieces that are needed to reload any Performance counter is the following:

Performance key must be created under the specified service
Library path must be specified to the appropriate DLL for the service
A PerfIniFile must be specified which is the name of the ini file that will reload a specific services performance counters
Lastly, we need to have the Close, Collect, and Open values which specify what method is used to retrieve the Performance Counter Data. Note: This is unique to each service, so they will not always have the same information

If we have these key pieces of information in the registry, we have the ability to reload said services performance counters. If we take the example above for ESE, if we opened a command prompt and navigated to the C:\Program Files\Microsoft\Exchange Server\bin\perf\AMD64 directory and then typed lodctr eseperf.ini, this will reload the counters for ESE. If the counters were loaded successfully, we should now see that we have also added the First Counter, First Help, Last Counter, Last Help values as shown below.. These values correspond to specific data that was loaded in to the Perflib library.

If everything went well and you reopen Perfmon, You should now hopefully see the counters loaded. If they have not loaded, refresh the registry to see if the Disable Performance Counters key shows back up, If not, check the application log for Perflib errors which should provide additional information regarding why these counters did not load successfully.

If you don’t know already, on Windows 2003 servers, you can actually pull up performance counters using the command Perfmon /WMI. If you do not see the newly added counters, then they have not been synchronized with the WMI repository yet. To help force this along, you could run wmiadap /f to force the reload of all counters in to the WMI repository.

If this was successful, you will now see some additional Wbem entries as shown in the below pictorial.

Pulling up Perfmon /WMI again should hopefully show the counters that you are looking for. In some cases, monitoring software can still not pick up the newly added counters until the WMI service (Windows Management Instrumentation) has been restarted.

If you ever wanted to unload Performance counters, one might think that you could simply unload the counters by running unlodctr eseperf.ini. Unfortunately, this will not work because the unlodctr utility requires that a service name be passed in instead of the ini file. To find the actual name of the service, you could simply open up eseperf.ini and at the top of the file, you should notice an entry similar to the following

[info]
drivername=ESE

Ahh, there is the service name. Now if I run unlodctr ESE, this will now be successful. Doing this will remove the First Counter, First Help, Last Counter, Last Help values from the registry.

Hopefully you are still with me at this time. Now what happens if the performance registry keys for all of your services went missing, now what do you do? Reinstall, flatten the box and reinstall to get them back? Well, unfortunately, there is not a direct way of recreating these registry keys as they are created during the installation of Exchange.

The majority of the folks just export the data from another server, clean out any of the data that references performance counter data from the old server and then import them on the affected server. This does in fact work and is what I am going to talk about next on how to recover from a complete Performance key meltdown.

Attached to this post is a zip file that contains all of the Performance keys across various different role combinations such as MBX, CAS, HUB, HUB/CAS, HUB/CAS/MBX. I’ve done all of the dirty work for you, so all you have to do is to perform some very simple modification steps to the files and then you are in business.

CAUTION!!!: DO NOT IMPORT these registry keys if the Performance registry keys already exist as it will overwrite the data that currently exists in the registry and could potentially break your Performance counters that are currently working. If you only need to reload the Performance key for a single service, then pull out the data for that specific service, save it to a reg file and then import only that data. Basically use it as a reference point to help get you back running again.

If you feel the need to use these reg import files due to all of the performance keys missing for all services, simply open the file that pertains to the role that you have installed and verify that the paths are correct to the correct library files. By Default, we install Exchange in to the to c:\program files\microsoft\Exchange Server directory, so if Exchange was installed outside of the default directory, you will need to update the file manually. Let’s take the ESE performance key below:

Here you will see that library has the following value:

"Library"="C:\\Program Files\\Microsoft\\Exchange Server\\bin\\perf\\%PROCESSOR_ARCHITECTURE%\\eseperf.dll"

What you will need to do is to replace the path with the correct path in which you have installed Exchange. If you installed Exchange on D: in the following directory (D:\\Program Files\\Microsoft\\Exchange Server\\bin), you would simply need to modify the first part of the path to show D:\\ instead of C:\\. A quick find and replace should work to hit all Performance keys. If you have installed it in to another Directory outside of the default paths, then you have a little more work to do to replace the path information. Just remember that for each backslash (\), you have to include double-backslashes (\\) to allow for proper importing of the reg files.

There are only a handful of entries you have to manually modify, so this really shouldn’t take too long. Once you have the paths changed, save the appropriate file as a .reg file and import it by double-clicking on the file. Verify the Performance reg keys are good and valid by opening the Registry Editor to verify.

Once the keys have been verified in the registry and look good, you can then run the powershell script to reload all of the Exchange performance counters. Simply copy the ReinstallAllPerCounters.pst.txt file to the Exchange server and then remove the .txt extension on the file. Open the Exchange Management Shell and then run the script. The screenshot below shows each ini file attempting to be loaded. Of course, on my server, I already had all of the performance keys, so we simply reported that the counters were already installed.

Note: If you would like to transfer this data to WMI, simply type Y when asked.

Once this has completed, be sure to check the application event log for details on any counters that failed to load. If everything went well, voila, you should have most if not all of your Exchange Performance Counters back once again.

If the counters are still not showing up for whatever reason in WMI, you can run the following 2 commands to clear the WMI Adap cache and then re-sync the counters again to hopefully kick start things once again.

See https://msdn.microsoft.com/en-us/library/aa394525(VS.85).aspx for more information on some of the additional commands included with the winmgmt command.

Hopefully this will help you out trying to get your Exchange performance counters going once again.

FixE2k7PerfCounters.zip

How to fix/repair broken Exchange 2007 counters

Additional resources