The Exchange Troubleshooting Assistant suite of tools was developed to help customers and Customer Service and Support (CSS) isolate issues faster. ExTRA is integrated into the toolbox in Exchange Server 2007 and is available directly from the Exchange Management Console.
Very often, running ExTRA is the first troubleshooting step that CSS will require in the case of Disaster Recovery, Mailflow and Performance cases.
As customers, running the tool when one of these three issues occurs will allow you to either solve the issue yourself or at the very least provide an upfront set of data to CSS for them to analyze and possibly dramatically reduce the time to isolate these problems.
These tools do really represent a massive win for customers, CSS and the Exchange Product Group. The following article details great success stories when using this tool suite:
As customers, there are three things that we'd like to ask of you:
- FAMILIARIZE YOURSELF WITH THE TOOLS
Download ExTRA from www.exbpa.com. All you need is a workstation running .NET 1.1 Framework 1.1. To get the most out of these tools requires you as customers to take the time to familiarize yourself with them.
- RUN THE TOOLS BEFORE YOU CALL CSS
Experience through a comprehensive analysis of cases has shown us that the earlier in the life of a call that these tools are run, the quicker the time to isolate and resolve.
- SHARE YOUR EXPERIENCES/FEEDBACK TO US
Do you have any ideas of how the tools could be improved? We want to hear from you on the following newsgroup:
Thanks for partnering with us to help make these tools the number one set of tools for administrators worldwide!!!
Now let's go into the overview each one of the tools contained in the suite:
Exchange Server Mail Flow Analyzer (ExMFA)
The Microsoft Exchange Server Mail Flow Analyzer Tool is a diagnostic tool that helps customers to perform the following functions:
- Start with the mail flow symptoms and move customers through the correct troubleshooting path.
- Provide easy access to various data sources that are required to troubleshoot problems with mail flow.
- Automatically diagnose the retrieved data and present an analysis of the possible root causes.
- Suggest corrective actions.
- Provide guidance to help users manually diagnose the data where and when automation is not possible.
The Microsoft Exchange Server Mail Flow Analyzer Tool collects data from different sources to produce the recommended remediation advice. Data is collected in discrete steps, and then it is analyzed. Depending on the results of each step, additional data collection may be necessary. Some of the data, such as the symptom, requires user input. Other data is collected automatically from the problem server.
Selecting the Symptom
As the first step in the analysis process, the Exchange Server Mail Flow Analyzer asks what symptom is being seen. Based on the symptom, different troubleshooting steps occur.
Users are complaining about receiving non-delivery reports when they send messages.
Select this symptom if messages bounce and non-delivery reports (NDRs) are returned to the senders. NDRs may be returned from outside the corporate Exchange organization or inside the Exchange organization. The Exchange Server Mail Flow Analyzer will try to guide users in determining the correct troubleshooting steps for this symptom. Be aware that because it is possible that the root causes are outside Exchange Server, the troubleshooting guidance is very important.
Messages coming from external senders over the Internet are delayed or are not received by some recipients.
This and the next symptom are similar. Select this symptom if the messages are not coming into the corporate Exchange Server organization, or the messages are not coming through the non-Exchange gateway servers at the boundary. The Exchange Server Mail Flow Analyzer will examine both inside the Exchange Server organization, and the gateway.
Messages that are destined for external recipients on the Internet are delayed or are not sent to some recipients.
Select this symptom if the messages that are supposed to be delivered outside the corporate Exchange Server organization are staying in the corporate Exchange Server organization, or the messages cannot go through the non-Exchange gateway servers at the boundary. The Exchange Server Mail Flow Analyzer will examine both inside the Exchange Server organization, and the gateway (as it did in the previous symptom).
Messages are backing up in one or more queues on a server.
Select this symptom if messages are staying in Exchange Server queues, notably, the Remote Delivery Queue, the Awaiting Directory Lookup Queue, and the Local Delivery queue. The backup affects the messages routed in the same Exchange Server organization in addition to the messages that are coming from and going to the Internet, depending on the path where messages are routed. Working on this symptom could then possibly address the second and third issues in this list of symptoms.
Like the other Analyzer tools, you can run the tool from a remote workstation and you do not need to install it to the server having mail flow problems. Also, the additional performance impact should be minimal when run remotely.
Microsoft Exchange Disaster Recovery Analyzer (ExDRA)
Disasters can happen in many different situations for example, Power Failure, Hardware Failure and Abnormal Application Execution. Each of these scenarios results in a Critical Situation where time to restore service with minimal data loss is the key
However, troubleshooting database recovery failures requires too much background knowledge, eg:
- Database and Log Signatures.
- Database States.
- Log File Generations.
- Logs Required.
Understandably, it's very easy to give up on databases that could be easily recovered without restoring from backup. EXDRA makes it faster to discover and assess options available. The tool when run allows customers to systematically make the correct decision in order to bring a server online.
The Microsoft Exchange Disaster Recovery Analyzer Tool scans Exchange Server databases and available transaction log files for issues that affect database recoverability. The tool reports missing or corrupted log files and recommends steps that you can take to bring the database to a clean, mountable state.
The Exchange Disaster Recovery Analyzer includes these features:
- A troubleshooter for problems with mounting databases caused by conditions that prevent successful transaction log replay.
- A wizard to safely move excess transaction log files to resolve "log drive full" conditions.
- An event log filter and viewer that lets you easily view and sort events related to Exchange databases.
- The ability to repair databases (only if tool is installed on Exchange server).
The Task Center page is the heart of the Microsoft Exchange Disaster Recovery Analyzer. The Task Center is where you choose the symptom or task that you want to address. Before you access the Task Center page, you must choose a server to work on, or select manual mode.
The following functions are available in the Task Center:
Troubleshoot Database Mount Problem
This troubleshooter reads an Exchange server's Application log to make an initial diagnosis of problems on the server. If multiple issues are found, you can choose the issue you want to address first. Additional analysis is then done to confirm the issue and to help you correct the problem.
Analyze Log Drive Space
This wizard checks available transaction log file space for each storage group on a server, and it will help you move logs to a backup location if you are low on space.
If an Exchange Server storage group runs out of disk space for its transaction log files, the databases will shut down. You cannot start the databases again until you correct the disk space problem. If you delete the wrong log files, you may leave your databases permanently unstartable. The Exchange Disaster Recovery Analyzer will determine which log files can be safely moved and will move them to a backup location on another drive or server. You should not delete moved logs until you have successfully taken a new full backup of all databases in the storage group.
Show Database Related Event Logs
Multiple warning and error events in the Application log may be associated with a database problem. It can be difficult to find that related events follow the "story" they tell. This function of the analyzer filters for events related to the Exchange database and presents them in a format that is easy to scan and sort.
This task can be used to repair corrupted databases which cannot be brought to the consistent mode through other approaches, such as log replay. Since repair can cause permanent data loss, administrator is advised to restore data from backups if available.
This task is enabled only when the tool is running against the local machine. The same Extensible Storage Engine (ESE) related binaries, such as ese.dll, used by the Exchange server will be used by this task. This prevents the version mismatch problem between database files and binaries.
In each step that the tool goes through, the task will display the action it is working on and the percentage of the progress. If administrator decides to cancel the action or close the tool among the above steps, the running step will be properly canceled to avoid leaving a long running process on the production server which can severe impact the server performance but will not undo actions already completed. The cancel action can take up to several minutes to complete, and the completed steps will not be rolled back.
Microsoft Exchange Performance Troubleshooting Analyzer (ExPTA)
Performance issues can be the worst to for CSS and customers to troubleshoot, since they are intermittent by nature and the reasons for these can number many.
The challenges of isolating these types of cases are as follows:
- Troubleshooting flow is complex and requires expert knowledge of entire system
- Data comes from many sources (eg, perfmon, event log, EWT trace, FCL tracing)
- Problems in one area may be due to an apparently unrelated problem
- Troubleshooting information is scattered
The Microsoft Exchange Performance Troubleshooting Analyzer Tool locates and identifies performance-related issues that could affect servers that are running Exchange. To find these issues, it looks at various settings, logs, and active data capture.
The Microsoft Exchange Performance Troubleshooting Analyzer Tool collects a variety of data from different sources to produce the recommended remediation advice. Data is collected in discrete steps, and then it is analyzed. Depending on the results of each step, additional data collection may be required. Some of the data, such as the symptom, requires user input. Other data is collected automatically from the problem server.
Selecting the Symptom
As the first step in the analysis process, the Exchange Performance Troubleshooting Analyzer asks for the symptom. Based on the symptom, different troubleshooting steps occur.
The number of RPC operations per second is higher than expected.
Select this symptom if the server that is running Exchange appears to be busier than expected. As part of the troubleshooting, the Microsoft Exchange User Monitor (ExMon) tool tracks the remote procedure calls (RPC) from Microsoft Office Outlook and other MAPI clients. If the Exchange Performance Troubleshooting Analyzer detects an abnormally high user load, the tool isolates the source of the problem, such as the individual user, that is causing the server performance problem.
Multiple users are complaining of delays when they are using Outlook, or are frequently seeing the Outlook cancelable RPC dialog box.
Select this symptom if users are seeing unacceptable time delays when performing operations within the Outlook client. For example, if there are severe delays when users are switching from one folder to another, such as from Inbox to Calendar.
Some versions of the Outlook client show a Requesting data window if the server does not respond to a client request in a timely manner. These issues are typically attributed to server load or a bottleneck on the server itself. The Exchange Performance Troubleshooting Analyzer will help isolate the source of the bottleneck.
The number of Outstanding RPC requests is high
Select this symptom if the number of Outstanding RPC requests is high
ExPTA can trace and analyze the performance of internal calls that the Microsoft Exchange Information Store service (Store.exe) makes. These long function calls can cause performance issues. You can use trace logging to isolate the component and the function call that may be the root cause of the performance issue.
The trace logging functionality in the Microsoft Exchange Information Store service keeps track of the relevant calls to the following components:
Performance data collection and analysis will not work when ExPTA is run on Windows Server 2000 and Windows XP Sp1 machines. ExPTA must be run on Windows XP SP2 or Windows Server 2003 machines. Note ExPTA can collect and analyze data from Windows Server 2000 machines, as long as ExPTA is running on a Windows XP SP2 or Windows Server 2003 machine.
Now that you have had a good read of all the great things in ExTRA, please also take the time to send feedback to us on suggestions, questions on functionality for these tools.
Click on the following link to take you to the Microsoft newsgroups site (navigate down the page and you will see the Exchange Tools site, which you can just browse or contribute to using Microsoft Outlook Express):
Some related previous blog posts on this blog: