Determining the proper App pool for Exchange dump collection

  • Article

At times we may need to collect memory dumps of a specific Exchange process for troubleshooting. For some of its operations Exchange leverages the IIS infrastructure by creating Application Pools inside IIS. Each of these applications runs in separate worker processes.

  •  MSExchangeSyncAppPool
  •  MSExchangeServicesAppPool
  •  MSExchangePowerShellAppPool
  •  MSExchangeOWAAppPool
  •  MSExchangeECPAppPool
  •  MSExchangeAutodiscoverAppPool

 

If we need dumps of one of these application pools as part of a case we need to make sure we are collecting dumps for the correct instance of the W3WP process.

The first possible approach is the use the TLIST command. Generally if we are looking at collecting memory dumps, you will probably have debugging tools already installed. TLIST is included in the debugging tools (the default path for the Debugging tools is “C:\Program Files\Debugging Tools for Windows (x64)”).

Make sure TLIST is in your path and simply pipe the TLIST output to the FINDSTR command for W3WP.

C:\Tools>tlist | findstr w3wp
2540 w3wp.exe
2620 w3wp.exe
3040 w3wp.exe
3424 w3wp.exe
1776 w3wp.exe

 

As you can see, this sample has 5 active W3WP processes on the server.
Now, use the TLIST command again with the “/L” command against each process. This provides a good deal of information about each process.
We pipe the output to FINDSTR as we’re only concerned with the command line used to invoke the process.
Here we can clearly see which process is associated with each application pool.

 

C:\>tlist /l 2540| findstr /i /c:"Cmdline"
CmdLine: c:\windows\system32\inetsrv\w3wp.exe -ap "MSExchangePowerShellAppPool" -v "v2.0" -l "webengine4.dll" -a \\.\pipe\iisipm1738da07-bf2b-4277-986d-3cb5c1b72dd3 -h "C:\inetpub\temp\apppools\MSExchangePowerShellAppPool.config" -w "" -m 0

C:\>tlist /l 2620 | findstr /i /c:"Cmdline"
CmdLine: c:\windows\system32\inetsrv\w3wp.exe -ap "MSExchangeOWAAppPool" -v "v2.0" -l "webengine4.dll" -a \\.\pipe\iisipmf668475f-2a0c-431c-b645-c1ba9a7ca845 -h "C:\inetpub\temp\apppools\MSExchangeOWAAppPool.config" -w "" -m 0

C:\>tlist /l 3040 | findstr /i /c:"Cmdline"
CmdLine: c:\windows\system32\inetsrv\w3wp.exe -ap "DefaultAppPool" -v "v2.0" -l "webengine4.dll" -a \\.\pipe\iisipm0049ba0c-adac-43ae-9c04-b99aea267648 -h "C:\inetpub\temp\apppools\DefaultAppPool.config" -w "" -m 0

C:\>tlist /l 3424 | findstr /i /c:"Cmdline"
CmdLine: c:\windows\system32\inetsrv\w3wp.exe -ap "MSExchangeECPAppPool" -v "v2.0" -l "webengine4.dll" -a \\.\pipe\iisipma7fa8dcc-0dd6-4a90-bef3-d897a0e6a1a9 -h "C:\inetpub\temp\apppools\MSExchangeECPAppPool.config" -w "" -m 0

C:\>tlist /l 1776 | findstr /i /c:"Cmdline"
CmdLine: c:\windows\system32\inetsrv\w3wp.exe -ap "MSExchangeServicesAppPool" -v "v2.0" -l "webengine4.dll" -a \\.\pipe\iisipmf9249731-25f4-41ed-a4cb-1bfe8c4b5d06 -h "C:\inetpub\temp\apppools\MSExchangeServicesAppPool.config" -w "" -m 0

 

You can also use TLIST to search all process for a given module of DLL file. We load specific modules for different process.
Here is an example of searching for the Microsoft.Exchange.Services.dll, which is specific for the MSExchangeServicesAppPool.

 

C:\>tlist /m Microsoft.Exchange.Services.dll
C:\Windows\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files\ews\84af5395\db5aaac4\assembly\dl3\6031191a\001a3f10_b802cb01\Microsoft.Exchange.Services.DLL - 1776 w3wp.exe

 

This shows that particular DLL loaded in the process with the PID of 1776.

Now we can use our favorite debugging tool, ADPlus, old school WinDBG or CDG, or the newer ProcDump, to collect a memory dump of this process.

 

 

A newer and somewhat simpler approach is to use Process Explorer from Sysinternals (https://technet.microsoft.com/en-us/sysinternals/default).

 

Within this tool, you can simply however you mouse over the process in question to get the command line used.

 

 

You can even generate a dump directly from here as well.
Once you identify the process, secondary click and choose Create Dump from the menu.

 

 

Lastly, you can use the PowerShell Get-Process cmdlet.  Like TLIST, this will allow you to list and search modules loaded within a process. I feel this method still has the shortcoming of not being able to access or display the command line parameters. I’m still looking into this.

 

Being able to identify the correct process will not only allow us to take memory dumps of the correct process, but we leverage other tools based on the process ID, such as performance monitor counters or advanced debugging features such a process cloning.