We recently had a couple of SharePoint support cases where customers reported hangs. When we analyzed user dumps of the affected processes we noticed that the application pool was in the process of recycling due to an exception that has caused the application pool to fail.
Unfortunatelly the recycling never finished - instead a message box was generated by .NET framework asking if the user would like to continue (which would allow recycle to perform) or to attach a debugger.
The problem here is that the message box was not displayed on the Console but invisible as the w3wp.exe process does not interact with desktop. Even if this option would be enabled it would be an undesireable result as usually no user sits in front of the SharePoint server waiting for such a message box to appear.
After collaborating with our colleagues from the .NET framework support team we identified that the root cause for the issue was a registry key which was set when installing Visual Studio on the machine. Even if Visual Studio is uninstalled later this registry key is not removed. This registry key allows developers to attach the debugger to a failing process right before it terminates by bringing up this message box.
The details of this registry key is explained here:
We also found that Microsoft has already an official guidance on how to clean up this after uninstalling Visual Studio:
After Visual Studio is installed on a server, the default behavior when an unhandled exception occurs is to show an Exception dialog that requires user intervention to either start Just-In-Time debugging or ignore the exception. This may be undesirable for unattended operation. To configure the server to no longer show a dialog when an unhandled exception occurs (the default behavior prior to installing Visual Studio), use the registry editor to delete the following registry keys:
- HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug\Debugger
On a 64-bit operating system also delete the following registry keys:
- HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug\Debugger
(see here for more details)
In case you ever had Visual Studio installed on one of your production servers, please be sure to check these registry keys and remove them to prevent this problem.
[Update from May 27th, 2009] I just got informed that the same registry key is created during the installation of the .NET Framework 2.0 SDK. So the same steps apply if this SDK has been installed on the server in the past.