Even More Exchange 2010 Management Tools Fun

Edited 1/17/2011 to add second error message listed below.
Edited 3/7/2011 with additional netsh winhttp set proxy information

This time we were getting the following errors on 3 new Exchange 2010 servers (2 MBX, 1 Hub/CAS):

When opening the Exchange Management Shell (EMS):

Connecting to remote server failed with the following error message : Access is denied.

OR:

Connecting to remote server failed with the following error message: The connection to the specified remote host was refused. Verify that the WS-Management service is running on the remote host and configured to listen for requests on the correct port and HTTP URL.

Opening the Exchange Management Console (EMC) returned:

The following error occurred when searching for On-Premises Exchange Server:
Connecting to remote server failed with the following error message: Access is Denied. For more information, see the about_Remote_Troubleshooting Help topic. It was running the command "discover-exchangeserver -usewia $true -suppresserror $true".

The strange thing is that we had 3 older 2010 servers that we could open the tools fine on. And, we could use those servers to connect to and manage the servers we were unable to connect from.

After doing a bunch of troubleshooting of WinRM to make sure it was all well, and finding nothing wrong, we finally ran a network capture with Network Monitor 3.4 while opening the EMS. One very nice feature of Netmon 3.4 is the ability to view the network traffic per application. This allowed us to select Powershell.exe and see all traffic from/to it. Here is what we saw:

The odd thing here is that when EMS was opening and trying to connect, it was indicating that it was trying to connect to all 6 servers in the organization and getting back the Access Denied, so why do we only see 2 addresses here? Looking at the addresses, we determined that the first was a DC, which would be expected, since we go to a DC first to determine our Exchange server names so we can then attempt to connect to them through Remote Powershell. The 2nd address however, turned out to be a PROXY SERVER!

In hindsight, this all makes sense, as Remote Powershell/WinRM ends up going through HTTP (or HTTPS) to IIS on whatever server, including the local one, that we are trying to connect to and run our Exchange management on. Going through the proxy server breaks the passing of our credentials using Kerberos. In the netmon capture, I could see this for each server we tried to access:

HTTP:Request, POST https://servername.domain.com/powershell, Query:serializationLevel=Full;ExchClientVer=14.0.639.21;PSVersion=2.0, Using Kerberos Authorization

HTTP:Response, HTTP/1.1, Status: Ok, URL: https://servername.domain.com/powershell, Using Kerberos YIGYBgkqhkiG9xIBAgICAG+BiDCBhaADAgEFoQMCAQ+ieTB3oAMCARKicARu/6C25Su4nr7mR4uEyaESeLJ1XVll6ptl9NPo3zocEHXezAJ/NT5ofY3N2kp8yD/FSJmqFKt7GGLMQV4IA2EUvL0tQux3wWRxr91I

HTTP:Request, POST https://servername.domain.com/powershell, Query:serializationLevel=Full;ExchClientVer=14.0.639.21;PSVersion=2.0
HTTP:HTTP Payload, URL: https://servername.domain.com/powershell

HTTP:Response, HTTP/1.1, Status: Unauthorized, URL: https://servername.domain.com/powershell, Using Kerberos Authentication

To verify the proxy setting, we used the following netsh winhttp command from a command prompt:

netsh winhttp show proxy

We then removed the proxy setting with:

netsh winhttp reset proxy

After we removed the proxy, our tools opened right up and connected fine.

So, if you’re getting Access Denied errors in the connect attempts when opening the Exchange Management Tools, look for a proxy server getting in the way…

However, what do you do if you need to have a proxy setting for external bound traffic from the server?

It turns out that there is a bypass-list setting that can be used with netsh winhttp set proxy. The full reference of the netsh winhttp commands are found here: https://technet.microsoft.com/en-us/library/cc731131(WS.10).aspx#BKMK_5

Looking at the set proxy commands, we can see there is an example of using the bypass-list setting (bypassing proxy for any contoso.com connections):
set proxy proxy-server="http=myproxy;https=sproxy:88" bypass-list="*.contoso.com"

So, if you need to use a proxy in your winhttp settings, use the bypass-list to bypass the internal domain name being used by your Exchange servers.