Testing RPC over HTTP through ISA Server 2006 Part 3; Common Failures and Resolutions
Common Failure States
In the sections below, you’ll find the most common failure states listed in “priority” order, including the relevant data provided by each tool or traffic processing point and the most appropriate resolutions to each. While it may not be immediately apparent from the list below, using RPCPing with the “-E” option immediately following a failure from “-s RpcServerFqdn” usage can help you quickly differentiate between HTTP- and RPC-based failures.
If you cannot satisfy the first category or state, you cannot even attempt the remaining ones. In other words, if the test client cannot resolve the name (or FQDN as appropriate) of the RPC Proxy, then there is no value in attempting to evaluate authentication problems. Likewise, if you are addressing authentication failures, then validating the connection state is probably a waste of time. I’ve intentionally limited the problem resolution steps to the most common corrective actions. Otherwise, this article would never see the light of day…
Client Connection Failures
This failure category specifically addresses those cases where the connection between the test client and the RPC Proxy fails. Note that if you’re testing ISA web publishing, the IIS logs may not contain an entry for your particular test unless ISA actually passed the traffic to the “real” RPC Proxy server. If you’re testing the RPC Proxy directly, then ISA logging and alerting will not be relevant.
1. Name Resolution for “RpcProxyFqdn” Failed
RPCPing:
Error 12007 returned in the WinHttpSendRequest.
HTTP_Test:
** Error 0x80072EE7 encountered during request for http://IsaServerFqdn; The server name or address could not be resolved
WinHTTPTrace:
WinHttpSendRequest: error 12007 [ERROR_WINHTTP_NAME_NOT_RESOLVED]
ISA Logs:
Not relevant, since the client never connected to it.
ISA Alerts:
Not relevant, since the client never connected to it.
IIS Logs:
Not relevant, since the client never connected to it.
Problem Resolution:
Make sure that:
1. “RpcProxyFqdn” in the command-line is correct
2. DNS or WINS server (or hosts file if necessary) used by the client has the correct records
2. Connection to “RpcProxyFqdn” Failed
RPCPing:
Error 12005 returned in the WinHttpConnect.
HTTP_Test:
** Error 0x80072EFD encountered during request for https://RpcProxyFqdn/rpc/rpcproxy.dll?RpcServerFqdn:6001; A connection with the server could not be established
WinHTTPTrace:
WinHttpSendRequest: error 12029 [ERROR_WINHTTP_CANNOT_CONNECT]
ISA Logs:
Not relevant, since the client never connected to it.
ISA Alerts:
Not relevant, since the client never connected to it.
IIS Logs:
Not relevant, since the client never connected to it.
Problem Resolution:
1. Ensure that the test client resolves “RpcProxyFqdn” to the correct RPC Proxy IP address
2. Verify that the RPC Proxy server listener is properly defined and operating:
Netstat –an at the command line on the RPC proxy should provide output similar to this:
Active Connections
Proto Local Address Foreign Address State
TCP 0.0.0.0:80 0.0.0.0:0 LISTENING
TCP 0.0.0.0:443 0.0.0.0:0 LISTENING
If the Local Address column indicates 0.0.0.0, the listener is defined for all available addresses on the server. In this case, you should verify that no other traffic filtering mechanism is blocking this traffic between the test host and the RPC Proxy.
If the Netstat output indicates specific IP addresses, make sure they agree with the IP address resolved by the test client.
If the Netstat output indicates no listeners, you will have to resolve this before continuing.
3. SSL Session Setup with “RpcProxyFqdn” Failed
If you intended to test HTTP bridging or direct HTTP connection to the RPC Proxy and you received any error indicated below, then you should review your test tool command-line options again.
RPCPing:
1. The server certificate subject from the RPC proxy (msstd:RpcProxyFqdn, fullsic:\<CN=IssuingAuthority>\<CN= RpcProxyFqdn >) does not match the one (msstd: RpcProxyFqdn) specified
2. Error 12175 returned in the WinHttpSendRequest.
HTTP_Test:
1. ** Error 0x80072F06 encountered during request for https://RpcProxyFqdn; The host name in the certificate is invalid or does not match
2. ** Error 0x80072F0D encountered during request for https:// RpcProxyFqdn; The certificate authority is invalid or incorrect
WinHTTPTrace:
1. Winsock/RPC/SSL/Transport error: 0x800b010f
2. Winsock/RPC/SSL/Transport error: 0x800b0109
ISA Logs (both cases) :
Result code =0x80074e20 FWX_E_GRACEFUL_SHUTDOWN
ISA Alerts:
None
IIS Logs:
Sc-status = 200
Problem Resolution:
1. Verify that the subject name of the certificate associated with the RPCProxy listener includes the same name as RpcProxyFqdn.
2. Verify that the test host includes the certificate for the issuing authority in the Local Computer Trusted Root Certification Authorities
Client Authentication Failures
This category addresses those cases where the credentials provided by the test tool are not acceptable to the RPC Proxy (HTTP) or the RPC server. If authentication to the ISA server fails, it’s often useful to test directly against the RPC Proxy because the IIS logs are much more informative with regard to authentication failures.
1. HTTP Authentication
RPCPing:
Response from server received: 401
HTTP_Test:
Failed to authenticate after 5 retries.
WinHTTPTrace:
HTTP/1.1 401 Unauthorized
ISA Logs (all cases) :
HTTP Status code = 12239 The server requires authorization to fulfill the request. Access to the Web server is denied. Contact the server administrator.
ISA Alerts:
None
IIS Logs:
1. sc-win32-status = 1326
2. sc-win32-status = 1331
3. sc-win32-status = 1907
4. sc-win32-status = 2148074302
Problem Resolution:
1. Ensure that
a. the credentials specified in the -P or /svruser & /svrpass options are correct
b. the Account is sensitive and cannot be delegated account options selection is disabled
2. Enable the test account
3. Disable the “User must change password at next logon” account options selection and reset the test account password
4. Disable the “Smartcard is required for interactive login” account options selection
2. RPC Authentication
RPCPing:
1. Exception 5 (0x00000005)
2. Exception 1331 (0x00000533)
3. Exception 1907 (0x00000773)
HTTP_Test:
Not relevant, since HTTP_Test cannot evaluate RPC
WinHTTPTrace:
HTTP/1.0 503 RPC Error: c0021012
ISA Logs:
Result Code = 0x80074e21 FWX_E_ABORTIVE_SHUTDOWN
ISA Alerts:
None
RPC Proxy (IIS) Logs:
Sc-status = 200
Problem Resolution:
1. Ensure that
a. the credentials specified in the -I option are correct
b. Account is sensitive and cannot be delegated account options selection is disabled
c. Smartcard is required for interactive login account options selection is disabled
2. Enable the test account
3. Disable the “User must change password at next logon” account options selection and reset the test account password
ISA Server to RPC Proxy Failures
In this context, RpcProxyFqdn refers to the server name or FQDN entered in the To tab of WebPubRuleName. This may be the same as used in the test tool, but this is not required and is determined by the individual scenario.
1. Name Resolution for “RpcProxyFqdn” Failed
RPCPing:
Response from server received: 500
HTTP_Test:
-- Request for https://RpcProxyFqdn/rpc/rpcproxy.dll produced HTTP Response '( The host was not found. )'
WinHTTPTrace:
HTTP/1.1 500 ( The host was not found. )
ISA Logs:
1. HTTP Status Code = 11001
2. HTTP Status Code = 11002
3. HTTP Status Code = 11004
ISA Alerts:
Published Web Server Name Not Resolvable
Description: ISA Server was unable to resolve the DNS name RpcProxyFqdn. Requests that use the Web publishing rule WebPubRuleName may be denied, or the response time may be slower than expected.
RPC Proxy (IIS) Logs:
Not relevant, since ISA never connected to it.
Problem Resolution:
1. Make sure that:
a. “RpcProxyFqdn” in the relevant web publishing rule To tab is correct
b. DNS or WINS server (or hosts file if necessary) used by the ISA has the correct records
2. This is typically an intermittent failure. If this occurs repeatedly, you need to troubleshoot your DNS or WINS server
3. This indicates a name record problem on the DNS server
2. Connection to “RpcProxyFqdn” Failed
RPCPing:
1. Error 12002 returned in the WinHttpReceiveResponse.
2. Response from server received: 500
HTTP_Test:
1. ** Error 0x80072EE2 encountered during request for http://RpcProxyFqdn/rpc/rpcproxy.dll; The operation timed out
2. -- Request for 'http:// RpcProxyFqdn/rpc/rpcproxy.dll' produced HTTP Response 500, '( Connection refused )'
WinHTTPTrace:
1. Winsock/RPC/SSL/Transport error: 0x274c [WSAETIMEDOUT]
2. HTTP/1.1 500 ( Connection refused )
ISA Logs:
1. HTTP Status Code = 10060
2. HTTP Status Code = 10061
ISA Alerts:
None
RPC Proxy (IIS) Logs:
Not relevant, since ISA never connected to it.
Problem Resolution:
(all items should be verified in either case)
1. Ensure that the
a. ISA resolves “RpcProxyFqdn” to the correct RPC Proxy IP address
b. Windows Firewall at the RPC proxy is properly configured to allow inbound connections from the ISA for the appropriate RPC Proxy NIC, transports and ports
c. correct port is specified in the web publishing rule Bridging tab
2. Verify that the RPC Proxy Server listeners are properly defined and operating:
Netstat –an at the command line on the RPC proxy should provide output similar to this:
Active Connections
Proto Local Address Foreign Address State
TCP 0.0.0.0:80 0.0.0.0:0 LISTENING
TCP 0.0.0.0:443 0.0.0.0:0 LISTENING
If the Local Address column indicates 0.0.0.0, the listener is defined for all available addresses on the server. In this case, you should verify that no other traffic filtering mechanism is blocking this traffic between the test host and the RPC Proxy.
If the Netstat output indicates specific IP addresses, make sure they agree with the IP address resolved by the ISA server.
If the Netstat output indicates no listeners, you will have to resolve this before continuing.
3. SSL Session Setup with “RpcProxyFqdn” Failed
RPCPing:
Response from server received: 500
HTTP_Test:
1. -- Request for 'https://ex.pub.isase.lab/rpc/rpcproxy.dll' produced HTTP Response 500, '( The target principal name is incorrect. )'
2. -- Request for 'https://ex.pub.isase.lab/rpc/rpcproxy.dll' produced HTTP Response 500, '( The certificate chain was issued by an authority that is not trusted. )'
WinHTTPTrace:
1. HTTP/1.1 500 ( The target principal name is incorrect. )
2. HTTP/1.1 500 ( The certificate chain was issued by an authority that is not trusted. )
ISA Logs:
1. HTTP status Code = 0x80090322
2. HTTP status Code = 0x80090325
ISA Alerts:
1. SSL connection failure with published server (name mismatch)
Description: ISA Server could not establish an SSL connection with the published server RpcProxyFqdn on port 443 because the name on the SSL server certificate used by the published server does not match the internal name of the Web server <IncorrectCert>, as specified in the publishing rule.
Verify that the internal name specified in the publishing rule is correct. If the problem persists contact the Web server administrator
2. SSL connection failure with published server (no trust)
Description: ISA Server could not establish an SSL connection with the published server RpcProxyFqdn on port 443 because it does not trust the issuer of the SSL server certificate used by the published server. Verify that the root certificate for the certification authority (CA) that issued the server certificate is installed on the ISA Server computer. If the problem persists contact the Web server administrator.
RPC Proxy (IIS) Logs:
None
Problem Resolution:
1. Verify that the subject name of the certificate includes the same name as the RpcProxyFqdn specified in the web publishing rule To tab.
2. Verify that the ISA Server includes the certificate for the issuing authority in the Local Computer Trusted Root Certification Authorities
4. Authentication Delegation Failure
Note that this state is specific to the type of authentication delegation chosen in the web publishing rule Authentication Delegation tab, and not is not related to incorrect credentials provided by the test tool user.
RPCPing:
Response from server received: 403
HTTP_Test:
-- Request for 'https://ex.pub.isase.lab/rpc/rpcproxy.dll' produced HTTP Response 403, 'Forbidden ( The server denied the specified Uniform Resource Locator (URL). Contact the server administrator. )'
WinHTTPTrace:
HTTP/1.1 403 Forbidden ( The server denied the specified Uniform Resource Locator (URL). Contact the server administrator. )
ISA Logs:
HTTP Status Code = 12202
ISA Alerts:
Credentials Delegation Failure
Description: ISA Server tried to delegate credentials, but the Web site does not accept the credentials provided by the authentication delegation scheme configured in the Web publishing rule Exch2K7 OA HTTPS-131.107.8.3. Verify that the credentials delegation scheme configured in the Web publishing rule matches an authentication protocol enabled on the published Web site.
RPC Proxy (IIS) Logs:
Sc-status = 401; sc-sub-status = 2; sc-win32-status = 5
Problem Resolution:
Ensure that the option chosen in the web publishing rule Authentication Delegation tab exactly matches the authentication method selected in the RPC Proxy /rpc virtual directory.
RPC Proxy to RPC Server Failures
It is not possible to differentiate between name resolution and connection failures with these tools. You will have to determine name resolution issues using nslookup or ping tests from the RPC Proxy itself.
Connection to “RpcServerFqdn” Failed
RPCPing:
Exception 1722 (0x000006BA)
HTTP_Test:
Not relevant, since HTTP_Test cannot evaluate RPC
WinHTTPTrace:
HTTP/1.1 503 RPC Error: 6ba
ISA Logs:
HTTP Status Code = 503 and 64 (separate, consecutive entries)
ISA Alerts:
None
RPC Proxy (IIS) Logs:
Sc-status = 200
Problem Resolution:
Make sure that:
1. “RpcServerFqdn” resolves to the correct RPC Server IP address
2. DNS or WINS server (or hosts file if necessary) used by the RPC Proxy has the correct records
3. The proper entries exist at the RPC Proxy in HKLM\Software\Microsoft\Rpc\RpcProxy\ValidPorts as defined by the RPC services
4. Windows Firewall at the RPC Server is properly configured to allow inbound connections from the RPC Proxy for the appropriate RPC Server NIC, transports and ports
5. the RPC Server listeners are properly defined and operating:
Netstat –an at the command line on the RPC proxy should provide output similar to this:
Active Connections
Proto Local Address Foreign Address State
TCP 0.0.0.0:6001 0.0.0.0:0 LISTENING
TCP 0.0.0.0:6002 0.0.0.0:0 LISTENING
TCP 0.0.0.0:6004 0.0.0.0:0 LISTENING
If the Local Address column indicates 0.0.0.0, the listener is defined for all available addresses on the server. In this case, you should verify that no other traffic filtering mechanism is blocking this traffic between the RPC Proxy and the RPC Server.
If the Netstat output indicates specific IP addresses, make sure they agree with the IP address resolved by the ISA server.
If the Netstat output indicates no listeners, you will have to resolve this before continuing.
That's all for this blog posting. If you're wondering if any of the techniques used in this article are valid for testing ISA publishing for protocols other than RPC over HTTP, the answer is an unqualified "yes". Both RPCPing and HTTP_Test can be used to validate any web publishing rule (using the proper options, of course). Feel free to experiment and see what you get.
Likewise, if you discover something I overlooked, or you discover a kewl technique, feel free to comment.
Happy testing,
Jim Harrison
Program Manager, ISA SE