Testing RPC over HTTP through ISA Server 2006 Part 2; Test Tools and Strategies

The Solution(s)

 

As luck would have it, we have several tools available:

· RPCPing.exe; thoughtfully supplied by the very same RPC team that gave us the RPC over HTTP protocol, you’ll find it included in the Windows 2003 Resource Kit Tools.

· WinHttpTraceCfg; this tool, which is also part of Windows 2003 Resource Kit Tools, allows you to control WinHTTP tracing to better understand the errors seen at the client.

· HTTP_Test.wsf; (ISATools.org) although it can’t “do” RPC, HTTP_Test provides a secondary method of validating the HTTP protocol handling for the traffic path under test and because it also uses WinHTTP, can be debugged using the same tracing as for RPCPing.exe

Test Tools Basics

 

1. To test Exchange RPC services, we must specify the Exchange RPC port. If you do not provide this information to RPCPing, it will use the value of 593.

2. RPCPing does not allow the user to specify non-standard ports for HTTP or HTTPS connections. HTTP_Test has no such limitation.

3. Testing the HTTP bridging scenarios require that you enable Basic (clear-text) authentication over HTTP in the related ISA Web Listener. This option exists in the Advanced properties of the web listener Authentication tab.

**Remember to revert this change on production ISA Servers before placing them back in public service**

4. If you wish to examine the traffic in unencrypted form between ISA and the RPC Proxy server, you must:

a. Configure ISA to send the traffic on HTTP only. If you configure the web listener to accept both HTTP and SSL and the rule is configured to redirect using both HTTP and SSL client connects using SSL, ISA will redirect using SSL.

b. Apply the RPC Proxy SSL Offload settings from this TechNet article. Exchange 2007 provides this as part of the Client Access options for the server.

c. Manually disable “Require Secure Channel” in the /RPC Virtual Directory (see IIS help for instructions).

d. Include the RPCPing -q option to suppress the “insecure auth method” warning produced for the combination of -F 2 (no SSL) and -H 1 (Basic auth) options. You can eliminate this if you want to respond to these prompts.

5. If your test client is operating from behind another proxy, you may have to add the following option:

RPCPing: -R ProxyFqdnOrIp:Port (RpcPing will attempt to authenticate using the interactive login)

HTTP_Test: /proxy:ProxyFqdnOrIp:Port /prxuser:Domain\Account /prxpass:Password

6. If you have configured the ISA listener for NTLM authentication, you need to replace H 1 with H 2 in each RPCPing example. Note that using NTLM authentication for ISA means that only Kerberos Constrained Delegation is possible

7. WinHTTPTraceCfg need only be executed twice during the testing; once when you start testing and once more when you end testing. While it may be interesting to see how many times WinHTTP is used when you’re not looking, this can fill up the drive with unnecessary files if you leave it enabled when not testing.

WinHttpTraceCfg.exe command-line examples

When WinHTTP tracing is enabled, each execution of RPCPing or HTTP_Test will produce a new WinHTTP log file in accordance with the settings you define using WinHttpTraceCfg.

Configure tracing

winhttptracecfg -l c:\ -d 0 -s 1 -t 1 -m 131072

Note: the -l option specifies the log file prefix. In this example, it logs to C:\. You should choose a logging location to suit your test client.

Successful response

Updated trace settings

Current WinHTTP trace settings under

  HKEY_LOCAL_MACHINE\

    SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\WinHttp\Tracing:

    TracingEnabled: 0

    ToFileOrDebugger: 0

    ShowBytes: 1

    ShowApiTrace: 1

    MaxFileSize: 131072

    FileNamePrefix: c:\

Enable tracing

winhttptracecfg -e 1

Successful response

Updated trace settings

Current WinHTTP trace settings under

  HKEY_LOCAL_MACHINE\

    SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\WinHttp\Tracing:

    TracingEnabled: 1

    ToFileOrDebugger: 0

    ShowBytes: 1

    ShowApiTrace: 1

    MaxFileSize: 131072

    FileNamePrefix: c:\

Disable tracing

winhttptracecfg –e 0

Successful response

Updated trace settings

Current WinHTTP trace settings under

  HKEY_LOCAL_MACHINE\

    SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\WinHttp\Tracing:

    TracingEnabled: 0

    ToFileOrDebugger: 0

    ShowBytes: 1

    ShowApiTrace: 1

    MaxFileSize: 131072

    FileNamePrefix: c:\

 

RPCPing and HTTP_Test command options for each bridging scenario using Basic authentication

HTTP Bridging

Client ( HTTP ) à ISA àRPC Proxy :

RPCPing -t ncacn_http -o RpcProxy=RpcProxyFqdn -P "Account,Domain,Password" -H 1 -u 10 -a connect -F 2 -q -E -v 3

Successful: Response from server received: 200

cscript HTTP_Test.wsf /url:http:// RpcProxyFqdn/rpc/rpcproxy.dll /svruser:Domain\Account /svrpass:Password /action:RPC_IN_DATA

Successful: -- Request for http://RpcProxyFqdn/rpc/rpcproxy.dll produced HTTP Response 200 'OK'

 

Client ( HTTP ) à ISA àRPC ProxyàExchange :

RPCPing -t ncacn_http -s win2k3-5 -e 6001 -I "Account,Domain,Password" -o RpcProxy=RpcProxyFqdn -P "Account,Domain,Password" -H 1 -u 10 -a connect -F 2 -q -v 3

Successful: Completed 1 calls in 341 ms (the value of ‘ms’ may differ)

cscript HTTP_Test.wsf /url:http:// RpcProxyFqdn/rpc/rpcproxy.dll ?RpcServerFqdn:6001 /svruser:Domain\Account /svrpass:Password /action:RPC_IN_DATA

Successful: -- Request for http://RpcProxyFqdn/rpc/rpcproxy.dll?RpcServerFqdn:6001 produced HTTP Response 200 'OK'

HTTPS Bridging

Client ( HTTPS ) à ISA àRPC Proxy :

RPCPing -t ncacn_http -o RpcProxy=RpcProxyFqdn -P "Account,Domain,Password" -H 1 -u 10 -a connect -F 3 -B msstd:RpcProxyFqdn -E -v 3

Successful: Response from server received: 200

cscript HTTP_Test.wsf /url:https:// RpcProxyFqdn/rpc/rpcproxy.dll /svruser:Domain\Account /svrpass:Password /action:RPC_IN_DATA

 

Successful: -- Request for https://RpcProxyFqdn/rpc/rpcproxy.dll produced HTTP Response 200 'OK'

 

Client ( HTTPS ) à ISA àRPC ProxyàExchange :

RPCPing -t ncacn_http -s win2k3-5 -e 6001 -I "Account,Domain,Password" -o RpcProxy=RpcProxyFqdn -P "Account,Domain,Password" -H 1 -u 10 -a connect -F 3 -B msstd:RpcProxyFqdn -v 3

Successful: Completed 1 calls in 341 ms (the value of ‘ms’ may differ)

cscript HTTP_Test.wsf /url:https:// RpcProxyFqdn/rpc/rpcproxy.dll ?RpcServerFqdn:6001 /svruser:Domain\Account /svrpass:Password /action:RPC_IN_DATA

Successful: -- Request for https://RpcProxyFqdn/rpc/rpcproxy.dll?RpcServerFqdn:6001 produced HTTP Response 200 'OK'

 

RPCPing command options for testing the RPC Server directly

In order to test the Exchange RPC service directly, you have to allow RPCPing to use the RPC Endpoint Mapper and query the chosen interface for its state. To do this, you have to specify the UUID of the desired interface using the “-f” option. In the case of Exchange RPC, we have these defined in Microsoft Knowledge Base article 831051.

Store Interface (TCP:6001)

RPCPing -t ncacn_ip_tcp -s RpcServerFqdn -f a4f1db00-ca47-1067-b31f-00dd010662da,0 -u 10 -a connect -v 3 -I "Account,Domain,Password!"

 

Successful: Completed 1 calls in 341 ms (the value of ‘ms’ may differ)

Directory Interface (TCP:6004)

RPCPing -t ncacn_ip_tcp -s RpcServerFqdn -f f5cc5a18-4264-101a-8c59-08002b2f8426,56 -u 10 -a connect -v 3 -I "Account,Domain,Password!"

 

Successful: Completed 1 calls in 341 ms (the value of ‘ms’ may differ)

 

 

Testing the whole thing

 

To test the whole of Exchange RPC over HTTP connectivity via the RPCPing utility, you could loop the command as shown:

for %i in (6001 6002 6004) do RPCPing -t ncacn_http -s win2k3-5 -e %i -P "Account,Domain,Password" -o RpcProxy=RpcProxyFqdn -P "Account,Domain,Password" -H 1 -u 10 -a connect -F 3 -v 3 -B msstd:RpcProxyFqdn

 

To test the whole of Exchange RPC over HTTP connectivity via the HTTP_Test script, you would loop the command as shown:

for %i in (6001 6002 6004) do cscript HTTP_Test.wsf /url:https://RpcProxyFqdn/Rpc/rpcproxy.dll?RpcServerFqdn:%i /action:RPC_IN_DATA /svruser:Domain\Account /svrpass:Password

 

To test the whole of Exchange RPC over TCP connectivity via the RPCPing utility, you could loop the command as shown:

for %i in ("a4f1db00-ca47-1067-b31f-00dd010662da,0" "f5cc5a18-4264-101a-8c59-08002b2f8426,56") do RPCPing -t ncacn_ip_tcp -s RpcServerFqdn -f %i -u 9 -a connect -v 3 -I "Account,Domain,Password!"

Problem Recognition

 

Ok; so now we know how to acquire and use some neato-keen tools and while things are working properly, all is good. It’s when things turn sour that we actually need to understand some common failures states and the techniques used to determine the cause and corrective actions. This is also when we want both tools in our toolkit. Fortunately, the tools’ names spell out their strengths and limitations. RPCPing.exe is an excellent tool for validating RPC connectivity, but it falls a bit short of the mark for HTTP protocol validation. Likewise, HTTP_Test.wsf is a great tool for testing the HTTP side of things, but has no RPC intelligence at all. Between the two of them, we can isolate most common problems and in both cases, WinHttpTraceCfg is our best friend.

 

As always, “use the logs!” ISA Live Monitoring combined with observation of the various error codes provided by RPCPing, WinHTTP and HTTP_Test as well as the IIS and WinHTTP tracing logs will most often lead you directly to the error point. We’ll address the most common errors in the following sections.

 

I’m a firm believer in the concept of “user-inserted problems” and no one is more dangerous than the “IT pro”. IT pros have the unique ability to inflict the greatest damage with the least effort of anyone in the office. Thus, this needs to be the *first place* we look when determining the problem source.

 

Client Application Oddities

Outlook can sometimes send you off in the wrong direction when it appears to connect via RPC over HTTP internally, but not externally. What’s interesting in this case is that even though you have configured Outlook as show below,

 

 

Fig. 9

 

Outlook may in fact not be using RPC over HTTP internally. This is because (as indicated in the option text) when Outlook cannot connect using RPC over HTTP, it will fall back on RPC over TCP (MAPI). Internally, this will make it seem as if RPC over HTTP is functional, when in fact it is not. You can verify the protocol used by Outlook this way:

1. Hold down the <CTRL> key and right-click the system tray Outlook icon

2. Click Connection Status

..and you’ll see a dialog similar to this:

 

Fig. 10

 

The “Conn” column will hold one of three values (“HTTP”, “HTTPS” or “TCP/IP”) for each server connection. If the “Conn” column shows “TCP/IP”, then Outlook is not using RPC over HTTP. You can disable this fallback mechanism by setting the following registry value (you will have to create it if it doesn’t exist):

 

Path

Outlook 2003

      HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\11.0\Outlook\RPC

Outlook 2007

      HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\12.0\Outlook\RPC

Name DisableRpcTcpFallback

Type DWORD

Value 1

Test Scenarios

Any time you have to test a protocol path, it’s frequently useful to break it up into manageable pieces. RPCPing allows us to do exactly this by using a specific command-line syntax. Whether you start your testing from the External Client or RPC Client is completely up to you, but you should make this decision based on the errors that started the testing in the first place. IOW, a connection failure to the ISA Server itself is not likely to be resolved by testing RPC over TCP.

External RPC over HTTP

 

Fig. 11

 

This is the most common starting point for testing RPC over HTTP when ISA Server is involved, since it has the ability to validate the entire traffic and authentication process from a single location (when things are working).

Internal RPC over HTTP

 

Fig 12

 

Connecting directly to the RPC Proxy can prove useful when you need to (dis)prove ISA as a failure point in the traffic path. If the test through ISA fails, but testing directly to the RPC Proxy succeeds, then you probably have to examine your ISA configuration for errors. Because the RpcProxyFqdn can (and frequently will) be different internally and externally, you should verify your command-line syntax before blaming ISA or the RPC Proxy. In both of the above scenarios, only the HTTP-based RPCPing syntax is used.

The command-line options that will normally change between internal and external RPC Proxy testing are:

- RPCPing:

o -F (according to the protocol defined between ISA and the RPC Proxy)

o -q (not required for HTTPS)

o -B (if the RPC Proxy FQDN differs from the ISA external FQDN)

- HTTP_Test:

o /url: (according to the protocol defined between ISA and the RPC Proxy)

- Both:

o RpcProxyFqdn (if the RPC Proxy FQDN differs from the ISA external FQDN)

RPC over TCP

Fig 13

 

In this scenario, only RPC over TCP is appropriate, since the RPC services themselves cannot accept HTTP traffic. In this case, you’ll have to use the RPCPing RPC over TCP command-line examples, even if the RPC Proxy and the RPC Server represent the same machine.


 

We'll continue this discussion in part 3 where I'll provide some common failure points and the most common resolution steps for each.

 

Happy testing,

 

Jim Harrison

Program Manager, ISA SE