Traffic incorrectly going to ISA Server 2006 instead of going directly to the server

Introduction

This post is about an interesting issue where a third party application that was configured as service was failing to establish a connection with the destination server because ISA Server was denying the traffic. For this brief description it really sounds like a trivial issue where you just need to create an access rule and resolve the problem, however the scenario a quite more complex. In order to explain it is important to known the scenario first. Diagram below shows the network topology:

 

Figure 1

 

As you can see we have four networks and the switch layer 3 is routing traffic through the networks. The problem resides on srv01.contoso.com, this application runs as service and for the application to work it needs to start using local system account. When the administrators starts the service this application tries to reach farmapp.contoso.com:portx (where portx is the port used by this application), oddly the application was sending the request through ISA Server instead of sending the request directly to the application server. In order to confirm that I used the Procmon tool and the result was similar to this one below:

Thatsthecustomapp.exe TCP Send farmapp.contoso.com:portx -> isasrv.contoso.com:8080

You might be thinking: time to check your IE settings, looks like you got a problem there. That’s what I thought too, however IE was clear, no proxy settings at all.

Moving Forward

Keep moving forward on the troubleshooting, next attempt it was to run proxycfg tool to see if there is anything in the registry and the result was:

Microsoft (R) WinHTTP Default Proxy Configuration Tool

Copyright (c) Microsoft Corporation. All rights reserved.

Current WinHTTP proxy settings under:

HKEY_LOCAL_MACHINE\

SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\

WinHttpSettings :

Direct access (no proxy server).

Apparently nothing on registry….this server was also not using ISA Server as default gateway, so there is no apparently reason why the traffic was going to ISA Server instead of going directly to the destination server. The key for the problem was: the problem only happened if the third party service started as Local System account.

A friend from the Windows Performance team helped me out and suggested to get a Dependency Walker result from the third party executable file. We got it and the result was:

Figure 2

Eric Kotz from Windows Performance engineer confirmed that this implementation was not supported and pointed out the article that explains that, where it says:

The problem with running WinInet in a service is that WinInet uses settings from the registry for SSL information, proxy information, and more. Services do not load the HKEY_CURRENT_USER registry hive, so this information is not available.

Warning: Microsoft does not support using WinInet APIs within the context of a System Service.

From: KB 238425 WinInet Not Supported for Use in Services

Fair enough, after running the command psexec -is c:\program files\internet explorer\iexplore.exe ( -i to interact with desktop and -s to run on the context of system account) it was possible to see that the proxy settings were there, after removing it the application started working just fine.