Crashes (svchost and Internet Explorer) while accessing Internet through ISA Server

Introduction

This post is about two scenarios which involved ISA server and third party content application. Let’s go over each one of those and the troubleshooting steps involved in order to fix it.

Scenario 1

ISA was doing a Proxy Chain with a third party product, when the client (using Windows XP SP2 with Internet Explorer 6) was browsing a HTTPS web site behind the ISA Server the Internet Explorer crashed with an access violation. If the clients bypassed the ISA Server and use only the upstream third party proxy the issue did not happen.

Scenario 2

Clients using Windows XP SP2 were unable to update via Windows update. When we took one of the laptops and bypassed ISA which was the proxy in the corporate LAN, they were able to update without any problem. Moreover, the automatic update service (which runs in the context of SVCHOST) was crashing when we tried from LAN. ISA was forwarding web traffic to a third party product.

Troubleshooting

The data that was collected to troubleshoot these included

1) Network Capture on ISA on all adapters concerned.
2) ADPLUS crash dump for the respective processes on the client.
3) Application logs from the client machine s.

Data Analysis

For the first scenario the ISA Logging did not show up any suspicious error and the network traces pretty much shows the normal traffic:

1. The client requesting the SSL object from a Web server.
2. The “CONNECT Server_name:443 HTTP/1.1” being sent to port 8080 on the ISA Server computer.
3. The ISA Server connecting to the destination Web server on port 443.
4. The TCP connection established and the ISA Server returning the HTTP/1.1 200 connection established.

Fortunately Internet Explorer was raising an exception (c0000005 – Access Violation) and it was possible to get the dump during the crash. The exception was happening on the module wininet.dll, handling the function HTTP_REQUEST_HANDLE_OBJECT OpenProxyTunnel_Fsm on the client side

For the second scenario the error in the Application log of the client machine when automatic update failed was

Event Type: Error
Event Source: Application Error
Event Category: (100)
Event ID: 1000
Date: 7/12/2010
Time: 2:18:22 PM
User: N/A
Computer: XXXX
Description:
Faulting application svchost.exe, version 5.2.3790.3959, faulting module winhttp.dll, version 5.2.3790.4427, fault address 0x000000000003e521.

Upon analysis of the crash dump from client machine we found:

The stored exception information can be accessed via .ecxr.
(97c.ba8): Access violation - code c0000005 (first/second chance not available)

We crashed in:

4d51b7d3 c7402400000100 mov dword ptr [eax+24h],10000h ds:0023:00000024=????????

The module and function in concern was

015ef4d8 4d51b9bc winhttp!HTTP_REQUEST_HANDLE_OBJECT::OpenProxyTunnel_Fsm+0x44b

Loaded symbol image file: winhttp.dll

    Image path: C:\WINDOWS\system32\winhttp.dll
Image name: winhttp.dll
Timestamp: Wed Aug 04 13:27:07 2004 (411096D3)
    CheckSum: 00055D26
ImageSize: 00058000
File version: 5.1.2600.2180
Product version: 5.1.2600.2180
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® Windows® Operating System
InternalName: winhttp.dll
OriginalFilename: winhttp.dll
ProductVersion: 5.1.2600.2180
FileVersion: 5.1.2600.2180 (xpsp_sp2_rtm.040803-2158)
FileDescription: Windows HTTP Services
LegalCopyright: © Microsoft Corporation. All rights reserved.

The reason it was crashing was because the TCP connection was getting closed. Reviewing the network capture we could see the following:

Client to ISA

CONNECT www.update.microsoft.com:443 HTTP/1.0
User-Agent: Windows-Update-Agent
Host: www.update.microsoft.com
Content-Length: 0
Proxy-Connection: Keep-Alive

ISA to Client

HTTP/1.1 200 Connection Established
Content-Length: 0
Proxy-Agent: ThirdPartyProduct/3.0
Via: 1.1 ISA
Connection: Keep-Alive
Proxy-Connection: Keep-Alive

The client was resetting the packet as soon as it got this packet.

Conclusion

Both the scenarios revolved around the same problem:

• First Scenario - During this scenario ISA Server was not the root cause for the problem; the wininet.dll had a bug that caused the access violation handling SSL tunneling. A hotfix was shipped to fix this issue and can be downloaded on the article 923535 - Internet Explorer 6 crashes when you try to access an HTTPS URL from a computer that is running Windows XP Service Pack 2. Such behaviour is described in this post by Yuri Diogenes.
• Second Scenario - We checked and we found that this issue have been taken care in windows 2008 and windows 7. Never the less bypassing the third party proxy fixed the issue.

Proxy Connect Method is used by the client to tunnel the TCP based traffic from the proxy server. After the connection is established the proxy server simply tunnels the traffic between the client and the internet web server. To tunnel SSL traffic though proxy, browsers or any other application can make a CONNECT request to the proxy server.

More info see http://tools.ietf.org/html/draft-luotonen-web-proxy-tunneling-01#section-3.2.1

Understanding the CONNECT Method

The SSL session is established between the client generating the request, and the destination (secure) Web server; the proxy server in between is merely tunneling the encrypted data, and does not take any other part in the secure transaction.

Request

The client connects to the proxy, and uses the CONNECT method to specify the hostname and the port number to connect to. The hostname and port number are separated by a colon, and both of them must be specified.

Example:

CONNECT mail.microsoft.com:443 HTTP/1.0
User-Agent Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; MS-AOC 4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; MS-RTC EA 2; MS-RTC LM 8; .NET4.0C; .NET4.0E)

Proxy Response

The client will wait for a response from the proxy. The proxy will evaluate the request, make sure that it is valid, and that the user is authorized to request such a connection. If everything is in order, the proxy will make a connection to the destination server, and, if successful, send a "200 Connection established" response to the client

An example of a tunneling request/response in an interleaved:

Although there is no specification details about Content length header in Proxy Connect Request and Response headers, never the less having a content length in the CONNECT header is not of much use. Connect is used only make an initial connection to the proxy server for requesting to tunnel traffic

Author
Saurav Datta
Security Support Escalation Engineer
Microsoft CSS Forefront Security Edge Team

Technical Reviewer
Yuri Diogenes
Sr Security Support Escalation Engineer
Microsoft CSS Forefront Security Edge Team