Office Communicator fails to make TLS connection to LCS

Disclosure - the content for this post comes from Jason on our escalation services team. He has provided permission to post.

March 30 - Forgive the delete and repeat posting but the images were on internal links. This should be resolved now.

Issue

Some Microsoft Office Communicator (MOC) clients are not able to establish a TLS connection to the Office Communications Server (OCS). Customer has noticed that the issue seems to occur more with one business group as compared to other business groups within the organization but has not been able to determine what is unique/different with the users or computers affected.

Data Collected

Network traces from a working and failing computer.
iDNA of Office Communicator during a failed connection attempt.

Network Trace Analysis

Network Trace filtered (tcp.port==5061)

Figure 1

• Frame 303: Client starts TCP 3-way handshake
• Frame 306: Client sends TLS "Client Hello" (for details see the TLS Technical Reference)
• Frame 311: Server responds with TLS "Server Hello"
• Frame 315: Client sends TLS "Cert, Client Key Exchange, Change Cipher Spec"
• Frame 316: Server responds with TLS "Change Cipher Spec, Encrypted Handshake"
• Frame 959: Server RESETS the connection due to a timeout (30 seconds)

What happens between frame 316 and frame 959? When a client is establishing a TLS session the client gets a copy of the server's certificate, validates it is issued from a trusted authority, and then the client verifies the certificate has not been revoked by downloading the Certificate Revocation List (CRL). If we looked at the certificate configured on the server, we could see the CRL was published at https://crl.verisign.com/pca3.crl.

Modifying the filter to include http traffic: tcp.port==5061 || http

Figure 2

Now we can see what was going on in the time between the last data sent by the server and the RESET sent by the server.

• Frame 407: Client sends http GET for https://crl.verisign.com/pca3.crl // attempt to download CRL
• Frame 408: Customer's proxy replies back with http 407 Proxy Auth required (advertising support for NTLM or basic auth)
• Frame 416: Client sends new GET request for CRL, this time including NTLM data.
• Frame 417: Proxy replies with expected 407 with the NTLM Challenge

The next thing we should see is the client sending a GET with a NTLM Challenge Response. The client never sends this packet. We need to determine why the client did not reply. We don't know if the client application did not receive data captured in frame 417. We know the data came across the NIC but from that we cannot prove the application received it. So I have two theories: 1. The client never saw the data in frame 417 and that is why he never answers or 2. The client received the data in frame 417 and for some reason did not like it.

iDNA Analysis

Generally speaking there are two sets of APIs used to communicate over the HTTP protocol: WinInet and WinHttp. In this case we don't know which one Office Communicator is using. One way you can guess which one a particular application is using is to list the modules loaded in the process. You can do this on a running system using "tasklist /m /fi "imagename eq communicator.exe" or using "lm" in a .run or .dump file. Since I have the iDNA (.run file) I listed the loaded modules (lm) and found the wininet.dll was loaded and winhttp was not. With this we can set a breakpoint on the wininet function that would go get content based on a URL. Quick search of the Platform SDK shows that InternetOpenURL does this. Setting a bp on this function and starting the run file (g).

First hit on breakpoint:

0:005> knL

# ChildEBP RetAddr

00 05c2f51c 0818420f wininet!InternetOpenUrlA

WARNING: Stack unwind information not available. Following frames may be wrong.

01 05c2f55c 0818266f eecrlrev+0x420f

02 05c2f6a0 0818231d eecrlrev+0x266f

03 05c2f6b8 08181a81 eecrlrev+0x231d

04 05c2f6d8 0818131e eecrlrev+0x1a81

05 05c2f738 081846ac eecrlrev+0x131e

06 05c2f840 77a96075 eecrlrev+0x46ac

07 05c2f940 77a95f31 crypt32!VerifyDefaultRevocation+0x1d0

08 05c2f9b0 77a96b04 crypt32!CertVerifyRevocation+0xb7 [frames removed]

0:005> x lpszUrl

05c2f528 lpszUrl = 0x05c2f58c https://crl.verisign.com/pca3.crl

Based on the call stack, we can see the call to CertVerifyRevocation led to the call to InernetOpenUrl. Dumping out the URL passed to the function, we validate it was the URL for the CRL.

From here I wanted to see us put the request on the wire. Set a breakpoint on ws2_32!send in order to see the data as it left the application.

When we hit the bp we can see the following in our outbound buffer.

0:005> x buf

05c2f100 buf = 0x00155420 "GET https://crl.verisign.com/pca3.crl HTTP/1.0..User-Agent: Entrust Certificate Revocation..Host: crl.verisign.com..Pragma: no-cache..Cookie: BCSI-CS0AEE9E8D=2..Proxy-Authorization: NTLM TlRMTVNTUAABAAAAB7IIogUABQAwAAAACAAIACgAAAAFASgKAAAAD1Q2Q0s5NDNZTUFQTEU=...."

We can see this is the buffer sent in the network trace, frame 416 (Figure 2)

Setting bp on the winsock receive fuction stepping to the point that the buffer is filled out.

Buffer address was @ 00170aa8

0:005> da 00170aa8

00170aa8 "HTTP/1.1 407 Proxy Authenticatio"

00170ac8 "n Required..Proxy-Authenticate: "

00170ae8 "NTLM TlRMTVNTUAACAAAABAAEADgAAAA"

00170b08 "Fgomi0cszMWNOhjgAAAAAAAAAAHoAegA"

00170b28 "8AAAABQLODgAAAA9GAEcAAgAEAEYARwA"

00170b48 "BABAAUwAzAFcAMAA0ADEANQA2AAQAFAB"

00170b68 "mAGcALgByAGIAYwAuAGMAbwBtAAMAJgB"

00170b88 "TADMAVwAwADQAMQA1ADYALgBmAGcALgB"

00170ba8 "yAGIAYwAuAGMAbwBtAAUAFABmAGcALgB"

00170bc8 "yAGIAYwAuAGMAbwBtAAAAAAA=..Cache"

00170be8 "-Control: no-cache..Pragma: no-c"

00170c08 "ache..Content-Type: text/html; c"

This aligns to frame 417 in figure 2

So at this point we know the client application (MOC) received the 407 proxy response with the NTLM challenge. From here we should expect the client to turn around and send the NTLM challenge response. We still have a bp on send and I added a bp for closesocket to catch anything going wrong if we don't send the response.

My CloseSocket bp hit.

0:005> k

ChildEBP RetAddr

05c2edc0 771bfb5a ws2_32!closesocket

05c2edf8 771ce87f wininet!ICSocket::Close+0x25

05c2ee08 771ce8d2 wininet!ICSocket::Disconnect+0x23

05c2ee24 771c5061 wininet!HTTP_REQUEST_HANDLE_OBJECT::ReleaseConnection+0x6d

05c2ee48 771c8c6c wininet!HTTP_REQUEST_HANDLE_OBJECT::CloseConnection+0x84

05c2f194 771c9d37 wininet!HTTP_REQUEST_HANDLE_OBJECT::ReadData_Fsm+0x5e8

05c2f1a0 771bcb14 wininet!CFsm_ReadData::RunSM+0x2e

05c2f1b8 771bcac2 wininet!CFsm::Run+0x39

05c2f1d0 771c9cb5 wininet!DoFsm+0x25

05c2f1e0 771e4050 wininet!HTTP_REQUEST_HANDLE_OBJECT::ReadData+0x38

05c2f20c 771bcc1d wininet!HTTP_REQUEST_HANDLE_OBJECT::HttpSendRequest_Start+0x134

05c2f220 771bcb14 wininet!CFsm_HttpSendRequest::RunSM+0x59

05c2f238 771bcac2 wininet!CFsm::Run+0x39

05c2f250 771bccc5 wininet!DoFsm+0x25

05c2f278 771c60be wininet!HttpWrapSendRequest+0x140

05c2f29c 771c5fc3 wininet!HttpSendRequestA+0x1d

05c2f3f4 771bcb14 wininet!ParseHttpUrl_Fsm+0x22f

05c2f40c 771bcac2 wininet!CFsm::Run+0x39

05c2f424 771c5d2a wininet!DoFsm+0x25

05c2f494 771c5b92 wininet!ParseUrlForHttp_Fsm+0x214

05c2f4a0 771bcb14 wininet!CFsm_ParseUrlForHttp::RunSM+0x2b

05c2f4b8 771bcac2 wininet!CFsm::Run+0x39

05c2f4d0 771c5a42 wininet!DoFsm+0x25

05c2f51c 0818420f wininet!InternetOpenUrlA+0x1d7

WARNING: Stack unwind information not available. Following frames may be wrong.

05c2f55c 0818266f eecrlrev+0x420f

05c2f6a0 0818231d eecrlrev+0x266f

05c2f6b8 08181a81 eecrlrev+0x231d

05c2f6d8 0818131e eecrlrev+0x1a81

05c2f738 081846ac eecrlrev+0x131e

05c2f840 77a96075 eecrlrev+0x46ac

05c2f940 77a95f31 crypt32!VerifyDefaultRevocation+0x1d0

05c2f9b0 77a96b04 crypt32!CertVerifyRevocation+0xb7

We see here the client closed the socket and if we step back to the wininet!HTTP_REQUEST_HANDLE_OBJECT::CloseConnection function we can see (code review) the disconnect reason was due to the client inability to support the server mandate for Keep-Alive.

Scenario Post-mortem

Figure 3

Resolution

Enable HTTP 1.1 and HTTP 1.1 through proxy under Internet Options > Advanced. Using HTTP 1.1. enables the client to support the http Keep-Alive options which is required to support NTLM Proxy Authentication.