How it works on the wire: IIS HTTP Client certificate authentication
I collaborated with a colleague recently where the IE client was failing to authenticate to IIS and I was requested to analyze a few network traces collected while reproducing the issue. The main issue was that the client (IE) wasn’t able to successfully authenticate to IIS server when certificate authentication was configured on IIS side and was getting the HTTP error 403.7.
First of all, I asked my colleague to setup a lab environment with IIS client certificate authentication configured so that we should be able to see how it works under normal circumstances. After the requested logs were collected, we had a chance to analyze the working traffic. Also I should note that we used server certificate with its private key to decrypt the SSL traffic.
=> Here is how it works in summary:
1) Client establishes an SSL session to the server.
2) Client then sends the initial HTTP Get request inside this SSL tunnel
3) Server responds to that request with a new SSL re-negotiate request since client certificate authentication is configured on the IIS server side.
4) Client establishes a new SSL session inside the original SSL session. This time IIS server also asks the client to send a certificate (IIS server also provides the client with the list of possible CAs from which the client certificate should be obtained)
5) Client sends its certificate and then the second SSL negotiation finishes successfully and the server returns HTTP 200 response right after the second SSL negotiation is finished successfully.
=> The TCP 3-way handshake to the IIS server:
No. Time Delta Source Destination Protocol Ack number Info
68 2013-12-26 13:03:51.654 0.000 10.10.10.16 10.10.10.15 TCP 49685 > 443 [SYN] Seq=923564226 Win=8192 Len=0 MSS=1460 WS=4 SACK_PERM=1
69 2013-12-26 13:03:51.654 0.000 10.10.10.15 10.10.10.16 TCP 923564227 443 > 49685 [SYN, ACK] Seq=4236088449 Ack=923564227 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
71 2013-12-26 13:03:51.657 0.003 10.10.10.16 10.10.10.15 TCP 4236088450 49685 > 443 [ACK] Seq=923564227 Ack=4236088450 Win=131400 Len=0
=> The initial SSL negotiation between the client and the IIS server:
72 2013-12-26 13:03:51.657 0.000 10.10.10.16 10.10.10.15 TLSv1 4236088450 Client Hello
=> Please note that IIS server doesn’t ask the client send a certificate at the initial SSL negotiation as there’s no “Certificate request” in the server response
73 2013-12-26 13:03:51.657 0.000 10.10.10.15 10.10.10.16 TLSv1 923564384 Server Hello, Certificate, Server Hello Done
74 2013-12-26 13:03:51.660 0.002 10.10.10.16 10.10.10.15 TCP 4236089827 49685 > 443 [ACK] Seq=923564384 Ack=4236089827 Win=130020 Len=0
75 2013-12-26 13:03:51.660 0.000 10.10.10.16 10.10.10.15 TLSv1 4236089827 Client Key Exchange, Change Cipher Spec, Finished
76 2013-12-26 13:03:51.664 0.004 10.10.10.15 10.10.10.16 TLSv1 923564710 Change Cipher Spec, Finished
77 2013-12-26 13:03:51.666 0.002 10.10.10.16 10.10.10.15 TCP 4236089886 49685 > 443 [ACK] Seq=923564710 Ack=4236089886 Win=129964 Len=0
Note: To be able to see traffic in clear text from this point on, you need to decrypt the SSL traffic in a way or another. I used the method that I mentioned in a previous blog post of mine:
=> The initial GET request (sent inside the SSL channel)
78 2013-12-26 13:03:51.666 0.000 10.10.10.16 10.10.10.15 HTTP 4236089886 GET /1.aspx HTTP/1.1
=> Instead of sending an HTTP response back to the client, the IIS server asks the client to re-negotiate SSL:
79 2013-12-26 13:03:51.666 0.000 10.10.10.15 10.10.10.16 TLSv1 923565216 Hello Request
80 2013-12-26 13:03:51.668 0.001 10.10.10.16 10.10.10.15 TCP 4236089923 49685 > 443 [ACK] Seq=923565216 Ack=4236089923 Win=131400 Len=0
81 2013-12-26 13:03:51.668 0.000 10.10.10.16 10.10.10.15 TLSv1 4236089923 Client Hello
=> This time the IIS server requires the client to send a certificate (as opposed to initial SSL negotiation) by adding “Certificate Request” to the response that it sends to client. It also includes the names of the accepted CA names from which a client certificate should be issued:
82 2013-12-26 13:03:51.669 0.000 10.10.10.15 10.10.10.16 TLSv1 923565413 Server Hello, Certificate, Certificate Request, Server Hello Done
83 2013-12-26 13:03:51.676 0.007 10.10.10.16 10.10.10.15 TCP 4236092088 49685 > 443 [ACK] Seq=923565413 Ack=4236092088 Win=131400 Len=0
84 2013-12-26 13:03:51.684 0.008 10.10.10.16 10.10.10.15 TCP 4236092088 [TCP segment of a reassembled PDU]
=> The client chooses an appropriate certificate from its certificate store (it should be a client certificate that should be issued by one of those CAs pointed to by the server and it also should have “client authentication” key usage. And then it sends it to the server within the response packet:
85 2013-12-26 13:03:51.684 0.000 10.10.10.16 10.10.10.15 TLSv1 4236092088 Certificate, Client Key Exchange, Certificate Verify
86 2013-12-26 13:03:51.684 0.000 10.10.10.15 10.10.10.16 TCP 923567204 443 > 49685 [ACK] Seq=4236092088 Ack=923567204 Win=131328 Len=0
=> The second SSL negotiation is finished successfully at this point:
87 2013-12-26 13:03:51.691 0.006 10.10.10.15 10.10.10.16 TLSv1 923567204 Change Cipher Spec, Finished
88 2013-12-26 13:03:51.694 0.002 10.10.10.16 10.10.10.15 TCP 4236092178 49685 > 443 [ACK] Seq=923567204 Ack=4236092178 Win=131308 Len=0
=> Then the server returns the response to the initial HTTP Get request:
89 2013-12-26 13:03:51.695 0.001 10.10.10.15 10.10.10.16 HTTP 923567204 HTTP/1.1 200 OK (text/html)
Note: The web page includes the word “Hello”
=> Then the client tries to retrieve favicon.ico file which doesn't exist on the server and finally the session is gracefully terminated by the client:
90 2013-12-26 13:03:51.701 0.005 10.10.10.16 10.10.10.15 TCP 4236092599 49685 > 443 [ACK] Seq=923567204 Ack=4236092599 Win=130888 Len=0
91 2013-12-26 13:03:51.795 0.094 10.10.10.16 10.10.10.15 HTTP 4236092599 GET /favicon.ico HTTP/1.1
92 2013-12-26 13:03:51.834 0.039 10.10.10.15 10.10.10.16 HTTP 923567582 HTTP/1.1 404 Not Found (text/html)
93 2013-12-26 13:03:51.844 0.009 10.10.10.16 10.10.10.15 TCP 4236094044 49685 > 443 [ACK] Seq=923567582 Ack=4236094044 Win=131400 Len=0
94 2013-12-26 13:03:51.844 0.000 10.10.10.16 10.10.10.15 TCP 4236094044 49685 > 443 [FIN, ACK] Seq=923567582 Ack=4236094044 Win=131400 Len=0
95 2013-12-26 13:03:51.844 0.000 10.10.10.15 10.10.10.16 TCP 923567583 443 > 49685 [FIN, ACK] Seq=4236094044 Ack=923567583 Win=130816 Len=0
96 2013-12-26 13:03:51.851 0.007 10.10.10.16 10.10.10.15 TCP 4236094045 49685 > 443 [ACK] Seq=923567583 Ack=4236094045 Win=131400 Len=0
After that, we collected the same set of logs from the customer environment and we realized that the IIS server wasn’t returning the CA name of which issued the client certificate to the client in the accepted CAs list even though that root CA certificate existed in the trusted root certificates store on the IIS server. So the IE client wasn’t sending the user certificate located at the user certificate store on its end.
Then we realized that the amount of root CA names were about 16 KB which seems to be an upper limit. After deleting some of the expired and unused root CAs from the IIS server the issue was resolved. You can find more details on this issue at the below article:
https://support.microsoft.com/kb/933430/EN-US Clients cannot make connections if you require client certificates on a Web site or if you use IAS in Windows Server 2003
The hotfix increases the Schannel security buffer to 16k. If you exceed this limit, you will still have issues that are described in the symptoms section of this article. This change has also been included with Windows Server 2008 and Windows Server 2008 R2. The workarounds described below will apply to Windows Server 2008 and Windows Server 2008 R2 as well.
But we were already hitting the 16 KB limit and hence the only solution seemed to be removing some expired or unused root certificates from the IIS server.
Hope this helps
Thanks,
Murat