Cannot Browse a HTTPs Site Published by ISA Server 2006 without using TLS 1.0 on Internet Explorer
1. Problem
This week I got a really interesting collaboration call from Michael Hunter from Directory Services Team where customer was having problem to access a HTTPS site published through ISA Server 2006 using a certificate issued by an internal CA. In summary the access to the HTTPS web site published by ISA Server was only possible when the external client workstation was using TLS 1.0 (under Internet Options) as shown in Figure 1:
Figure 1 – Web Publishing rule was working only when client was using TLS 1.0 on IE.
The case was first worked with Directory Service team because the main concern was that they were using a third party external certificate in the Production ISA Server 2006 (similar configuration on ISA perspective) and there they didn’t have to use TLS 1.0 on the client to make it work. Based on this they were thinking that the issue was either on the internal CA (Microsoft Certificate Services) or in the certificate issued by the internal CA. In customer’s view, the only difference between Production and Development environments was the source of the certificate. The failing state was using a certificate issue by Microsoft Certificate Services and working state was using and external issued certificate by 3rd Party Certification Authority.
The topology was the following:
Figure 2 – Scenario where IE access works and does not work.
On failure scenario when external client unchecks TLS 1.0 setting on Internet Explorer user gets the error “Internet Explorer cannot display the webpage” as shown in Figure 3:
Figure 3 – Error message when client tries to browse the web site without using TLS 1.0.
2. Collecting Data and Understanding the Difference on both Scenarios
The first thing that we did was to get a netmon trace without TLS and with TLS enabled to see the difference. Here is the trace from the external without TLS enabled:
· TCP Handshake happen just fine from Client to ISA Server (external interface):
192.168.0.34 192.168.0.53 TCP TCP:Flags=......S., SrcPort=1281, DstPort=HTTPS(443), PayloadLen=0, Seq=3171858872, Ack=0, Win=65535 (scale factor 0) = 65535
192.168.0.53 192.168.0.34 TCP TCP:Flags=...A..S., SrcPort=HTTPS(443), DstPort=1281, PayloadLen=0, Seq=1737098665, Ack=3171858873, Win=16384 (scale factor 0) = 16384
192.168.0.34 192.168.0.53 TCP TCP:Flags=...A...., SrcPort=1281, DstPort=HTTPS(443), PayloadLen=0, Seq=3171858873, Ack=1737098666, Win=65535 (scale factor 0) = 65535
· SSL handshake starts and client presents SSL 3.0 but ISA Server sends a TCP FIN right after that:
192.168.0.34 192.168.0.53 SSL SSL: Client Hello.
- Ssl: Client Hello.
- SslV3RecordLayer: SSL 3.0 HandShake
ContentType: HandShake
+ Version: SSL 3.0
Length: 65 (0x41)
- SSLHandshake: SSL HandShake SSL 3.0 ClientHello(0x01)
HandShakeType: ClientHello(0x01)
Length: 61 (0x3D)
- ClientHello: SSL 3.0
+ Version: SSL 3.0
+ RandomBytes:
SessionIDLength: 0 (0x0)
CipherSuitesLength: 22
+ CipherSuites: SSL_RSA_WITH_RC4_128_MD5 { 0x00,0x04 }
+ CipherSuites: SSL_RSA_WITH_RC4_128_SHA { 0x00,0x05 }
+ CipherSuites: SSL_RSA_WITH_3DES_EDE_CBC_SHA { 0x00,0x0A }
+ CipherSuites: SSL_RSA_WITH_DES_CBC_SHA { 0x00,0x09 }
+ CipherSuites: TLS_NTRU_NSS_WITH_AES_256_CBC_SHA { 0x00, 0x64 }
+ CipherSuites: TLS_NTRU_NSS_WITH_3DES_EDE_CBC_SHA { 0x00, 0x62 }
+ CipherSuites: SSL_RSA_EXPORT_WITH_RC4_40_MD5 { 0x00,0x03 }
+ CipherSuites: SSL_RSA_EXPORT_WITH_RC2_CBC_40_MD5 { 0x00,0x06 }
+ CipherSuites: SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA { 0x00,0x13 }
+ CipherSuites: SSL_DHE_DSS_WITH_DES_CBC_SHA { 0x00,0x12 }
+ CipherSuites: TLS_NTRU_NSS_WITH_AES_128_CBC_SHA { 0x00, 0x63 }
CompressionMethodsLength: 1 (0x1)
CompressionMethods: 0 (0x0)
192.168.0.53 192.168.0.34 TCP TCP:Flags=...A...F, SrcPort=HTTPS(443), DstPort=1281, PayloadLen=0, Seq=1737098666, Ack=3171858943, Win=65465 (scale factor 0) = 65465
ISA Monitoring Logging just shows HTTPS Initiated and closed connection as shown in Figure 4:
Figure 4 – No much information here also.
Now let’s see how is the traffic with TLS 1.0 enabled on Internet Explorer (skipping the TCP 3 Way Handshake)):
· SSL Handshake with client sending the Hello:
192.168.0.34 192.168.0.53 SSL SSL: Client Hello.
- Ssl: Client Hello.
- TlsRecordLayer:
ContentType: HandShake
+ Version: TLS 1.0
Length: 65 (0x41)
- SSLHandshake: SSL HandShake TLS 1.0 ClientHello(0x01)
HandShakeType: ClientHello(0x01)
Length: 61 (0x3D)
- ClientHello: TLS 1.0
· ISA Server answers with the Server Hello:
192.168.0.53 192.168.0.34 SSL SSL: Server Hello. Certificate. Server Hello Done.
- Ssl: Server Hello. Certificate. Server Hello Done.
- TlsRecordLayer:
ContentType: HandShake
+ Version: TLS 1.0
Length: 1432 (0x598)
- SSLHandshake: SSL HandShake TLS 1.0 Server Hello Done(0x0E)
HandShakeType: ServerHello(0x02)
Length: 70 (0x46)
+ ServerHello: 0x1
HandShakeType: Certificate(0x0B)
Length: 1350 (0x546)
+ Cert: 0x1
HandShakeType: Server Hello Done(0x0E)
Length: 0 (0x0)
After the SSL Handshake is finished successfully the client can see the ISA Server FBA page. So clearly TLS 1.0 was playing a big role here since it was the only method that allowed the SSL handshake to complete. However, there was no setting on ISA Server that was forcing that to happen, this negotiation relies on Windows OS. Therefore, we have to focus on the differences from the Windows where the Production ISA Server and the Windows on the Development ISA Server.
4. Comparing Local Policies
For this type of subject one of the first things that should be reviewed is the Local Security Policy, questions like:
· Any hardening was applied too this server?
· Any specific group policy?
· Does this server belong to any OU that has GPO on it?
After some reviewing we found the following policy setting that was enabled on the ISA Server that was not working:
Figure 5 – FIPS Enabled on the non-working ISA Server.
This option can indeed cause this behavior; per KB811834 we have the following definition about this policy:
“If this setting is enabled, the security channel provider of the operating system is forced to use only the following security algorithms: TLS_RSA_WITH_3DES_EDE_CBC_SHA. This behavior forces the security channel provider to negotiate only the stronger Transport Layer Security (TLS) 1.0 protocol when you use applications such as Microsoft Windows Messenger, Microsoft MSN Messenger, and Internet Explorer to visit SSL sites.”
5. Conclusion
After disable that policy setting on ISA Server 2006 the issue was fixed and one more time ISA was only a victim. If that is a security requirement for the environment, it is important to remember that the clients will need to use TLS 1.0 enabled on IE.
Author
Yuri Diogenes
Security Support Engineer
Microsoft CSS Forefront Edge Team
Technical Reviewer
Daniel Mauser
Premier Field Engineer (ISA and PKI Specialist)
Microsoft Chicago