ISA 2006 / TMG 2010: DISABLE CLIENT-INITIATED SSL RENEGOTIATION, PROTECTING AGAINST DOS ATTACKS AND MALICIOUS DATA INJECTION

In these days we received a considerable number of support requests asking for more info about SSL/TLS Renegotiation and the risk it introduces of being exposed to DoS attacks and malicious code injections.

The requests in object were focused on ISA/TMG products, considering they are used as reverse proxy for web publishing purposes, but the below considerations can be considered valid for every kind of Windows server/client supporting SSL/TLS connections.

First, what is exactly SSL/TLS Renegotiation?

"TLS [as defined in RFC 5246] allows either the client or the server to initiate a renegotiation -- a new handshake that establishes new cryptographic parameters. Unfortunately, although the new handshake is carried out using the cryptographic parameters established by the original handshake, there is no cryptographic binding between the two. This creates the opportunity for an attack in which the attacker who can intercept a client's transport layer connection can inject traffic of his own as a prefix to the client's interaction with the server"

The above definition is taken from RFC 5746.

The following is a graphic representation of a basic SSL/TLS Handshake:

Under certain circumstances, the client could be asking the server a renegotiation of the SSL/TLS parameters using the same TCP socket:

If the SSL/TLS is not secure (as per RFC 5746 recommendations) a MITM could use the renegotiation to send the server malicious data, pretending to be "good" user.

What chances do we have to mitigate this issue?

You should make sure, that the following security hotfix is installed:

http://support.microsoft.com/kb/980436

This fix is making the system compliant with RFC 5746, mitigating the risk of malicious data injection.

To provide backward compatibility, this security update works in the following modes: STRICT and COMPATIBLE

Compatible mode

o If this security update is applied to the server, and the server is in compatible mode, the server allows all clients to set up and renegotiate Transport Layer Security (TLS) sessions. This occurs whether the clients are updated or are not updated by using this security update.

o Similarly, if this security update is applied to the client, and the client is in compatible mode, the client can set up and renegotiate TLS sessions with all the servers for which this security update is applied or is not applied.

Strict mode

o If this security update is applied to the server, and the server is in strict mode, the server allows only those clients to which this security update is applied to set up and renegotiate TLS sessions. The server does not allow the clients to which this security update is not applied to set up the TLS session. In this case, the server terminates such requests from the clients.

Similarly, if this security update is applied to the client, and the client is in strict mode, the client can set up and renegotiate TLS sessions with all the servers for which this security update is applied. The clients cannot set up TLS sessions at all with servers for which this security update is not applied. The client cannot move ahead with a TLS negotiation attempt with such servers.

By default, this security update enables the TLS or Secure Sockets Layer (SSL) client or server to stay in compatible mode. An administrator can use the AllowInsecureRenegoClients and the AllowInsecureRenegoServers entry DWORD values in the following registry path to enable strict mode on the client or on the server:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL

The following table shows how these DWORD values can be used:

Malicious data injection is not the only problem related to Renegotiation: it can be in fact used to perform DoS attacks against a server.

When a new SSL/TLS connection is being negotiated, the server will typically spend significantly more CPU resources than the client. A malicious user, leveraging the Renegotiation, could be able to enhance the server's CPU usage causing DoS.

In order to have a better mitigation for both malicious data injection and DoS attacks, the best option would be to reject the client-initiated SSL/TLS renegotiation at all.

The following Microsoft Security Advisory explains how:

http://support.microsoft.com/kb/977377/en-us

As reported in the article, the behavior can be modified by changing the value of the following registry key:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\DisableRenegoOnServer

Notes

· If the DisableRenegoOnClient subkey is present and has any nonzero value:

o The client will not initiate renegotiation.

o The client will not respond to renegotiation.

· If the DisableRenegoOnClient subkey is missing or is present and has a zero value:

o The client will initiate renegotiation.

o The client will respond to renegotiation.

· If the DisableRenegoOnServer subkey is present and has any nonzero value:

o Server initiated renegotiation is not allowed.

o The server will not respond to renegotiation requests from the client.

· If the DisableRenegoOnServer subkey is missing or is present and has a zero value:

o Server initiated renegotiation is allowed.

o The server will respond to renegotiation requests from the client.

Of course, this may have an impact on the use of specific applications requiring SSL/TLS renegotiation feature.

The KB article underlines the following:

o After you install this security update, you cannot use the legacy provisioning service parameter ( –UseLegacyProvisioningService) when you create a federation trust with the Microsoft Federation Gateway. The security update will prevent the federation trust from working correctly. This problem will occur if you install this security update on a computer that is running Exchange Server 2010 or Exchange Server 2010 Service Pack 1 before you have created a federation trust. To avoid this problem, you must create the federation trust before you install this security update. 
For more information about how to create a federation trust by using the –UseLegacyProvisioningService parameter, visit the following Microsoft webpage:
Create a Federation Trust

o When you install this update on a computer that has the Microsoft Online Single Sign-In services client installed, you may experience the following issues:

· The Sign In client cannot obtain user configuration information. This only affects new users who are running the Sign In client for the first time. The Sign In client cannot obtain information to configure Outlook. If the Sign In client has already run and configured the applications, there are no additional issues with the Sign In client.

· Outlook users cannot see free/busy information. Therefore, this update also affects existing Outlook users.
To resolve this problem, set the DisableRenegoOnClient registry entry to a value of 0 (zero), and then restart the computer.

o This update disables TLS/SSL renegotiation, common protocol functionality that is required for specific applications. This may cause this software to no longer function as expected. If any side effects are experienced, customers should uninstall the workaround to resolve the issue.
The following software has been tested by Microsoft and that has been found to experience problems when you install this update:

· Windows 7 DirectAccess: The IP HTTPS interface will not function.

· Exchange ActiveSync: Does not function when it uses certificate client authentication.

· Internet Information Services (IIS) : In certain configurations, IIS using certificate client authentication, including certificate mapping scenarios, will be affected. Site-wide client certificate authentication will not be affected and will continue to function.

· Internet Explorer: When you browse Web sites that require client certificate authentication, but not site-wide client certificate authentication, you may not successfully be able to connect.

Of course, it's not possible to predict the implications of disabling client-initiated renegotiation for various applications: this solution should be appropriately tested in each specific environment.

From a security point of view, though, this is the recommended way to mitigate all the above described problems.

Hope this can help!

Author:  

Daniele Gaiulli

Support Engineer – Microsoft Forefront Edge Security Team

Reviewer:

Philipp Sand

Sr. Support Escalation Engineer – Microsoft Forefront Edge Security Team