Fun with ISA Server and AES Cipher Suites
What is “AES”?
“AES” stands for “Advanced Encryption Standard”; a symmetric encryption algorithm used in several encryption schemes, such as FIPS-197. http://csrc.nist.gov/archive/aes/index.html provides links to detailed discussions of AES.
1. Netaction Encryption Guide article
2. NIST FIPS-197 article
3. Microsoft Knowledgebase article 246071 Description of Symmetric and Asymmetric Encryption
4. Microsoft Knowledgebase article 948963 Update to add two AES cipher suites to Windows Server 2003
5. Microsoft Knowledgebase article 922706 How to use Certificate Services Web enrollment pages with Windows Vista or Windows Server 2008
6. Microsoft Developer Network article Microsoft Cryptographic Service Providers
7. Microsoft Developer Network article Microsoft AES Cryptographic Provider
8. Microsoft Developer Network article AES Provider Algorithms
AES was initially offered in Windows XP WPA2 updates and later in Vista and Windows Server 2008 as part of the improved cipher suites. These same encryption algorithms may be used to create the public and private keys used to sign certificates as well, which brings us to the point of this article – certificate keys generates using AES cipher suites.
The Fun
Anyone who has used ISA Server to publish a secure Web application has become familiar with the necessity for obtaining and installing a certificate suitable for server authentication. The methods available for acquiring a certificate suitable for use in the Web listener vary greatly.
Server Authentication Certificates
Some folks like to use the command-line tools certreq and certutil, while others prefer to use the certificate enrollment pages offered by Windows Certificate Services, and still others will opt for a popular open-source tool, such as SelfSSL. The point is that you must create a certificate that meets the following criteria:
1. The issuing CA certificate is stored in the local computer Trusted Roots store
2. Enhanced Key Usage includes Server Authentication (1.3.6.1.5.5.7.3.1)
3. Key Usage includes Digital Signature, Key Encipherment
4. Subject or Subject Alterative Name includes a name appropriate to the URL used by the client
When you install the certificate, you must satisfy the following criteria:
1. The certificate is stored in the local computer Personal store
2. The private key is stored in the Local computer Personal store
The ISA Server 2006 Supportability Update (included in Service Pack 1) allows ISA Server to validate all of these requirements and will generate alerts when any of them are not satisfied. One thing ISA does not do until it actually tries to use the certificate through CAPI is to validate that the encryption keys are usable; in other words, that they can be used to sign and encrypt data. This is where the fun begins.
Using the Certificate Enrollment Page
By default, Windows Server 2003 cannot use AES-based encryption so any certificate key that is created using AES-based cipher suites will generate certificate access errors for the application that is trying to use the certificate. If you’re not paying attention to this difference when you request the certificate, you could generate a certificate that appears perfectly functional for the intended purpose, but will actually cause a failure when ISA Server attempts to apply the policies because Windows Server 2003 cannot process AES-based ciphers.
If you obtain your certificate from a Windows Server 2008 Certificate Authority enrollment web site, you have the ability to select the Cryptographic Service providers (CSP) for use in the private key. If you connect using a Windows XP or Windows Server 2003 host, the CSP provided by default is the Microsoft Enhanced Cryptographic Provider v1.0 as shown in Figure 1.
Figure 1 Default CSP for Windows XP and Windows Server 2003 clients
If you connect using a Windows Vista or Windows Server 2008 client, the default CSP chosen will be Microsoft Enhanced RSA and AES Cryptographic Provider as shown in Figure 2.
Figure 2 Default CSP for Windows Vista and Windows Server 2008 clients
The upshot of this is that if you:
1. requesting a certificate from the Windows Server 2008 certificate enrollment pages
2. connect using a Windows Vista or Windows Server 2008 computer
3. accept the default CSP
..the certificate will not usable at a Windows Server 2003 computer that does not have the patch offered by MSKB 948963.
ISA Server Certificate Errors
If you install a certificate that uses AES-based CSP on an ISA Server, you will encounter startup or policy application failures when ISA Server attempts to apply the policy updates to include the new certificate, as shown in the following examples:
Figure 3 ISA Server Standard Edition Certificate Access Error
Figure 4 ISA Server Enterprise Edition Certificate Access Error
You should notice that the common point between these two events is that they both indicate the same error code; 0x030d. ISA Server Enterprise Edtition translates the CAPI error code 0x8009030d to a WINERR 0x8007030d, but they're the same lower-level error. This error resolves to SEC_E_UNKNOWN_CREDENTIALS, which in this case means “CAPI couldn’t process the private key”.
Identifying the Problem
Unfortunately, you can’t use the certificates MMC to determine if an AES CSP is the cause of the certificate failure. Fortunately, the Windows security team saw fit to include a nice command-line tool that will tell you quite a lot about your certificate storage called certutil.exe. The syntax you would use to examine the certificates contained in the local computer Personal store is certutil -store my. The following provides an example of two certificates located in the local computer Personal store that were created using identical criteria except for the CSP:
C:\> certutil -store my
================ Certificate 0 ================
Serial Number: 14024454000000000002
Issuer: CN=Contoso-CA, DC=contoso, DC=com
Subject: CN=oa.contoso.com
Non-root Certificate
Cert Hash(sha1): 41 4c 34 72 70 95 32 06 07 e2 0f a5 46 c4 cf 5b a7 eb e8 80
Key Container = {DF949654-2A4D-4860-AC31-9BA23242AE6B}
Provider = Microsoft Enhanced RSA and AES Cryptographic Provider
Private key is NOT exportable
Encryption test passed
================ Certificate 1 ================
Serial Number: 16e30e94000000000004
Issuer: CN=Contoso-CA, DC=contoso, DC=com
Subject: CN=oa.contoso.com
Non-root Certificate
Cert Hash(sha1): 3c 58 e3 7b 90 8b 8c 4b 78 28 f8 f8 ad ad ed 99 fa 2f 00 9f
Key Container = {BED1CF85-B454-4C8D-9EB6-529265D7B1B7}
Provider = Microsoft Enhanced Cryptographic Provider v1.0
Private key is NOT exportable
Encryption test passed
When you select the certificate listed as “Certificate 0” for use in an ISA Server Web listener, if the ISA Server does not have the patch offered in MSKB 948963 installed, it will fail to use the certificate and generate alerts and event log entries indicating SEC_E_UNKNOWN_CREDENTIALS errors. When you select the certificate listed as “Certificate 1” in the certutil output, ISA Server will use the certificate normally. It's worth noting that although certutil indicates "encryption test passed", this does not mean that it actually used the keypair to encrypt & decrypt anything with them. What certutil does is validate that the oldest (most commonly available) cipher is usable for encryption. This means that in the case of the RSA / AES cipher, AES is skipped.
Solution
You can solve this problem in a couple of ways:
1. Ensure that your certificate request specifies a CSP that does not include AES ciphers
2. Install the patch offered in MSKB 948963
Wrap-Up
The problem is a bit obscure if you’re not fully conversant with CAPI error codes and since it took me the better part of the day to sort out all the various aspects of the problem, I thought it might be useful information for the larger ISA community.
HTH,
Jim Harrison, Program Manager, FF Edge CS
Reviewers
Avi Ben-Menahem, Principle Group Program Manager, FF Edge
Yuri Diogenes, Sr Support Engineer, CSS Security
Mohet Sexana, Technical Lead, CSS Security