Bad Data error message in FIM CM web portal

  • Article

A customer with a FIM CM installation called in with the following problem description:

We have an issue with our FIM CM portal where some smartcards are failing unblock or retire operations.
Some cards are working fine but others give a "Bad Data" error message when a management operation is attempted on them.
Issuing new cards works without any problems.

Looking at the event log the following events are seen:

General
Information
*********************************************
Additional Info:
EnvelopedCMS decryption failed. Fall back to AES method.
1) Exception Information
*********************************************
Exception Type: System.Security.Cryptography.CryptographicException
Message: ASN1 bad tag value met.

What this means is that the first attempt to decrypt the message read from a table in SQL using the EnvelopedCms message that contains the identifying information that is necessary for the decryption failed - we're going to retry using whatever algorithm is specified in the CLM.Encryption.Algorithm entry in web.config.

This is typically because the data we're attempting to decrypt wasn't originally encrypted using the EnvelopedCms class - this method of encrypting was introduced in FIM CM as well as the fallback to the previous decryption method to cater for data in SQL that had been encrypted before that time.

The decrypted data would then in turn be written back to the SQL database in an encrypted format using the EnvelopedCms class - which should mean that fallback would only be required the first time the entry is decrypted and subsequent decryptions would use EnvelopedCms.

I.e. the "EnvelopedCMS decryption failed. Fall back to AES method" message is in itself not a fatal error message and can be expected behaviour - especially if you are using smartcards that have been managed using CLM or ILM 2007 prior to the introduction of EnvelopedCms.

However, there are instances where the fallback will also fail and you see an error message similar to the following:

General
Information
*********************************************
Additional Info:
Error happened when decrypting with AES
1) Exception Information
*********************************************
Exception Type: System.Security.Cryptography.CryptographicException
Message: Bad Data.

Possible causes for the decryption failure are:

  • You don't have the private key of the certificate used for the encryption present in the certificate store of the CLM Service account (typically the CLM Agent account)

    --> The certificate store of the CLM Agent account should contain the private keys of *all* CLM Agent certificates that have ever been used

  • The CLM Agent account doesn't have permissions to read the private key of the CLM Agent certificate used for the encryption (running Procmon during the repro should show Access Denied errors)

    --> The CLM Agent account needs at least Read permissions to the private key

  • The CLM Agent certificate was renewed using a new private/public key pair

    --> this requires the hash of the previous CLM Agent cert to be specified in the Clm.Decryption.Certificate.Hash value in the web.config file on the FIM CM server.

  • The algorithm specified inClm.Encryption.Algorithm in the web.config file is different from the algorithm used in the certificate specified in the Clm.Decryption.Certificate.Hash value in the web.config file. --> this can happen when a CLM Agent certificate has been enrolled for using the Microsoft Enhanced Cryptographic Provider v1.0 CSP (which does not support AES) instead of the Microsoft Enhanced RSA and AES Cryptographic Provider CSP

If you are using the Diversify Admin Key option for the cards then the Admin PIN for each will be reset to a random value that is then recalculated on the fly at each Admin operation that requires the Admin PIN (such as Unblock).

The Admin PIN is typically a 48-character number (many vendors default to 48 zeroes) - it may be possible to get into a state where you brick the smartcards permanently if the wrong Admin PIN is sent to the smartcard more often than the Admin PIN block threshold for the card allows (typically 3-5 failed attempts).
The error message returned when the card has been permanently bricked is typically "Bad PIN" or similar.

Further details:

EnvelopedCms Message
http://msdn.microsoft.com/en-us/library/bb924603.aspx

FIM2010 CM Data Item Collection and Verification
http://social.technet.microsoft.com/Forums/en-US/ilm2/thread/ad6a5c0c-8f78-4762-b481-36da6bb4b3c2

clmAgent certificate renewals
http://blogs.msdn.com/b/ms-identity-support/archive/2009/06/11/clmagent-certificate-renewals.aspx