Replacing Security Token Service (STS) certificate of the on-premises SharePoint Server fails

Background
==========
When a trust relationship is needed between two SharePoint Server farms, or when a farm is configured to participate in a hybrid environment, you must use a common STS certificate between the trust members. We recommend that you buy a new STS certificate from a public Certificate Authority (CA). This gives you the highest level of certificate security, and reduces the possibility that a self-signed certificate will have integration issues with other applications and services. You can use a self-signed certificate for test or pilot environments.

https://technet.microsoft.com/en-us/library/dn551378.aspx

Issue
====
You are following steps given in the article and receive below error when using SHA256 algorithm based certificate(Self-signed or CA).
"The timer job completed, but failed on one or more machines in the farm"

Command used:
$pfxPath = "<path to replacement certificate (.pfx file)>"
$pfxPass = "<certificate password>"
$stsCertificate = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 $pfxPath, $pfxPass, 20
Set-SPSecurityTokenServiceConfig -ImportSigningCertificate $stsCertificate

 Error in ULS: 
02/06/2018 14:31:52.29 OWSTIMER.EXE (0x04B0) 0x0FD8 SharePoint Foundation Timer 6398 Critical The Execute method of job definition Microsoft.SharePoint.Administration.SPServiceApplicationInstanceProvisioningJobDefinition (ID 64a8d164-df91-49b2-b568-462a12a1cb71) threw an exception. More information is included below. Invalid provider type specified. 8cf6479e-5da4-5081-f6a1-6b94c53300d8

02/06/2018 14:31:52.29 OWSTIMER.EXE (0x04B0) 0x0FD8 SharePoint Foundation Timer 72ae Unexpected
 Exception stack trace:
 at System.Security.Cryptography.Utils.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
 at System.Security.Cryptography.Utils.GetKeyPairHelper(CspAlgorithmType keyType, CspParameters parameters, Boolean randomKeyContainer, Int32 dwKeySize, SafeProvHandle& safeProvHandle, SafeKeyHandle& safeKeyHandle)
 at System.Security.Cryptography.RSACryptoServiceProvider.GetKeyPair()
 at System.Security.Cryptography.X509Certificates.X509Certificate2.get_PrivateKey()
 at Microsoft.SharePoint.Administration.SPCertificateProvisioningAssistant.ProvisionCertificate(String storeName, StoreLocation storeLocation, SecureString exportedCertificate, CryptoKeySecurity security)
 at Microsoft.SharePoint.Administration.SPCertificateAuthority.ProvisionCertificate(String storeName, StoreLocation storeLocation, SecureString exportedCertificate, CryptoKeySecurity security)
 at Microsoft.SharePoint.Administration.Claims.SPSecurityTokenServiceApplication.ProvisionLocal(SPServiceInstance serviceInstance)
 at Microsoft.SharePoint.Administration.SPServiceApplicationInstanceProvisioningJobDefinition.Execute(Guid targetInstanceId)
 at Microsoft.SharePoint.Administration.SPAdministrationServiceJobDefinition.ExecuteAdminJob(Guid targetInstanceId)
 at Microsoft.SharePoint.Administration.SPTimerJobInvokeInternal.Invoke(SPJobDefinition jd, Guid targetInstanceId, Boolean isTimerService, Int32& result)
 8cf6479e-5da4-5081-f6a1-6b94c53300d8

Cause:
=====
The certificate created by CA or Self-Signed is created without the KEYSPECS set to "AT_KEYEXCHANGE".

The KeySpec property identifies how a key generated or retrieved by Microsoft CryptoAPI (CAPI), from a Microsoft legacy Cryptographic Storage Provider (CSP), can be used.
A KeySpec value of 1, or AT_KEYEXCHANGE, can be used for signing and encryption. A value of 2, or AT_SIGNATURE, is only used for signing.
The most common KeySpec mis-configuration is using a value of 2 for a certificate other than the token signing certificate.
For certificates whose keys were generated using Cryptography Next Generation (CNG) providers, there is no concept of key specification, and the KeySpec value will always be zero.

To see a certificates value you can use the certutil command line tool.
The following is an example: This will dump the certificate information to the screen
certutil –v –store my

You will notice that ProviderType=0 for failing certificate

  CERT_KEY_PROV_INFO_PROP_ID(2):
 Key Container = CertReq-8186d92c-a1ba-4a0d-8057-2c6e164e54c2
 Unique container name: 19a9eec0e1d58579ec9521356b1f3dab_7b6bef17-b3d8-4810-b027-f4271b5b8c99
 Provider = Microsoft Software Key Storage Provider
 ProviderType = 0
 Flags = 20 (32)
 CRYPT_MACHINE_KEYSET -- 20 (32)
 KeySpec = 0 -- XCN_AT_NONE
 However, a working certificate will have below details:
 CERT_KEY_PROV_INFO_PROP_ID(2):
 Key Container = 27925d79f29201c6ff775267682bcdec_7b6bef17-b3d8-4810-b027-f4271b5b8c99
 Simple container name: te-1a815876-5775-40db-b18f-9829d9aead99
 Provider = Microsoft Strong Cryptographic Provider
 ProviderType = 1
 Flags = 20 (32)
 CRYPT_MACHINE_KEYSET -- 20 (32)
 KeySpec = 1 -- AT_KEYEXCHANGE

More Information: /en-us/windows-server/identity/ad-fs/technical-reference/ad-fs-and-keyspec-property

Solution:
=======
You would require to get a new CA certificate or self-signed certificate which is created with KeySpec value of 1, or AT_KEYEXCHANGE. This is required because we cannot update the new certificate using "Set-SPSecurityTokenServiceConfig" command. The command does not have option to import the certificate and set the KeySpec value to 1.

For other applications: Changing the KeySpec value does not require the certificate to be re-generated or re-issued by the Certificate Authority. The KeySpec can be changed by re-importing the complete certificate and private key from a PFX file into the certificate store using the steps below:

  1. Export the certificate including private key to a PFX file.
  2. Delete the existing certificate
  3. Open an elevated PowerShell command prompt and import the PFX file using the cmdlet syntax below, specifying the AT_KEYEXCHANGE value:
    certutil –importpfx certfile.pfx AT_KEYEXCHANGE
  4. Restart the service.

You can also generate the a self-signed certificate using below command:

New-SelfSignedCertificate -DnsName "dns.domain.com" -KeySpec "1" -HashAlgorithm "SHA256" -CertStoreLocation "cert:\LocalMachine\My"
Note: This command will run successfully on Windows 10 and Windows Server 2016