How to troubleshoot Certificate Enrollment in the MMC Certificate Snap-in

Article
11/06/2007

Hi, Seth Scruggs here from the Directory Services team. Today I’m going to discuss how to troubleshoot certificate enrollment in Windows using a Windows Server 2003 Certification Authority (CA). Before you read on, make sure you have the Windows Server 2003 Resource Kit, the Windows Server 2003 or Windows XP Support tools, and the Windows Server 2003 admin pack installed. All will be needed for troubleshooting!

There are four ways we enroll for certificates in Windows:

MMC based enrollment
Auto Enrollment
Web Enrollment
Manual Enrollment (certreq.exe, lcscertutil.exe, etc)

This blog is going to specifically cover how to troubleshoot enrollment through the MMC Certificate Snap-in. If you are troubleshooting auto enrollment, the first step is to always try MMC-based enrollment; if you find this fails, there is no point troubleshooting auto enrollment until MMC-based enrollment works.

We call this MMC enrollment because we do this from the Certificates snap-in that can be added to an MMC. To use this snap-in, click Start | Run | MMC. Then in the blank MMC, click File | Add/remove snap-in | Certificates. I am specifically writing this blog to cover user based enrollment (as opposed to computer based enrollment), but the concepts can be applied to both.

MMC or snap-in based enrollment breaks in one of two spots; when you launch the wizard or when you click finish at the end of the wizard.

Troubleshooting errors when you launch the wizard

Let’s start by doing a high level walkthrough of the beginning steps and understand why it might fail here:

1. We first query Active Directory and search for a list of available CAs. When the client retrieves the result of the query, it filters out the results based on the following:

Is this an Enterprise CA?
Do I trust this CA?
Am I sure this CA isn’t revoked?
Do I have permissions to the CA?

2. Then we query Active Directory for a list of certificate templates. When the client retrieves the result of the query, it filters out the results based on the following:

Do I have enroll permissions on any certificate templates?
Are any of the templates that I have enroll permissions on available on the CA/s we identified before?
Is the template configured correctly to allow for enrollment in the MMC?

If the combination of these filters leaves your template or CA list blank, then you receive an error when you launch the wizard:

Do you have this error? Let’s use some tools and troubleshoot this. If you don’t see this error when you launch the Request Wizard, you can read this next section just for fun or skip this and go directly to “Troubleshooting errors when you click finish at the end of the Wizard”.

Querying Active Directory

We started by looking for CAs that are published in AD. Specifically, the client does an LDAP query for objects in the following container:

CN=Enrollment Services,CN=Public Key Services,CN=Services,CN=Configuration,DC=contoso,DC=com

This container should hold one object for every internal Windows CA installed in the forest. If for some reason the user doesn’t have permission to read this container, the objects beneath it, or there are no objects beneath it, we fail. By default Authenticated Users have read access on this container; so start by verifying this is the case in your forest. We’re going to use ADSIEdit.msc from the Windows Support tools to look at permissions.

Launch ADSIEdit.msc, then expand CN=Configuration | CN=Services | CN=Public Key Services | CN=Enrollment Services. This should look like:

You can see from the screen shot that in my lab there is one object under CN=Enrollment Services. If you don’t have an object in your environment (but you know you have a CA) then something went wrong during installation or an administrator in your forest deleted the object. In either case, you can re-populate the object by logging on to the CA as an Enterprise Admin or Forest Root Domain Admin and doing a backup and restore of the CA in the CertSrv.msc console.

If you have verified that one or more objects actually exist in the CN=Enrollment Services, let’s check permissions. Right click on the CN=Enrollment Services container, select properties and then click on the Security tab. We should see that Authenticated Users have read permission on the container.

If Authenticated Users is not listed here, you’ll need to add this group and assign Read Permissions to this container and any child container beneath it.

You could reset permission on the container to the default permissions as defined by the schema using DSACLS.exe, but this is a shotgun approach and you could remove any custom permissions that were previously delegated to this container. If you feel like doing it anyways, the syntax is here:

DSAcls “CN=Enrollment Services,CN=Public Key Services,CN=Services,CN=Configuration,DC=contoso,DC=com” /S

OK, so you made sure Authenticated Users have read permissions on CN=Enrollment Services, and you made sure that there is actually one or more objects in the container. The next step is to make sure that your CA is an Enterprise CA and that the object/s in ADSIEDit.msc reflect this. We’ll do this in two steps.

First, logon to your CA and at the command prompt run certutil –cainfo. In the output find the field labeled CA type. If it says Enterprise Subordinate CA or Enterprise Root CA then we are fine. If it displays Standalone Root or Standalone Subordinate CA then MMC enrollment will not be possible and you must choose another request method.

After you have verified that you actually have an Enterprise CA, let’s look at the CA object in ADSIEdit.msc and make sure the flag that identifies it as an Enterprise CA is set correctly. It is very unusual to see the flag set incorrectly, but all the same it is possible. As we did before, launch ADSIEdit.msc, then expand CN=Configuration | CN=Services | CN=Public Key Services | CN=Enrollment Services. Right click the CA in the right pane that you want to enroll from and click properties. Find the flags attribute; and verify that it is set to 10. If it isn’t set to 10, then set it to 10 using ADSIedit.msc and allow for Active Directory replication to complete.

Verifying the CA certificate

The next step is to make sure that I trust the CA, and that I can make sure the CA is not revoked. Certificate verification is kind of a big topic, and I’m going to barely touch it. If you want in-depth knowledge, you can read this whitepaper.

The easiest way to verify do this is to launch PKIView.msc (available in the Windows Server 2003 Resource Kit). Once open, right click on Enterprise PKI and select Manage AD Containers. When this opens, click on the Enrollment Services tab.

We can already tell that the user trusts this CA, because the status is OK. This can be a bit misleading, but is a good start. Now let’s do an in-depth verification:

Highlight the certificate and click View. When the certificate dialog box opens, click on the Details tab, then click Copy to file. Export the certificate as a .cer file (DER or Base-64 encoding is fine). At the command prompt, run certutil –verify C:\filename.cer >verifyresults.txt, replacing C:\filename.cer with the path and file name of the certificate file you exported. After it runs, open verifyresults.txt and scroll down to the very bottom. Any error message at the bottom indicates a chaining or revocation checking problem; either of which would cause an enrollment to fail.

A common chaining problem looks like this:

A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider. 0x800b0109 (-2146762487) ------------------------------------ Verifies against UNTRUSTED root

If you see a chaining error message, make sure that the root certificate is installed in the trusted root store. You also might have to make sure any intermediate certificates are installed in the intermediate store.

A typical revocation error looks like this:

ERROR: Verifying leaf certificate revocation status returned The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613).

CertUtil: The revocation function was unable to check revocation because the revocation server was offline.

If you see a revocation checking error message, run certutil –verify –urlfetch C:\filename.cer >urlfetch.txt. Open urlfetch.txt and find the CDP sections of the output. In this section, examine each path and figure out why this client isn’t able to reach the path. A client doesn’t need to be able to reach all paths, but does need to be able to reach at least one path for each CA. An example looks like this:

---------------- Certificate CDP ---------------- Failed "CDP" Time: 0 Error retrieving URL: The specified network resource or device is no longer available. 0x80070037 (WIN32: 55) ldap:///CN=2003Dom%20Enterprise%20Issuing%20CA,CN=2003DOMCA01,CN=CDP,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=2003Dom,DC=com?certificateRevocationList?base?objectClass=cRLDistributionPoint

Failed "CDP" Time: 0 Error retrieving URL: The server name or address could not be resolved 0x80072ee7 (WIN32: 12007)

https://2003domca01.2003dom.com/CertEnroll/2003Dom%20Enterprise%20Issuing%20CA.crl

In this case you can see we can’t get to either CRL distribution point URL. Again, we don’t need to get to both, but we have to be able to get to one of them.

Request Permissions

If everything looks good with chaining and revocation status, let’s move on to CA request permissions. Permissions are placed on the CA and are saved on the object that is stored in the Enrollment Services container. To verify the client has permission to request from the CA, open CertSrv.msc on the CA, right click on the name of the CA, and then click on the Security tab. By default Authenticated Users have the Request Certificates permission. If Authenticated Users don’t have this permission, make sure that the client requesting the certificate is a member of some group that does have this permission.

Querying Active Directory (again)

At this point we have queried for CAs that are available in the forest and we have filtered down our list. If we have identified one or more valid CAs, we do another LDAP query for all of the Certificate Template objects in the following container:

CN=Certificate Templates,CN=Public Key Services,CN=Services,CN=Configuration,DC=contoso,DC=com

First we look at the permissions on each object returned, and determine if the client has Read and Enroll permissions on the certificate template. An easy way to verify permissions is to logon as the requesting user and run certutil –template on the client (on XP, you must install the Windows Server 2003 Admin pack to use this utility). This will dump all of the certificate templates in the CN=Certificate Templates container; and if the user doesn’t have permission to one of them it will display Access is Denied next to that template.

If the certificate template you want to use has an Access is Denied next to it check the user’s group memberships and/or the permissions on the Certificate Template. To verify group membership of the user, you can run whoami /groups (whoami is part of the Windows Support tools on XP and included in the OS with Windows Server 2003). To check permissions on the certificate template, open CertTmpl.msc, find and double click the certificate template you want, then go to the security tab. If you are making constant changes to this stuff, remember that AD replication latency is a factor.

OK, now that we have confirmed permissions are OK, let’s make sure this CA offers the certificate template we want. The Wizard does this by looking at the Certificate Templates attribute on the objects in CN=Enrollment Services. We can do this by opening certsrv.msc on the CA, then clicking on the Certificate Templates folder. All you need to do is make sure you see the desired template in the list:

If your certificate template is in the list, then you’re good. If not, right click on Certificate Templates, select New Certificate Template to issue, and then choose the correct template to add.

Verifying Certificate Template Settings

The last thing to check is if the template settings allow it to ever show up in the MMC. If the Certificate Template is set to supply the subject name in the request, it will never appear in the MMC because the MMC (in 2K/XP/2003) doesn’t allow you to enter this value. For the template to be offered in the MMC, the subject name must be built from Active Directory. The setting on the template should look like one of these:

At this point we have covered all of the reasons that a request through the MMC Snap-in might initially fail. Let’s look at what happens when you finish the wizard, and why that might fail.

Troubleshooting errors when you click finish at the end of the Wizard

Let’s do a high level walkthrough of the process when you click the Finish button in the wizard:

We use the settings from the selected Certificate Template to build a certificate request.
Send the request to the CA using an RPC connection and receive it back.

If one of these steps fails, you could receive a few different errors, but the most common is:

Let’s break down the steps and figure out why we might see this error message.

We start by reading the certificate template settings and building a request. To do this, we use the Cryptographic Service Provider (CSP) specified by the template (or chose one locally based on the template setting) to generate a key pair and create a key container to store the private key. Then we sign the data in our request with the private key and store the request locally in the Pending Request certificate store.

The most common cause of failure in this step is permissions on the directory where we create key containers. For a user, we create the private key container in the user’s profile in:

%userprofile%\Application Data\Microsoft\Crypto\RSA\<SID of user>

If you look, you might find that you already have dozens of files in this directory; or you could have none. What is important is that the permissions on this directory look like this; with all three security principals having inherited full control.

If yours isn’t like this, restore them to defaults. This should be:

Administrators Full Control
<The Username> Full Control
SYSTEM Full Control

These should be inherited, and apply to This Folder, subfolders, and files. If this looks good and we are sure that writing to this directory isn’t our problem, let’s move on to the next step.

Submitting the request to the CA

Once we build the request, we send it to the CA using DCOM/RPC. Using the dnsHostName attribute on the CA’s object in CN=Enrollment Services, we grab the DNS name of the server, resolve the server name using DNS, then send an RPC end point mapper (EPM) request to the CA over port 135. The EPM on the CA returns a port to which the client may send the request. In order for any of this DCOM magic to work, both the client and the CA must have DCOM enabled and configured correctly. Check on your client and server by opening the Component Services snapin: Start | Run | DCOMCNFG. Once open, expand Component Services | Computers | My Computer. Right click on the Default Properties tab. It should look like this on both the client and server:

Now let’s check for port connectivity. As I stated before, we must have access to port 135; in addition to this the random port range that must be open is 1024 – 65534. This can be changed, but is out of scope for this blog. For additional information check out the ports article.

There are many ways to test connectivity; including telnet.exe, network traces, or portqryui.exe. We are going to use Port Query. Before we test connectivity, let’s find out which ports the CertSrv.exe service is listening on. Since the port is allocated from a dynamic range and is a moving target, we’ll need to do this in two steps. First, dump the list of processes on the Certificate Server:

Tasklist >tasklist.txt

Open this file, and find the PID (process ID) of the certsrv.exe process (in this case it is 1152)

Second, find out which ports are open by PID:

Nestat –ano >ports.txt

Open this file, and find each instance of your PID (again, my PID was 1152). It is expected to see more than one port open for this process, since we have multiple interfaces for the service. We are interested in the port bound to 0.0.0.0, in this case port 1104.

So, we have two ports of interest, 1104 and 135. Now download, extract, and launch portqryui.exe on the client. Enter the IP or FQDN of the server that hosts Certificate Services, choose Manually input query ports, and enter the port you found plus port 135.

You can see in the output that port 135 was identified as the endpoint mapper and is LISTENING (this is good). Now scroll down to the bottom and look for the evaluation of port 1104.

Again, LISTENING, so we shouldn’t have any connectivity issues.

After checking port connectivity, the next step is to consider DCOM permissions on the CA. In Windows 2K3 SP1, we hardened security on the CertSrv Request DCOM interface. Basically, we create a new Local group called CERTSVC_DCOM_ACCESS and only allow members of this group permission to hit this interface. On a member server, this group is local and it contains the Everyone group. However, if the CA is installed on a domain controller, we create the group as a domain local group and add Domain Users and Domain Computers from that domain. The problem is that this doesn’t include the Domain Controllers group; or the Users, Computers, or Domain controllers from any other domain in the forest.

A good article on this change is here. Basically, make sure that your user is in a group that is a member of the CERTSVC_DCOM_ACCESS group. If you have Windows Server 2003 with SP1 or higher and you don’t see this group, follow the steps in the article to recreate the group using certutil -setreg SetupStatus -SETUP_DCOM_SECURITY_UPDATED_FLAG. Again, if the CA is on a DC, it will be a Domain Local group, otherwise it will be a regular local group.

Keep in mind that if you make adjustments to the group by adding new members, the client must be restarted to build its logon token. One way to test permissions (and connectivity) is to run certutil –config FQDN\CAName –ping at the command prompt on the client where FQDN is the fully qualified name of the server and CAName is the subject name of the certificate for that CA. An example would be:

certutil –config “server01.contoso.com\Contoso Enterprise Sub CA” –ping

If you don’t have permission to the DCOM interface, an Access is Denied message will be returned.

Conclusion

Well, that is about it. Hopefully if you are troubleshooting an issue your problem is now resolved. These steps obviously won’t fix all issues, but easily the top 90%. Please feel free to add comments or other questions!