Deep dive into UAG DirectAccess (Certificate Enrollment)

Today I want to talk about how to configure the UAG DirectAccess server security policy to enable certificate enrollment from the Certificates MMC console.

By default, when you try to use the Certificates MMC console for certificate enrollment from the UAG DirectAccess server you will see the RPC server is unavailable message, as seen in figure 1.

clip_image002 

Figure 1

This is due to the default security policy on the UAG machine.

To help you solve this problem, I’d like to shed some light about the networking protocols that are initiated behind the scenes when you try to request a certificate using the Certificates MMC console and how you can enable these protocols in the Forefront UAG server security policy so that you can successfully request certificates using the Certificates MMC.

The Networking Protocols used by the Certificates MMC for Certificate Enrollment

When you run the Certificate Enrollment wizard, in most cases you will try to connect to the Active Directory Enrollment Policy. That policy exists in Active Directory, and requires LDAP connectivity from the UAG DirectAccess server to your domain controllers.

After the Wizard connects to the Active Directory, it retrieves the different elements associated with Certificate Enrollment process, including a list of Certificate Authorities, and the templates list used by the Certificate Authorities. Then, after the wizard has this information it will ask you to answer a few questions about the characteristics of the certificate you want to enroll. At the end of the wizard the certificate request is created on the client and sent to the CA. The important thing to note here is that communication between the client and the CA is done using DCOM, and requires DCOM connectivity between the client and the CA.

Networking Policy Changes Required on the Forefront UAG Server to Make the Certificate Request

To enable DCOM and LDAP to the CA and domain controllers, you need to configure Forefront UAG security policy to enable LDAP traffic from the Forefront UAG server to the domain controllers, and DCOM traffic from the Forefront UAG server to the CAs.

Fortunately, the default security policy already allows LDAP connectivity from the Forefront UAG server to the domain controllers. This leaves us with the task of enabling DCOM traffic between the Forefront UAG server and the CA.

DCOM traffic starts by using TCP port 135, which is the RPC endpoint mapper’s port. Later it switches to a different TCP port based on negotiations that took place after connecting to the endpoint mapper. The port number it switches to is given to the DCOM client (the Certificates MMC in this example) during the initial conversation on TCP port 135. This conversation is encrypted by default. Since the traffic is encrypted, we can’t tell in advance what port will be used to enable traffic from the Forefront UAG server to the CA.

Another important point is that by default the Forefront UAG server looks into all conversations on TCP port 135 and when it sees encrypted traffic it blocks the communication altogether (before the two sides have a chance to negotiate the port that would be subsequently used).

Therefore, what we need to do is:

  • Enable all traffic from the Forefront UAG Server to the CA
  • Tell the Forefront UAG server not to terminate encrypted communications between the Forefront UAG server and the CA on TCP port 135

Enabling Certificate Enrollment from the TMG console on the Forefront UAG Server – Step by Step

The Forefront UAG server includes Forefront TMG server technology. TMG acts as a firewall for the UAG machine. What we want to do is open the TMG management console and enable DCOM traffic and encrypted conversations on TCP port 135.

Important note In general, we recommend that you do not change the TMG configuration (customize TMG firewall rules, create new firewall rules) from within the TMG management console. The exceptions are when you use documented procedures such as this one. The reason for this is that UAG behavior relies on certain elements configured in TMG, and changing them might make Forefront UAG server act in an unpredictable way.

Step 1:

The first thing you need to do is open the TMG management console by starting the Forefront TMG Management application.

clip_image004Figure 2

Step 2:

Next, left click on the Firewall Policy node and view the list of existing Firewall rules. Then, left click on the first rule named Publishing Rule::Anchor::Begin

clip_image006  Figure 3

Step 3:

The next step is to create a new Access Rule. To do this, right click on the Firewall Policy node in the left pane of the console, point to New and click on Access Rule.

clip_image008Figure 4

 

a. On the Welcome to the New Access Rule Wizard page, name the rule Allow all protocols from localhost to CA.

clip_image010
Figure 5

b. On the Rule Action page, configure the rule as an Allow rule.

clip_image012
Figure 6

c. On the Protocols page, configure the rule to allow All outbound traffic.

clip_image014
Figure 7

d. On the Access Rule Sources page, set the source Network as the Local Host Network.

clip_image016
Figure 8

e. On the Access Rule Destination page, create a new Computer Object for your CA and set that CA as the destination for the Access Rule. The Computer Object should contain the IP address of the CA.If there is more than one CA you should create multiple Computer Objects with different IP address and add all of them.

clip_image018
Figure 9

f. On the User Sets page, accept the default, which is All Users.

clip_image020
Figure 10

Step 4:

Next thing you need to do is enable encrypted traffic on TCP port 135. To do that you need to open the System Policy Rules. Right click on Firewall Policy in the left pane of the console, point to All Tasks and click Edit System Policy.

clip_image022
Figure 11

a. On the left side of the System Policy Editor, navigate to the Authentication Services Configuration Group and click on Active Directory.

b. On the General tab, remove the checkmark from the Enforce strict RPC compliance checkbox.

clip_image024
Figure 12

Step 5:

Click the Apply button to save the changes to the TMG configuration. We recommend that you export the configuration before applying the changes

clip_image026
Figure 13

After the changes are applied, it should take only a few moments for the Forefront UAG server to apply the new configuration. Once the new settings are applied you should have no problems requesting or renewing certificates from the Forefront UAG server.

That’s about it for today!

Thanks -

 

Author:
Ben Bernstein, Senior Program Manager, Forefront Edge

Reviewer:
Tom Shinder, Technical Writer, Forefront Edge