LINUX Certificate Enrollment and Automated Renewal Using NDES (Updated)

(APR 2018)  NDES continues to evolve (slowly) as everyone embraces the deprecation of SHA-1.  This post has been updated to reflect those changes and the impact on NDES.  Note the difference between the cryptography used in NDES / SCEP communications versus the cryptography in the certificates provided through NDES / SCEP.  For more detail on the reason for an update, check here.

We continue the customer conversation of managing heterogeneous environments... The conversations often revolve around “I know how to do this in (Windows / Linux)… how do I do this for both”.  In most cases, it is a matter of understanding risks, benefits and capabilities of the available technology.  Today we cover issuing and renewal of certificates for Linux systems issued from a Windows based Certificate Authority.

Certificates are needed for many reasons.  The example use case presented here would be to provide secure communications for an infrastructure agent like System Center Configuration Manager or to secure a web site running on Apache.  In those scenarios the certificate expiring will cause a service outage.  The solution would be to have the deployed certificates renewed (replaced) prior to expiration automatically.

Many Windows administrators are familiar with the native capability to auto-enroll domain joined systems with an enterprise Certificate Authority (CA). In simple terms, the process involves implementing an Active Directory integrated CA, configuring templates and enabling group policy to automatically enroll clients. (See: https://technet.microsoft.com/en-us/library/cc731522.aspx )

The reality is that implementing a proper CA hierarchy can be a lengthy task depending upon your organization’s size, security needs, requirements and policies.  (See: https://technet.microsoft.com/en-us/library/cc754057.aspx )  On a complexity scale (easy to difficult), it would be a moderate task to implement primarily because of the planning and design that should go into it.

This article is based on the existence of an already implemented Enterprise CA with Windows clients already auto-enrolled.

With your Windows clients taken care of the next question would be, “how do I do this with my Linux clients?” Auto-enrollment of domain joined Windows clients is made easier because the clients can already securely authenticate and be authorized by the enterprise CA. Non-domain joined systems like Linux clients and network devices require intervention to tell the enterprise CA, “I trust this device and authorize it to be enrolled”. To solve this, we will discuss two methods to obtain certificates; manual enrollment and automated using solutions like NDES.

MANUAL ENROLLMENT

Certificates can be obtained manually from a CA; often through web enrollment. The administrator authenticates to the CA on behalf of the device or system and generates (or submits) a certificate request.  While manual enrollment works perfectly, it does not allow to scale to large numbers of systems or address management of the certificates when they near expiration.

AUTOMATED ENROLLMENT USING NETWORK DEVICE ENROLLMENT SERVICE

The Network Device Enrollment Service (NDES) allows software on routers and other network devices running without domain credentials to obtain certificates based on the Simple Certificate Enrollment Protocol (SCEP) (See https://technet.microsoft.com/en-us/library/hh831498(v=ws.11).aspx )

The Network Device Enrollment Service performs the following functions:

  • Generates and provides one-time enrollment passwords to administrators.
  • Submits SCEP enrollment requests to the CA.
  • Retrieves enrolled certificates from the CA and forwards them to the network device.

SIMPLE CERTIFICATE ENROLLMENT PROTOCOL

(APR 2018) SCEP, as a protocol, remains in draft with the IETF.  For a number of years it garnered little attention.  The industry continues to heavily embrace its use though. The SCEP draft has been updated more frequently over the past few years with the most recent update as of MAR 2018 (see https://tools.ietf.org/html/draft-gutmann-scep-10 )  This of course makes SCEP security a moving target for all vendors and the open source community.  Hardware vendors for network devices will need to update firmware and software vendors as well as the open source community will need to update code to accommodate the changes.

SECURITY ADVISIORY

In 2012 a security advisory was posted regarding the vulnerability of SCEP. (See https://www.kb.cert.org/vuls/id/971035 ) The advisory states that Simple Certificate Enrollment Protocol (SCEP) does not strongly authenticate certificate requests made by users or devices. SCEP was designed for use in closed environments and not for the purpose of mobile device management. This solution may not meet your security requirements and warrant a thorough review. The use case example is for protected Linux systems, not fully untrusted mobile devices. There are a number of mitigations may allow this solution to meet your security requirements.

The above advisory is also subject to change as the spec is updated and the changes begin to be implemented.

IMPLEMENTATION OVERVIEW

The scenario outlined here should be executed in a lab allowing you to build a functional solution that you can then tune to meet your organization requirements.  The solution has been tested with number of Linux distributions.

The lab scenario setup for this article utilizes the following platforms:

  • Windows Server 2012 R2 Enterprise Certificate Authority in  a two tier hierarchy (offline root, online enterprise issuing subordinate) (APR 18) Tested with Windows Server 2016 in a two tier hierarchy.  The CA hierarchy should be configured to issue SHA2 certificates (SHA256 or higher). For this lab, the subordinate issuing authority is configured to issue SHA384 certificates.
  • CentOS 6.8 - 7.4

Within the LINUX environment, execute the actions with elevated privileges (via sudo or su). Ensure the hostname of your Linux system is the fully qualified domain name of the host. Example: linux1.contoso.corp The FQDN will be used in the certificate submission as the DNS name field for the certificate.

IMPLEMENTATION STEPS

  1. SETUP NDES FOR ACTIVE DIRECTORY CERTIFICATE SERVICES

The first step in the process is to enable your environment to support NDES. Follow the implementation guidance in the NDES WIKI https://social.technet.microsoft.com/wiki/contents/articles/9063.network-device-enrollment-service-ndes-in-active-directory-certificate-services-ad-cs.aspx

 

  1. CONFIRM THE DEFAULT TEMPLATES EXIST

Once NDES is deployed, confirm that the IPSEC Intermediate (Offline) template is deployed. The out-of-box template cryptography settings will likely be 1024 bit SHA 1. (APR 2018) The minimum key size should be updated to 2048.  The request hash can remain at SHA1 for testing or be moved to a higher value if all clients and network devices serviced by this NDES server support it.  For the purpose of the example it should be moved to SHA256

Note: Once your test environment is functional, you can go back and create new templates that meet the organizational needs.  Review the NDES wiki for the section on updating the registry to point to the new template name.

 

  1. INSTALL THE SSCEP CLIENT

The Fedora hosted DogTag project is a collection of Open Source Certificate Authority technologies. One of the components is Simple SCEP (SSCEP). SSCEP can be found in the Extra Packages for Linux (EPEL) or can be pulled from GIT (see https://github.com/certnanny/sscep ).  To install SSCEP from EPEL, first add the epel repository and then install the client.  On the Linux system, execute the following:

 yum install epel-release
yum install sscep

 

  1. INSTALL THE CA CHAIN ON THE LINUX CLIENT

The Linux endpoint requires the CA hierarchy chain to being SCEP enrollment. In the following syntax example:

  • Replace “ContosoCA” with a simple name that reflects your CA (to indicate where the file came from).
  • Replace “NDES.CONTOSO.CORP” with the FQDN that reflects the CA running NDES’s fully qualified domain name.

Execute the following command:

 sscep getca -F sha1 -c /etc/pki/ca-trust/source/anchors/contosoCA.crt -u https://ndes.contoso.corp/certsrv/mscep/mscep.dll/pkiclient.exe?

Note the use of SHA 1.   This is specifying use for the fingerprint algorithm.  For more detail on why the SHA1 can be used with NDES / SCEP, see section 8.8 of the current draft (See https://tools.ietf.org/html/draft-gutmann-scep-10).

In a two tier hierarchy, the command will pull down four certificate files.

  • contosoCA.crt-0                               The certificate for the issuing subordinate CA root Registration Authority (RA)
  • contosoCA.crt-1                                The certificate for the subordinate CA root CEP Encryption (CE)
  • contosoCA.crt-2                               The certificate for the CA root.
  • contosoCA.crt-3                               The certificate for the issuing subordinate CA root.

For an understanding of the LINUX system PKI file structure, see https://www.mankier.com/8/update-ca-trust   If you are running CentOS 6, you may have to enable your CA trust store to be updated by running:

 update-ca-trust enable

Next, execute the following command:

 update-ca-trust extract

This will add the downloaded CA chain to the Linux system’s trusted CA list.

 

  1. GENERATE A CSR FOR THE INITIAL ENROLLMENT

Log into the issuing CA server website with the required credentials of a certificate administrator. The example URL is https://ndes.contoso.corp/certsrv/mscep_admin

Obtain the NDES enrollment challenge password. In the NDES WIKI example, the password is 302034237AE9A1D1 (See the picture) . By default, the enrollment challenge password is for one time use and has a time expiration. This can be changed to allow longer time periods and multi-device use. Those changes however may not meet your organization security requirements.

On the LINUX client, run the command mkrequest to generate a Certificate Signing Request (CSR). Insert the issued enrollment challenge password. Example:

 mkrequest -dns $(hostname) 302034237AE9A1D1

The mkrequest script will generate two files by default. The two files are the CSR and the certificate private key. (local.csr and local.key).

 

  1. EXECUTE THE NDES INITIAL ENROLLMENT

The next step is to execute the actual NDES enrollment using the SCEP client. An example enrollment would be:

 sscep enroll -E 3des -S sha1 -c /etc/pki/ca-trust/source/anchors/contosoCA.crt-0 -e /etc/pki/ca-trust/source/anchors/contosoCA.crt-1 -k local.key -r local.csr -l $(hostname).crt -u 'https://ndes.contoso.corp/certsrv/mscep/mscep.dll/pkiclient.exe?' -d -v

This command will execute a certificate enrollment using the CSR, the certificate private key and the one time enrollment password.  Again, note the -E and -S switches.  -E is the PKCS#7 encryption algorithm, and the -S is the signature algorithm.  Again, refer to section 8.8 of the current draft.  This is the encryption used with NDES, not the encryption settings for the certificate that the client will receive.  That is specified in the template and will be SHA256 or higher (based on your template).

 

CERTIFICATE RENEWAL

Certificate renewal is a very similar process. However instead of using an enrollment challenge password, the enrollment uses the private key of the existing certificate and the valid existing certificate as the authorization credential.

  1. GENERATE A CSR FOR THE RENEWAL

For a renewal, no password is required. You will be using a combination of the old key, new key, old certificate and a new CSR. If you are executing this command in the same directory containing the original private key, it must be renamed so that it is not overwritten. Execute the mkrequest script including just the DNS name for the certificate. Example:

 mkrequest -dns $(hostname)

 

  1. EXECUTE THE RENEWAL (REPLACEMENT CERTIFICATE)

For a renewal, you will need the existing certificate, existing private key, the new CSR and the new private key.

 sscep enroll -E 3des -S sha1 -c /etc/pki/ca-trust/source/anchors/contosoCA.crt-0 -e /etc/pki/ca-trust/source/anchors/contosoCA.crt-1 -k local.key -K existinglocal.key -O existing.crt -r local.csr -l new.crt -u 'https://ndes.contoso.corp/certsrv/mscep/mscep.dll/pkiclient.exe?' -d -v

AUTOMATING THE SOLUTION

The primary objective is to automate certificate enrollment and renewal. This can be accomplished a number of ways depending on what existing capabilities your organization may already have. The automation can be:

  • A simple CRON job executing a script
  • PowerShell Desired State Configuration (DSC) for Linux
  • A Puppet module

The following shell script demonstrates what the workflow for initial enrollment and renewal would look like.
(Updated APR 2018)

 

 #!/bin/bash
# Linux Certificate Enrollment Using NDES and SCEP
##################################################################
# This script is provided as an example for illustration only,
# without warranty either expressed or implied, including, but not
# limited to, the implied warranties of merchantability and/or
# fitness for a particular purpose.
##################################################################
#
# Insert the NDES Enrollment Password here.
ndeskey=ABC123DEF456ABC789DEF012ABC
# Insert the NDES server FQDN
ndesserver=ndes.contoso.corp
# CA Filename
cafilename=contoso
# Number of days ahead of cert expiration to renew
warning_days=45
###############################################################################
#EDIT ONLY THE VALUES ABOVE
###############################################################################
shopt -s -o nounset
# Create a log file directory
if [ ! -d /var/log/pki/ndes ]; then
    mkdir -p /var/log/pki/ndes/
fi
DTG=$(date +%Y%m%d%H%M)
LOGFILE=/var/log/pki/ndes/ndes-enrollment$DTG.log
if [ ! -f $LOGFILE ]; then
    touch $LOGFILE
fi
###############################################################################
# LOG FUNCTION TO WRITE LOCAL LOG FILE AND TO /VAR/LOG/MESSAGES
###############################################################################
writelog() {
    echo ${*} 2>&1 >> $LOGFILE
    if [ -f /bin/logger ] 
    then 
       logger ${*}
    fi 
}
###############################################################################
writelog  "sscep: Checking for required packages"
if rpm -qa | grep epel-release  2>&1 > /dev/null;
        then
        writelog "sscep: epel-release is installed."
else
    yum -y install epel-release 2>&1 >> $LOGFILE
fi
if rpm -qa | grep sscep  2>&1 > /dev/null;
        then
        writelog "sscep: sscep is installed."
else
    yum -y install sscep 2>&1 >> $LOGFILE
fi

MKREQUESTSCRIPT=/usr/bin/mkrequest
sed -i 's/KEYBITS=1024/KEYBITS=2048/g' $MKREQUESTSCRIPT
if ! grep -q "string_mask = nombstr" $MKREQUESTSCRIPT ; then
       sed -i '/\[ req \]/a string_mask = nombstr' $MKREQUESTSCRIPT
fi

if [ ! -f /etc/pki/tls/certs/$(hostname).crt ] ; then
        writelog "sscep: $(hostname) does not have an existing certificate.  Executing initial enrollment."
        writelog "sscep: Requesting CA certificate chain as x509 .crt files."
        sscep getca -F sha1 -c /etc/pki/ca-trust/source/anchors/${cafilename}CA.crt -u https://${ndesserver}/certsrv/mscep/mscep.dll/pkiclient.exe? 2>&1 >> $LOGFILE
        writelog "sscep: Adding the CA chain to the host CA trusted chain."
        update-ca-trust enable 2>&1 >> $LOGFILE
        update-ca-trust extract 2>&1 >> $LOGFILE
        writelog "sscep: Generating CSR."
        mkrequest -dns $(hostname) $ndeskey 2>&1 >> $LOGFILE
        writelog "sscep: Moving CSR and new private key." 
        mv local.key /etc/pki/tls/private/$(hostname).key
        mv local.csr /etc/pki/tls/private/$(hostname).csr
        writelog "sscep: Executing enrollment using NDES key."
        sscep enroll -E 3des -S sha1 -c /etc/pki/ca-trust/source/anchors/${cafilename}CA.crt-0 -e /etc/pki/ca-trust/source/anchors/${cafilename}CA.crt-1 -k /etc/pki/tls/private/$(hostname).key -r /etc/pki/tls/private/local.csr -l /etc/pki/tls/private/$(hostname).crt -u https://${ndesserver}/certsrv/mscep/mscep.dll/pkiclient.exe? -d -v 2>&1 >> $LOGFILE
        writelog "sscep: Checking generated certificate status."
        CERTSTATUS=$(openssl verify /etc/pki/tls/private/$(hostname).crt | cut -d: -f2)
        if [ $CERTSTATUS == "OK" ]; then
                writelog "sscep: Certificate status is: " $CERTSTATUS
                writelog "sscep: Copying new certificate to /etc/pki/tls/certs/$(hostname).crt" 
                cp /etc/pki/tls/private/$(hostname).crt /etc/pki/tls/certs/$(hostname).crt 2>&1 >> $LOGFILE
        else
                writelog "sscep: The certificate is not valid per $CERTSTATUS"
        fi
else
        output=$(openssl x509 -in /etc/pki/tls/certs/$(hostname).crt -noout -subject -dates 2>/dev/null)
        cert=$(echo $output | sed 's/.*CN=\(.*\).*not.*/\1/g')
        start_date=$(echo $output | sed 's/.*notBefore=\(.*\).*not.*/\1/g')
        end_date=$(echo $output | sed 's/.*notAfter=\(.*\)$/\1/g')
        start_epoch=$(date +%s -d "$start_date")
        end_epoch=$(date +%s -d "$end_date")
        epoch_now=$(date +%s)
        if [ "$start_epoch" -gt "$epoch_now" ]; then
                writelog "sscep: Certificate for [$cert] is not yet valid"
                writelog $output
        fi
        seconds_to_expire=$(($end_epoch - $epoch_now))
        days_to_expire=$(($seconds_to_expire / 86400))
        writelog "sscep: Days to expiry: ($days_to_expire)"
        warning_seconds=$((86400 * $warning_days))
        if [ "$seconds_to_expire" -lt 0 ]; then
                writelog "sscep: Certificate [$cert] has expired.  Remove the certificate and rerun to start a new enrollment."
        else
                if [ "$seconds_to_expire" -lt "$warning_seconds" ]; then
                        writelog "sscep: Certificate [$cert] is soon to expire ($seconds_to_expire seconds)"
                        writelog "sscep: Existing certificate found for $(hostname).  Executing re-enrollment."
                        writelog "sscep: Backing up current private key."
                        mv -f /etc/pki/tls/private/$(hostname).crt /etc/pki/tls/private/$(hostname).crt.bak
                        mv -f /etc/pki/tls/private/$(hostname).key /etc/pki/tls/private/$(hostname).key.bak
                        writelog "sscep: Generating CSR."
                        mkrequest -dns $(hostname) 2>&1 >> $LOGFILE
                        writelog "sscep: Moving CSR and new private key."
                        mv -f local.key /etc/pki/tls/private/$(hostname).key
                        mv -f local.csr /etc/pki/tls/private/$(hostname).csr
                        writelog "sscep: Executing enrollment using existing certificate."
                        sscep enroll -E 3des -S sha1 -c /etc/pki/ca-trust/source/anchors/${cafilename}CA.crt-0 -e /etc/pki/ca-trust/source/anchors/${cafilename}CA.crt-1 -k /etc/pki/tls/private/$(hostname).key -K /etc/pki/tls/private/$(hostname).key.bak -O /etc/pki/tls/private/$(hostname).crt.bak -r /etc/pki/tls/private/$(hostname).csr -l /etc/pki/tls/private/$(hostname).crt -u https://${ndesserver}/certsrv/mscep/mscep.dll/pkiclient.exe? -d -v  2>&1 >> $LOGFILE
                        writelog "sscep: Checking generated certificate status."
                        CERTSTATUS=$(openssl verify /etc/pki/tls/private/$(hostname).crt | cut -d: -f2)
                        if [ $CERTSTATUS == "OK" ]; then
                                writelog "sscep: Certificate status is: " $CERTSTATUS
                                writelog "sscep: Copying new certificate to /etc/pki/tls/private/$(hostname).crt"
                                cp -f /etc/pki/tls/private/$(hostname).crt /etc/pki/tls/certs/$(hostname).crt
                        else
                                writelog "sscep: The certificate is not valid per $CERTSTATUS"
                                writelog "sscep: Restoring the backup existing certificate and keys."
                                mv -f /etc/pki/tls/private/$(hostname).crt.bak /etc/pki/tls/private/$(hostname).crt
                                mv -f /etc/pki/tls/private/$(hostname).key.bak /etc/pki/tls/private/$(hostname).key
                        fi
                fi
        fi
fi

A useful workflow would include:

  • Installation of the SSCEP client; if not already installed.
  • Check to see if a certificate exists; if not, execute initial enrollment.
  • If a certificate exists, check its expiration date. If it is within the defined renewal period, renew the certificate.
  • The logic would have to handle rotation of the private keys and matching certificates used for each subsequent renewal.

TUNING THE LAB TO MEET YOUR REQUIREMENTS

(APR 2018) At this time, SHA1 has been deprecated and the expected minimum key length is 2048. Hash algorithms should be SHA2 (SHA-256) or higher.

Changing the certificate length is easy. The /usr/bin/mkrequest file is actually a script. The default value of for KEYBITS=1024 can be changed to KEYBITS=2048. On the Windows CA, the CA template used for NDES would need to be updated to be set to 2048 bit as well.  The sample script above modifies the KEYBITS value during execution to update the value automatically.

In my previous version of this post, I outlined some of the cryptographic limitations of the SSCEP client.  Many of those have now been resolved as the clients have received a number of revision updates.

 

ALTERNATE OPTIONS

If the above solution does not meet your requirements, you might consider some of the other certificate solutions that are beyond the scope of this article. There are third party LINUX agents / add-ons that allow Linux clients to emulate Windows client capabilities to leverage group policy and certificate auto-enrollment. (For example, see https://www.centrify.com)

 

Q&A

Q:        My environment has hundreds of Linux systems that need certificates. How can I do this with each one requiring a unique enrollment challenge password?

A:        This is a conversation with your organization security official. NDES can be configured to provide a multi-use enrollment challenge password. Additionally the use time limit can be extended. Your organization has to evaluate the risk of that key being compromised and certificates being obtained by systems you do not trust.

 

Q:        Can I use this solution to deploy certificates to my mobile devices?

A:        NDES was designed for closed (internal trusted protected) network devices. Mobile devices should be managed with a Mobile Device Management (MDM) solution like Microsoft Intune. https://www.microsoft.com/en-us/cloud-platform/mobile-device-management

 

Q:        My organization determined that using the enrollment challenge password is not acceptable and I have not found another cost effective solution. What can I do to decrease the security risk?

A:        There are a number of options. One simple option would be to manually issue / install the initial certificate onto the Linux system but continue to configure it to use NDES for renewal. This ensures that you have installed certificates on fully trusted machines and the renewals will occur using a valid existing certificate and matching unique private key.

 

Q:        How do I test enrollment and renewal in the lab? I don’t want to wait a year or two until the certificate expires.

A:        In the lab, configure your deployed certificate template used by NDES to have a shorter validity period (4 hours for example).

 

Q:        Have I looked at CERTMONGER as an alternative to the SSCEP client?

A:        See https://blogs.technet.microsoft.com/jeffbutte/2018/04/05/linux-certificate-enrollment-and-automated-renewal-using-ndes-v2/

 

Q:        I am getting the following error:

 sscep: pkistatus: FAILURE
sscep: finding attribute failInfo
sscep: allocating 1 bytes for attribute
sscep: reason: Transaction not permitted or supported
sscep: illegal size of payload

On the issuing CA, I see Event 29 in the Application event log.

 The password in the certificate request cannot be verified. It may have been used already. Obtain a new password to submit with this request.

A:      This issue is most frequently caused by the version of OpenSSL that the system is running.  Since the creation of sscep, OpenSSL has undergone some significant revisions.  Recent security relevant updates to OpenSSL no longer allow UTF-8 fields added to the CSR.  Additional fields  like the SCEP password must be sent in PrintableString.  The resolution is to edit the /usr/bin/mkrequest file and add ‘string_mask = nombstr’ in the [req] section (Contributed by Todd Mote)