PowerShell Remoting Kerberos Double Hop Solved Securely

The struggle is real.

Are you facing issues with PowerShell remoting and credentials? You remote into your jump box, but then any remoting beyond there gets a big red ACCESS DENIED. Maybe you’ve tried CredSSP, but people say that isn’t safe. Read today’s post for a completely legit, secure, safe, and easy way to enable Kerberos double hop for PowerShell remoting.

The Problem

image

It’s a tale as old as time:

  • ServerA talks to ServerB
  • ServerB talks to ServerC
  • Access denied!

You would have better luck asking a cheerleader to the prom. We call this the kerberos double hop. Yeah, it’s like a dance.

The struggle is real. Just check out this forum post on PowerShell.org from last month. After many years of PowerShell remoting we are still searching for a secure method of passing credentials to that elusive ServerC.

Neo vs. the Architect

Many have come before you. Let’s look at some of the popular solutions for Kerberos double hop in PowerShell remoting:

Method Pros Cons Links
Grant access to the ServerC resource for the ServerB computer object. n/a It works for some other double hop use cases, but not PowerShell remoting.
CredSSP It works! It’s not totally secure.Requires configuration of the client and server roles. Accidental Sabotage: Beware of CredSSP [MS-CSSP]: Credential Security Support Provider (CredSSP) Protocol - 5.1 Security Considerations for Implementers
PSSessionConfiguration using RunAs It works! Requires PSSessionConfiguration and RunAs credential password maintenance on every ServerB. Another solution to multi-hop PowerShell remoting
JEA – Just Enough Administration It works!When using a virtual account or group managed service account (gMSA) there is no password maintenance. Requires WMF 5.0 or above.Requires role capability module and PSSessionConfiguration on every ServerB. Just Enough Administration
Pass fresh credentials inside the Invoke-Command scriptblock ($using:cred) It works!No special server configuration.No Windows Server 2012 requirement. Awkward code technique.Easiest using WMF 3.0 or above. Also possible by using the WMF 2.0 syntax for passing arguments to a remote session. See the very bottom of this article for a code sample.
Kerberos Constrained Delegation It may work if you can figure it out.No special coding required. Moves authority from the back-end resource owner to the front-end application owner.Limited to one domain; cannot cross a trust.Requires domain administrative rights to update objects and SPNs.Not documented for PowerShell remoting.
Kerberos Unconstrained Delegation It works!No special coding required. It’s not totally secure.Allows delegation of credentials with no control over where they get used.
Resource-Based Kerberos Constrained Delegation No stored credentials.Easy to configure.Works across domains and forests.No special coding required. Requires Windows Server 2012 and above for most servers involved.See KB2665790 for 2008 R2 support.Support for limited commands running as SYSTEM.Does not support WinRM. See links at the bottom of the article.

 

Resource-Based Kerberos Constrained Delegation

Every release of Windows Server packs tons of new features, many that do not make big headlines. In my opinion this solution has been around four years now, and no one has uncovered its use for PowerShell remoting. I have researched this topic thoroughly, and I have not found anyone else online documenting this feature as a solution for PowerShell remoting.

Windows Server 2000 included Kerberos delegation (unconstrained). This allowed ServerB to delegate credentials anywhere else in the domain. Not good.

Windows Server 2003 modified this concept to constrained delegation, limiting delegation from ServerB to only designated service principal names (SPNs) on ServerC. This was much better, and it is common practice today. Many have struggled to get this working for PowerShell remoting, since it is not a documented solution.

Windows Server 2012 simplified the design by instead configuring the delegation on the computer object of ServerC, called resource-based delegation, specifying from whom it will receive delegated credentials. The same attribute can be set for user accounts and service accounts as well.

Resource-based Kerberos constrained delegation requires Windows Server 2012 or above for the servers involved, including at least one 2012 domain controller in each related domain. I am not going to include all of the details in this post, because the technology is well-documented. Read the links at the bottom of the article for all the particulars.

Show me some `Shell

This code for setting up the permissions requires a Windows box with the Windows Server 2012 Active Directory PowerShell RSAT available.

 PS C:\> Add-WindowsFeature RSAT-AD-PowerShell

PS C:\> Import-Module ActiveDirectory

PS C:\> Get-Command -ParameterName PrincipalsAllowedToDelegateToAccount

CommandType Name                 ModuleName     
----------- ----                 ----------     
Cmdlet      New-ADComputer       ActiveDirectory
Cmdlet      New-ADServiceAccount ActiveDirectory
Cmdlet      New-ADUser           ActiveDirectory
Cmdlet      Set-ADComputer       ActiveDirectory
Cmdlet      Set-ADServiceAccount ActiveDirectory
Cmdlet      Set-ADUser           ActiveDirectory

Notice that some of the Active Directory cmdlets for Windows Server 2012 and above include a new parameter PrincipalsAllowedToDelegateToAccount. This parameter sets the Active Directory object attribute msDS-AllowedToActOnBehalfOfOtherIdentity. That attribute actually holds an access control list (ACL) determining who has permissions to delegate credentials to ServerC.

 # Set up variables for reuse            
$ServerA = $env:COMPUTERNAME            
$ServerB = Get-ADComputer -Identity ServerB            
$ServerC = Get-ADComputer -Identity ServerC            

# Notice the StartName property of the WinRM Service: NT AUTHORITY\NetworkService            
# This looks like the ServerB computer account when accessing other servers over the network.            
Get-WmiObject Win32_Service -Filter 'Name="winrm"' -ComputerName $ServerB.name | fl *

The WinRM service by default runs as the NetworkService account. Therefore, we will allow ServerC to receive the computer object of ServerB for delegation.

 # Grant resource-based Kerberos constrained delegation            
Set-ADComputer -Identity $ServerC -PrincipalsAllowedToDelegateToAccount $ServerB            
            
# Check the value of the attribute directly            
$x = Get-ADComputer -Identity $ServerC -Properties msDS-AllowedToActOnBehalfOfOtherIdentity            
$x.'msDS-AllowedToActOnBehalfOfOtherIdentity'.Access            
            
# Check the value of the attribute indirectly            
Get-ADComputer -Identity $ServerC -Properties PrincipalsAllowedToDelegateToAccount

The output of the ACL Access property shows a simple access control entry (ACE) for ServerB to delegate credentials to ServerC:

 ActiveDirectoryRights : GenericAll
InheritanceType       : None
ObjectType            : 00000000-0000-0000-0000-000000000000
InheritedObjectType   : 00000000-0000-0000-0000-000000000000
ObjectFlags           : None
AccessControlType     : Allow
IdentityReference     : CONTOSO\ServerB$
IsInherited           : False
InheritanceFlags      : None
PropagationFlags      : None

Here is the one snag. The KDC has a 15 min SPN negative cache. If ServerB has already tried to talk to ServerC, then there is a negative cache entry. You need to clear the cache on ServerB using one of the following techniques:

  1. klist purge -li 0x3e7 (preferred and fastest method)
  2. Wait 15 minutes for the cache to clear automatically.
  3. Reboot ServerB.
 Invoke-Command -ComputerName $ServerB.Name -Credential $cred -ScriptBlock {            
    klist purge -li 0x3e7            
}

or

 Restart-Computer $ServerB.Name -Force -Wait -For WinRM

Once that step is complete we can successfully run code like this from ServerA through ServerB to ServerC:

 # Capture a credential            
$cred = Get-Credential Contoso\Alice            
            
# Test kerberos double hop            
Invoke-Command -ComputerName $ServerB.Name -Credential $cred -ScriptBlock {            
    Test-Path \\$($using:ServerC.Name)\C$            
    Get-Process lsass -ComputerName $($using:ServerC.Name)            
    Get-EventLog -LogName System -Newest 3 -ComputerName $($using:ServerC.Name)            
}

Note that the $using variable prefix allows ServerB to reference the $ServerC variable that lives in memory on ServerA. This makes the code entirely flexible. Just modify the $ServerB and $ServerC variables above with the computer names you want to use. Read more about $using in about_Remote_Variables.

You may want to allow multiple servers to delegate credentials to ServerC. In that case, set the parameter to an array of computer or user objects like this:

 # Set up variables for each server            
$ServerB1 = Get-ADComputer -Identity ServerB1            
$ServerB2 = Get-ADComputer -Identity ServerB2            
$ServerB3 = Get-ADComputer -Identity ServerB3            
$ServerC  = Get-ADComputer -Identity ServerC            
            
# Grant resource-based Kerberos constrained delegation            
Set-ADComputer -Identity $ServerC `
    -PrincipalsAllowedToDelegateToAccount @($ServerB1,$ServerB2,$ServerB3)

According to this whitepaper include the domain controller FQDN in the Server parameter of the Get-ADComputer command to make it work across domains:

 # For ServerC in Contoso domain and ServerB in other domain            
$ServerB = Get-ADComputer -Identity ServerB -Server dc1.alpineskihouse.com            
$ServerC = Get-ADComputer -Identity ServerC            
Set-ADComputer -Identity $ServerC -PrincipalsAllowedToDelegateToAccount $ServerB

To undo the configuration, simply reset ServerC’s attribute to null.

 Set-ADComputer -Identity $ServerC -PrincipalsAllowedToDelegateToAccount $null

A Practical Example

You have a jump box server that you connect to for daily administration. From that server you access all the other servers in your environment. The jump box server would be ServerB, so all the other servers in your environment (ServerC) would need ServerB allowed. Here is a code sample to query servers from an OU and set them all for resource-based Kerberos constrained delegation:

 $ServerB = Get-ADComputer -Identity JumpBox            
$Servers = Get-ADComputer -Filter {Name -ne $ServerB.Name} `
    -SearchBase 'OU=Servers,OU=NA,DC=contoso,DC=com' -SearchScope Subtree            
ForEach ($ServerC in $Servers) {            
    Set-ADComputer -Identity $ServerC -PrincipalsAllowedToDelegateToAccount $ServerB            
}

Summary

Kerberos double hop for PowerShell remoting can now be solved with one simple cmdlet:

 $ServerB = Get-ADComputer -Identity ServerB            
$ServerC = Get-ADComputer -Identity ServerC            
Set-ADComputer -Identity $ServerC -PrincipalsAllowedToDelegateToAccount $ServerB
# Then on ServerB: KLIST PURGE -LI 0x3e7

The benefits are many:

  • No PowerShell code modification.
  • No more SPNs for constrained delegation!
  • Credentials are not stored on ServerB.
  • Multiple domains and forests supported across trusts.
  • Easier setup and administration.
  • ServerA can now talk to ServerC through ServerB.

Once again the world is a happy place. Now go try it for yourself. Use the comment area below for feedback. Let me know how it goes.

What's New in Kerberos Authentication
Resource-based constrained delegation across domains and forest
https://technet.microsoft.com/en-us/library/hh831747.aspx

How Windows Server 2012 Eases the Pain of Kerberos Constrained Delegation, Part 1 https://windowsitpro.com/security/how-windows-server-2012-eases-pain-kerberos-constrained-delegation-part-1
"Constrained delegation in Server 2012 introduces the concept of controlling delegation of service tickets using a security descriptor rather than an allow list of SPNs. This change simplifies delegation by enabling the resource to determine which security principals are allowed to request tickets on behalf of another user."
"Resource-based constrained delegation functions correctly regardless of domain functional level and number of domain controllers (DCs) running a version of Windows Server prior to Server 2012, provided you have at least one Server 2012 DC in the same domain as the front-end server and one Server 2012 DC in the domain hosting the back-end server."

How Windows Server 2012 Eases the Pain of Kerberos Constrained Delegation, Part 2 https://windowsitpro.com/security/how-windows-server-2012-eases-pain-kerberos-constrained-delegation-part-2

Understanding Kerberos Constrained Delegation for Azure Active Directory Application Proxy Deployments with Integrated Windows Authentication https://aka.ms/kcdpaper

[MS-ADA2]: Active Directory Schema Attributes M
2.210 Attribute msDS-AllowedToActOnBehalfOfOtherIdentity
https://msdn.microsoft.com/en-us/library/hh554126.aspx
"This attribute is used for access checks to determine if a requestor has permission to act on the behalf of other identities to services running as this account."

[MS-SFU]: Kerberos Protocol Extensions: Service for User and Constrained Delegation Protocol
1.3.2 S4U2proxy
https://msdn.microsoft.com/en-us/library/cc246079.aspx

Resource Based Kerberos Constrained Delegation https://blog.kloud.com.au/2013/07/11/kerberos-constrained-delegation/

Remote Administration Without Constrained Delegation Using PrincipalsAllowedToDelegateToAccount https://blogs.msdn.microsoft.com/taylorb/2012/11/06/remote-administration-without-constrained-delegation-using-principalsallowedtodelegatetoaccount/

 

image

When All Else Fails... $using:cred

If you still have no Windows Server 2012 domain controllers in your environment then you can use this technique:

 # This works without delegation, passing fresh creds            
# Note $Using:Cred in nested request            
$cred = Get-Credential Contoso\Administrator            
Invoke-Command -ComputerName ServerB -Credential $cred -ScriptBlock {            
    hostname            
    Invoke-Command -ComputerName ServerC -Credential $Using:cred -ScriptBlock {hostname}            
}

This works, because the $using:cred passes a fresh copy of the credential variable into the remoting session without storing it anywhere. You could also swap out the Invoke-Command cmdlets with Enter-PSSession. The only requirement is WMF 3.0 or above on your servers.

Edit 8/31/16: Thanks to fellow PFE Martin Schvartzman for the KLIST PURGE syntax!
Edit 9/26/16: Added a code sample for the $using:cred alternative.
Edit 12/5/16: After much research internally we have concluded that this technique does not support double-hop for WinRM-based commands (Invoke-Command, Enter-PSSession, Get-CimInstance, etc.). I plan to release more details on this later.

Edit 12/13/16: The product group has turned this article into documentation here: https://msdn.microsoft.com/en-us/powershell/scripting/setup/ps-remoting-second-hop. They included some additional links you may want to review.
Edit 4/10/17: Minor tweaks to the JEA and Resource-Based Kerberos Constrained Delegation items in the matrix.