The caveats of using Group Policy Preferences on Terminal Servers

 Note: this entry is about the Group Policy Preferences component and one aspect of it (which is resolved in later hotfixes or service packs) - for a more generic entry that describes troubleshooting slow logons in general see http://blogs.technet.com/b/instan/archive/2008/04/17/troubleshooting-the-intermittent-slow-logon-or-slow-startup.aspx.

The GPPref component is an extension to the Group Policy functionality on Windows systems that is available as an OOB release for Windows XP/Windows 2003/Vista RTM and that comes in-box on Windows 2008 and Windows 2008 R2.

GPPref is an extremely powerful addition to the GP arsenal that is available to domain admins to control settings on servers and workstations – when used on terminal servers however the following needs to be considered when designing the infrastructure.

One feature of the Group Policy Preferences client is the Group Policy History functionality; it allows the admin to tick a box in each GPPref-enabled policy that configures the machine to remove the setting on the machine whenever it is no longer being applied to it ( “Remove this item when it is no longer applied” ).

On a Windows 2008 Terminal Server, the data for Group Policy History is kept on disk under “c:\programdata\microsoft\group policy\history\” .

Underneath that folder we have one folder for each GPPref-enabled policy that has applied to the machine (“{GUID of GPO”} and underneath the GUID folder we have one subfolder with the SID for each user that has applied that policy while logged on to the server. Underneath the SID folder we then have from 1-5 subfolders depending on which settings are being applied in the policy ( \Preferences, \Folders, \Registry, \Shortcuts, \Drives, etc . )

For a workstation or a normal server this works beautifully. On a terminal server that is in a terminal server farm where you can have hundreds or thousands of users that log on to it and apply policy while logged on the scenario becomes more challenging.

The potential number of folders under “c:\programdata\microsoft\group policy\history\” is based on the following equation:

(#users that can log on to the TS)*(#of GPPRef-enabled policies that apply to those users)*(#of settings being applied to those users in each of those policies).

Example; let’s say you have 1000 users in your organization that are logging on to your TS farm and you have created 10 different GPO’s that modify GPPRef settings to add a shortcut, create a specific folder, change a registry setting and map a drive for each of the policies,

The total number of folders that will have been created on a TS in the farm when all users have eventually connected via RDP to that server is then 1000*10*4==40000 folders.

At each logon those 40000 folders will be enumerated for each user as a foreground refresh of Group Policy occurs – the same will happen during the periodical background refresh of Group Policy that occurs for every logged on user.

This will put some additional CPU strain on the system as the Gpsvc service that hosts the GPPref client will be quite busy during peak logon hours – this in turn can mean increased logon time for users during peak logon periods or when a lot of the logged on users happen to be applying Group Policy during a background refresh.

Note that the processing of the GPH folders has been changed in W2k3 with the GPPref roll-up fix that is available in http://support.microsoft.com/kb/974266 so that it only occurs during a background refresh.
That change does not apply to either Windows Server 2008 or Windows Server 2008 R2.
Note: The processing of the Group Policy History files should now be identical for W2k3, W2k8 and W2k8 R2 after you install the relevant GPPRef roll-up fixes for each or SP1 for Windows 2008 R2.
I.e. with the correct fixes in place, high CPU usage and slow logons related to the number of subfolders underneath the GPH folder should not happen.

If you are seeing high CPU usage or slow logons on a terminal server and the GPPref fixes that modify the GPH-processing are in place then that issue should be investigated separately as you may be encountering a new issue with similar symptoms but a different root cause.

Footnote: Since I wrote this post KB974266 has also been ported to Windows Server 2008 (See http://support.microsoft.com/kb/977983).

2nd Footnote: As Andrew points out in a comment below this has been ported to W2k8 R2 in SP1 - there is no RTM hotfix however. The logic has also been changed to simply skip the enumeration of the folders during both foreground and background GPO refresh if the server is determined to be in terminal services mode.

Additional information:

Group Policy Preferences Overview
http://www.microsoft.com/downloads/details.aspx?FamilyID=42e30e3f-6f01-4610-9d6e-f6e0fb7a0790&displaylang=en

Group Policy Preferences [what’s new in Win7 and Windows Server 2008 R2]
http://technet.microsoft.com/en-us/library/dd367850(WS.10).aspx

Microsoft's official Group Policy blog
http://blogs.technet.com/grouppolicy/archive/2008/03/04/gp-policy-vs-preference-vs-gp-preferences.aspx

GPP tags on the GP Team blog
http://blogs.technet.com/grouppolicy/archive/tags/gpp/default.aspx

Group Policy preferences client-side extension hotfix rollup for Windows Vista and Windows Server 2008
http://support.microsoft.com/kb/977983

Group Policy Preferences Client-Side Extension Hotfix Rollup [W2k3]
http://support.microsoft.com/kb/974266

Technet Edge: Group Policy Preferences
http://edge.technet.com/Media/Group-Policy-Preferences/

How Core Group Policy Works
http://technet.microsoft.com/zh-tw/library/cc784268(WS.10).aspx

Enabling Group Policy Preferences Debug Logging using the RSAT
http://blogs.technet.com/askds/archive/2008/07/18/enabling-group-policy-preferences-debug-logging-using-the-rsat.aspx

MKleef’s GPPref blog
http://blogs.technet.com/mkleef/