I was troubleshooting PIC connectivity with a customer recently and we ran into an issue when following the steps in Scott Oseychik's blog post on changing the cipher suite order (http://blogs.msdn.com/b/scottos/archive/2009/04/03/resolved-ocs-2007-r2-pic-fails-against-aol.aspx). I've used the steps outlined in the blog post on Windows 2008 before, but never on Windows 2008 R2. It just so happened that this customer had a Windows 2008 R2 Edge server. When we went to paste the newly ordered cipher suite back into the Group Policy Editor, we noticed that the whole string didn't paste back in:
The string got cut off close to the end, but it was missing the last few ciphers. It appeared that the text box could only accept so many characters and in doing a little research, that's exactly the case. Scrolling down in the help section on the right, you will find the instructions for editing the cipher suite order:
Step 6 is the key. The maximum length is 1023 characters. So the text box will only accept 1023 characters, but I never ran into this issue on Windows 2008? Digging a little deeper it appears that there were some additional ciphers added between Windows 2008 and 2008 R2. In Windows 2008 the list of ciphers is 831 characters. In Windows 2008 R2 the list of ciphers is 1071 characters. So we can't use the Group Policy Editor to reorder the list of cipher suites.
The workaround that I came up with is to modify the registry key that this group policy object updates. The standard warnings about modifying the registry apply...be careful when editing the registry! The key that the group policy object updates is:
Type = REG_SZ
Name = Functions
Data = TLS_RSA_WITH_RC4_128_MD5,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,...
The "Functions" value shouldn't already exist, so you will probably need to create a new "String Value". Take your newly ordered list of ciphers and paste them into the value data field. This time the whole list will paste in correctly. Reboot your server and the new cipher suite order should be in effect.