Troubleshooting your RMS Server and Group Membership.

I don't talk to much about the RMS 'server' side of things very often, pretty much because if you've set it up properly (the first time), and your domain is healthy, you won't run into problems. Most supportability issues stem from the client.

Anyways, I wanted to give you all some registry keys and settings that I use when trying to track down potential problems on an RMS server. The only time that I usually will get on an RMS server to track things down (once it is setup and provisioned), is when I get a call from someone who say's 'I am sending this RMS/IRM protected message to a group, and people in the group cannot open the message'.

Immediately my mind flips to group expansion problems, so I go through the normal Q&A with them.

Q. What type of group is this?
R. The reason I ask this is because we recommend that for cross domain scenarios you 'always' use Universal Distribution Groups. These are the only types of groups that will replicate the group membership on the GCs throughout the forest. Since RMS uses GC queries, and the GC that it connects to is not guaranteed (We grab the first 5 returned from an AD query, and round-robin them). So if someone tells me it is a domain local group, or a security group my initial thoughts go to, 'Well, the group membership is not replicated to the GC.'

Q. What domain/forest functional level are you using?
R. The reason I ask this (when I remember) is because if you are in Windows 2000 native forest functional level the RMS service account will not be able to resolve hidden group memberships. To fix this you simply add the rmsservice account to your BUILT-IN\Pre-Windows 2000 group membership. I also recently had a customer that did a setspn HTTPS/cname.domain.com on his RMS server and it resolved the issue, and it is probably for the same reason, but I'm not positive on that one (I'll need to research that one). (Thanks Joe).

If this is in fact a universal distribution group and everything else jives, then I'll need to find out if it is possibly a caching issue. Essentially the way that RMS works is that in order to keep RMS from pinging the life out of your GCs every single time someone in a group opens an RMS protected piece of content, we cache the group membership in memory, and in the database (What did you think the purpose of the DRMS_Directory... database was?). By default we cache group memberships for 12 hours, but this is adjustable. Let me tell you how to turn it off first:

Here is a .reg file that you can copy and paste into a .reg file on your RMS server, and double-click on it.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\DRMS\1.0\DirectoryServices]
"MinGc"=dword:00000000
"MaxGc"=dword:00000000
"PrincipalCacheMax"=dword:00000000
"PrincipalCacheExpireMinutes"=dword:00000000
"GroupIDCacheMax"=dword:00000000
"GroupIDCacheExpireMinutes"=dword:00000000
"GroupMembershipCacheMax"=dword:00000000
"GroupMembershipCacheExpireMinutes"=dword:00000000
"ContactMembersofGroupCacheMax"=dword:00000000
"ContactMembersofGroupCacheExpireMinutes"=dword:00000000

Let's go through these.

MinGC and MaxGC - As I said, we will by default take the first 5 GCs we get back from a query and round robin them. If you want to control this you can set the minimum and maximum GCs to start with using these values.

'TimeOutforGC' is how long you want to wait before letting the GC query timeout.

All of the cache values are how long in minutes you want to cache the item listed in memory.

Setting these to 0 turns them all off.

To turn off the DirectoryServices database caching, you should get the RMSToolkit, and connect to your SQL database using the RMSConfigEditor tool (or you can just use SQL Enterprise Manager). Connect to the DRMS_Config_... database, and then the DRMS_ClusterPolicies table. Find the UseDirectoryServicesCacheDatabase and set its value to 0.

You will need to do an IISReset after you change any settings in the registry or Dbase.

At this point I would try to repro the issue to see if the problem is with caching. If the problem still exists, I will have to get a debug trace (the big gun), to try to figure out whats going on. It would be to difficult to explain every single problem that you could face in the trace, but if you want to look at a trace and try to figure it out for yourself (check the MSMQ property bags, check the PL, check the GCs, check the results), hereis how you get a detailed dump from the server.

1. Open the the Web.config file that exists in the Certification and Licensing directories under C:\IntetPub\WWWRoot\_wmcs, and then add the following lines right after the closing </DRMS> tag section that is in the file:

<system.diagnostics>
<switches>
<add name="Microsoft Windows Rights Management Services-Global" value="4" />
<add name="Microsoft Windows Rights Management Services-TimeStamps" value="1" />
<add name="Microsoft Windows Rights Management Services-Indents" value="0" />
</switches>
<trace autoflush="false" indentsize="4"/>
</system.diagnostics>

2. Restart IIS.

3. After you collect the data that you need (using DebugView - making sure that Capture>Global Win32 is checked (do a https://www.live.com/ search for this tool)), remove the lines from the .config file that you added in step #1

4. Restart IIS.

Hopefully this information is useful to some people who want and need to dig into RMS a little more than the average RMS admin.

-Jason