Understanding multi-domain DL update and delegate issues after application of Exchange 2003 SP2


EDIT 1/24/2008: Please see the Part 2 of this post which has additional information on how to resolve this problem.

After applying SP2 to an Exchange 2003 server, users may have problems updating Distribution list membership and delegating permissions to specific users. This problem is occurring because of a chain of events that occurred after applying specific hotfixes to both the client and the Exchange server.

The first issue was introduced with the application of the Outlook Post SP2 hotfix package in 913696 on the Outlook clients. When this patch is applied, clients could no longer assign delegates, whereas, they used to prior to the hotfix. The user would receive the following error when attempting to add delegate permissions:

The Delegates settings were not saved correctly. Unable to activate send-on-behalf-of list. You do not have sufficient permission to perform this operation on this object.

To elaborate on this a little more, the hotfix package contains an update to address an issue when you attempt to add Delegate permissions for UserB to UserA’s mailbox, we now fail to add these delegate rights if we cannot assign the "Send on Behalf" of right so that the delegate can send mail on behalf of UserA. When assigning delegate permissions based on the above scenario, we actually do two things, we first update the publicDelegates attribute for UserA listing UserB and we then add a rule to UserA’s mailbox for UserB access. This publicDelegates attribute controls the ability to Send on Behalf of that user. The ability to update this attribute for that user requires that the SELF account have at least write access to "Write Personal Information" on the user account that is trying to add the delegate. Prior to this hotfix, if we failed to update publicDelegates, we still added the Delegate Rule in the mailbox to provide limited access to the mailbox. From the end user perspective, they were able to get in to the delegated folder and be able to manipulate items. This "appeared" to the end user that everything was working as expected. Now, if that user attempted to send mail on behalf of that user, they would receive the following error:

You do not have the permission to send this message on behalf of the specified user.

Data in Active Directory is stored in the domain partition and maintained on domain controllers. If you have the appropriate permissions, you can then successfully write to the domain partition. In order to facilitate a global directory, we have to provide that data globally; thus the need for global catalog (GC) servers. Global catalog servers are special domain controllers that hold a writeable copy of the domain partition for their domain, as well as read-only copies of certain data from all other domains within the Active Directory Forest.

In a multi-domain environment, the ability to update the publicDelegates attribute is controlled by whether you can access a writeable copy of the object. This means that we must connect to a GC that is from the same domain as the mailbox-enabled user object is stored, in order for the user to update this attribute. If Exchange referred us to a GC that was not where the user account lived, Outlook would fail to update that attribute and cause the aforementioned delegate error. This delegation error was one of the reasons that Microsoft made a change to the DSProxy referral mechanism in SP2 so that Exchange will now direct you to a domain where you user account lives instead of the domain where the Exchange servers live. This helped alleviate some of the pain points that other customers were seeing and this change helped address those issues. Starting with SP2, Exchange now uses a point system that uses a constraint algorithm to determine the best suitable domain controller that is sent back to the Outlook user. This new algorithm is discussed in http://technet.microsoft.com/en-us/library/b7f8fbf4-732c-4a87-a9d5-3c4c375e5948.aspx.

This unfortunately had a negative impact to update DL membership when the DL’s were housed in the Exchange domain, thus causing the inability to update DL membership from an Outlook client. This occurs because Exchange is now referring you to a domain where your user account lives instead of the Exchange domain. You basically had a 50/50 shot that this was going to work depending on what domain controller Exchange refers you to. Exchanges preference once SP2 is applied is a user domain GC instead of an Exchange domain GC.

Microsoft then came out with a Post SP2 hotfix (912584) that adds a new registry key of "RFR Prefer In-Site GCs" to help ensure that a GC in the same AD site as the Exchange Server will always be used. That doesn’t guarantee that it will be a GC from the same domain as the Exchange sever. UserGCs first, then any other GC sorted by point cost.. If you had Exchange and user domain GCs in the same Active Directory site, then this new registry change does not make any difference with the point system and will still refer you to a user domain GC as a first preference. To resolve this behavior so that the point system works, you would need to move the user domain GCs out of the Exchange Active Directory site, thus leaving only the Exchange GCs in site where Exchange lives. Once this is done, the point system will work in a way that we will now prefer Exchange domain GC’s instead of user domain GC’s. This mimics the behavior prior to SP2 of Exchange.

Ok, so now you have configured your Exchange server to refer Outlook clients to the Exchange GC’s. This now breaks the ability to assign delegates since the user accounts live in another domain in which we only have a read only partition to make updates against. This of course will get you an access denied every time.

So in the end, there is really only one solution to make this work 100% which is moving all of your users and DL’s to the same domain which some companies cannot do or cannot afford based on the way that Active Directory was architected in their environment.

Listed below are some workarounds to make certain pieces of this work, but still does not allow everything to work in large multi-domain environments and other solutions needs to be found based on your needs to help address those issues.

Possible Workarounds

Pros

Cons

For delegate access problems, Assign the "Write Personal Information" right to all users or a subset of users in the domain that need the ability to assign delegate rights.

This helps alleviate some of the permissions issues that 913696 introduces

This requires that all users reside in the same Active Directory domain. You can still have multiple domains, but any user that needs the ability to assign delegate rights, those users accounts need to reside in the same domain. Exchange still needs to refer you to a GC with writable context to update the publicDelegates attribute.

For DL update issues, Move the distribution groups to the same domain as the user accounts (See 238394 or this for information using MoveTree to move Distribution Groups.) This is the recommended approach to resolve these issues 100%

This allows DSProxy requests to refer you to correct GC to perform the update. Since the DL’s and users are all in the same domain, we are able to update DL’s and assign delegate permissions.

This requires a good understanding of your AD Site/Subnet topology since the Exchange RFR process now tries to direct you to a user domain GC instead of an Exchange GC based on their current subnet.

 

Combining both users and DL’s in the same domain may break any custom applications that were written exclusively for a particular domain and would have to be changed to accommodate the new location for these groups.

 

This only works in an environment where there is a single domain to house both the users and DL’s. You can have multiple domains, but for this to work correctly, all users and DL’s must reside in the same domain and Exchange has to be setup to refer you to the correct GC’s as well. This referral mechanism is outside the scope of this document and is better explained in the following links

·         http://technet.microsoft.com/en-us/library/b7f8fbf4-732c-4a87-a9d5-3c4c375e5948.aspx

·         http://msexchangeteam.com/archive/2005/11/04/413669.aspx

·         http://msexchangeteam.com/archive/2006/03/17/422350.aspx 

 

Apply hotfix and registry entry of "RFR Prefer In-Site GCs" per 912584 to ensure that a GC in the same AD site as the Exchange Server will always be used.

 

 

Allows you to force clients to use GC’s within the same AD site as the Exchange Server instead of using User Domain GC’s that are not in the same AD site.

If  User Domain GC’s and Exchange GC’s are in the same Active Directory site as the Exchange server, the point system does not change with this registry key being set which is unfortunate. The only way to make this registry key work is to move all User Domain GC’s out of the Exchange Active Directory site so that the point system can work and then refer you to Exchange GC’s instead of User Domain GC’s

 

This will result in additional load on the GCs in the same Domain and AD Site as your Exchange servers, but this should be minimal.

 

This will result in issues with delegate permissions that the change to DSPROXY.DLL in SP2 was intended to address if the user accounts are in another domain outside of the Exchange domain.

Use a 3rd party tool to provide the ability to update distribution lists via a web interface. See this page for a list of Active Directory partners that may include this functionality.

 

This allows a single web based interface in to all DL management functions

Microsoft currently does not provide a DL management solution at the current time.

Change the details template per this. See Figure 1 below for a sample of what this might look like.

Less intrusive on the client as this change will only need to be changed in one place which is server side.

The details templates would need to be updated for each language.

 

No workaround still to manage Distribution Lists. Have to purchase 3rd party solution or you can create their own home grown application

Per 319206, you can use a GPO to force OL clients to use a specific GC via the "Closest GC" and "DS Server" registry keys

Temporary fix to workaround issues in certain scenarios where some users may not have the need to perform either a DL Management or Delegate function. You unfortunately cannot have the best of both worlds in multi-domain environments.

The client-detected global catalog server may be out of date or semi-functional. If the global catalog server is having problems but still responds to Named Service Provider Interface (NSPI) requests, Outlook may not stop responding and return to the DSProxy for a new referral.

 

In multi-domain environments, the global catalog server that you select may not be in the same domain as Active Directory group objects. Therefore, users cannot update group membership because the local global catalog server has a read-only copy of the group.

 

This key cannot be used in the Exchange Resource Forest Scenario as the GC used by Outlook must be one in the Resource Forest where the mailboxes are located.

 

Exchange performs a series of suitability tests to make sure a GC is ready/suitable for use by Exchange and its clients; this registry key bypasses that option, which could lead to a bad experience for the client.

 

If the server goes down that is specified in the DS Server key, Outlook will need to be restarted to connect to another GC. If outlook detects during startup that the specified GC is down, we will go and request a referral to a GC from the Exchange server.

Use RPC over HTTP (Outlook Anywhere) for users who manage Distribution Groups. Starting with Exchange 2003 SP1 and later, we no longer enable RFR for HTTP requests, so Exchange is doing all of the proxying to the domain controllers for user requests. This change helped with security issues where GC’s were exposed over the internet.

If DL’s reside in the same domain in which the Exchange servers live, then the ability to update DL membership might work. This all depends on what DC exchange used to perform the update.

This may work in certain situations, but is not the end all solution as you may get referred to an incorrect GC to perform the DL updates and the configuration of delegates.

Use the "No RFR Service" registry key per 302914 to disable the RFR interface on the Exchange Server so that we no longer refer you to a GC, but instead, Exchange handles all of the NSPI Proxy requests. If a specific GC is required with this key set, they you can use the "NSPI Target Server" registry key to hard code Exchange to use a specific GC to proxy requests to.

This may help alleviate DL update issues if they reside in the same domain as the Exchange server, but can have adverse affects for the ability to update Delegates.

This adds a performance hit on the Exchange servers as the clients have to use the Exchange server for all Address List Lookups instead of offloading this to a GC.

 

Using this key will require you to recreate all Outlook user profiles as we cache the previously referred GC in the registry of that profile.

 

Hard Coding an NSPI Target Server will provide undesired results to the client and the Exchange server should that server go down or is unavailable.

In a small environment, you can have your Exchange Administrator administer Distribution lists rather than Outlook Users.

Administrators or help desk personnel that have been delegated rights to update group membership can easily update DL membership for their users.

Adds additional overhead for the Exchange or Active Directory Administrator to update DL membership.

For delegate scenarios, you can add the Send On Behalf of right on the user account in which you want to delegate permissions to.

This helps alleviate cross domain delegate issues, but you still need to grant specific permissions to the folders once that right is added.

This requires a user to open Active Directory Users and Computers and then grant this right provided they have access to update the user in Active Directory which adds additional overhead. If that user doesn’t have permission, then this requires another person with at least account operator permissions to perform this update.

 

Granting permissions in this way grants no permission by default and requires the user to go back in to Outlook and then grant the appropriate folder permissions for the delegated user. If the user that was granted delegate rights needs to grant someone else delegate access, they need to go through the same steps as above to get this added.

The below image shows a sample of a modified details template that has removed the details button and replaced it with some textual context for the users to refer to:

RELATED LINKS:

Exchange Server 2003 Service Pack 2 DSProxy Referral Process Changes
http://msexchangeteam.com/archive/2005/11/04/413669.aspx

Exchange 2003 post SP2 DSProxy Referral Update
http://msexchangeteam.com/archive/2006/03/17/422350.aspx

RELATED ARTICLES:

318074 "Do not have sufficient permissions" error message occurs when you use Outlook Address Book to manage distribution list membership
http://support.microsoft.com/default.aspx?scid=kb;EN-US;318074

921927 How to prevent users from downloading the Offline Address Book without disabling Cached Exchange Mode
http://support.microsoft.com/default.aspx?scid=kb;EN-US;921927

319206 How to configure Outlook to a specific global catalog server or to the closest global catalog server
http://support.microsoft.com/default.aspx?scid=kb;EN-US;319206

912584 The DSProxy service does not direct an Outlook client to the global catalog servers that you want to use in Exchange Server 2003
http://support.microsoft.com/default.aspx?scid=kb;EN-US;912584

912584 The DSProxy service does not direct an Outlook client to the global catalog servers that you want to use in Exchange Server 2003
http://support.microsoft.com/default.aspx?scid=kb;EN-US;912584

924085 Description of the security update for Outlook 2003: January 9, 2007
http://support.microsoft.com/default.aspx?scid=kb;EN-US;924085

238394 How to Use the MoveTree Utility to Move Objects Between Domains in a Single Forest
http://support.microsoft.com/default.aspx?scid=kb;EN-US;238394

913696 Description of the Outlook 2003 post-Service Pack 2 hotfix package: January 23, 2006
http://support.microsoft.com/default.aspx?scid=kb;EN-US;913696

How to control the DSProxy process for RPC over HTTP connections in Exchange Server 2003 SP1
http://support.microsoft.com/default.aspx?scid=kb;EN-US;872897

Mike Lagase


Share this post :

Comments (9)
  1. bday says:

    Hello, can you please address an additional issue that does not seem to be mentioned in the workarounds?

    Our AD Site Topology is very hub/spoke. We have 13 domains, 1 of which is where Exchange lives, and it has a dedicated Exchange Site in AD. There are around 23k mailboxes and around 120 GCs. None of the users live in the domain that Exchange is in.

    We have experienced the following, preventing many (thousands) of our users from being able to Edit DLs and Delegates.

    If there is no UserDomain GC in the Exchange Site or a Site directly connected to the Exchange’s site via an AD Site link, that User’s Outlook client will NEVER get a GC in their domain. Outlook is referred to a GC in Exchange’s domain in the Exchange AD Site even though it scores 27 points instead of 28 points for a GC in the User’s domain.

    Can you please explain why even if a GC is sitting on the same subnet as the user, Exchange will still tell Outlook to use a GC in Exchange’s site even though it is not a member of the user’s Domain. I guess it confuses us why Exchange will only refer Outlook to GCs in its own site, or one directly connected to its own site. From our point of view it seems to makes load balancing GCs a bit trickier and introduces unnecessary WAN traffic.

    We have many perfectly good GCs sitting on the same subnets as users, but they’re never used by Outlook for this reason. Many times the sites directly connected to the Exchange Site only have UserDomain DCs, not GCs for these domains. We could put UserDomain GCs in the Exchange site, but that could create an enormous amount of WAN traffic that seems to be unecessary.

    We’d prefer not to use Glosest GC reg keys so that we can retain dynamic GC selection in the event of that GC ever being offline.

    Thank you very much, I hope that made sense.

  2. Mike Lagase says:

    Something doesn’t sound right with Exchange not referring you to the correct GC’s. Have you added the RFR prefer In-Site GC’s reg key on the exchange server?

    If not, I would open a support ticket with PSS to do some additional investigation on what might be happening in your environment.

    Hope this helps.

    Mike Lagase

  3. bday says:

    Mike thanks for the reply,

    No, the RFR prefer In-Site GC’s reg key is not set on our servers.

    I found this in a previous Blog entry:

    http://msexchangeteam.com/archive/2005/11/04/413669.aspx

    In Section: What Doesn’t the SP2 DSProxy Change Solve?

    Content: If there are no home-domain global catalog servers in the Exchange server’s AD site or in any of the Exchange server’s directly connected AD sites, then the Outlook client will receive a referral to a global catalog server that contains a read-only copy of the mailbox-enabled user.

    With that said picture our AD Like this….

    Exchange Site with Exchange Domain GCs Only —site link—> Domain Hub Site with User-Home-Domain DCs only —site link—> User Site with User-home-domain GCs.

    Our users only get an Exchange Home Domain GC in this setup even though User-home-domain GCs live in the same place the users do because (a) There is no user-home-domain GC in the Exchange site and (b) There are no user-home-domain GCs in the AD site directly linked to Exchange.

    I encourange anyone to test this in a lab with default SP2 reg keys and no special hotfixes in place. I don’t really want to burn another one of our PSS Support Contract cases for something that appears to be "by design". :)

  4. Mike Lagase says:

    Well, that explains it then. Since the GC’s are not directly connected to the Exchange site, that is the reasoning for only getting Exchange GC servers.

    You would need to modify your AD Site Links to be directly connected with the Exchange site to make this work properly

    Any reason why those DC’s are not also GC’s in the Domain Hub site?

    Mike

  5. bday says:

    I honestly do not know, I wasn’t employed here when the forest was first stood up.

    We can do the site links, but kind of like to keep those to a minimum when possible, we have over 100 AD sites. I guess our main inquiry here is why can’t Exchange tell Outlook to use the GC sitting on their user’s subnet since it would score a 28 vs 27 for a GC in Exchange’s site? Why does the limitation of the Exchange site or a directly connected site built in when it isn’t Exchange itself using the GC for these operations (or am I misunderstanding it?)?

    I beleive my primary goal is going to be obtaining and deploying new GCs in the hub sites for each domain and modeling our Site links to look more like our DNS heirarchy configuration.

    We just want to know "why" it is this way. :)

    Thanks!

  6. Mike Lagase says:

    The ability to discover these GC’s is done during topology discovery that essentially does something similar to the following…

    1. Find bootstrap servers when DSaccess is first loaded to find 1DC, 1GC and then confirm that a CDC is available based on those servers. If a CDC server is available, then we initiate a full topology discovery after the SA service has started.

    2. Full Topology discovery is performed querying the local site and any directly connected sites along with any Site Cost Information.

    3. Once a list of servers is found, we then perform suitability checks.

    4. After the checks are done, we then have full topology and if logging for topology is turned to Medium, you should see the 2080 event logged in the application log stating the servers that were found and are usable by exchange and outlook clients. This topology information is what DSProxy uses to pick what GC’s outlook can be directed to.

    If this is not populating any GC’s from the sites where the userdomainGC’s are, then your AD Site links would need to be modified to allow the correct referral process to occur.

    Mike

  7. bday says:

    Thanks again, Mike, it is obvious we need some redesigning to be done. I guess "full topology" is a little overzealous there in its assumption that you don’t have sites beyond the "first hop". ;) I’m going to enable that topology diag logging just so get a better understanding of it. Thanks for your help!

  8. jmancuso says:

    I’ve encountered a similar issue to the one described.  Specifically, Outlook users are not directed to a GC they have access to. Because of the use of firewalls and a “flat” Sites & Services topology at my University, the behavior change introduced by E2K3 SP2 has caused some headaches for me since April 2006.    

    The forest is currently comprised of 8 domains including 1 root and 7 children. Schools and major administrative units on campus have deployed firewalls to segregate and control traffic to/from their network subnets and the campus enterprise network.  The current topology of the Active Directory enterprise forest is “flat”; all domain controllers pass information and replicate with each other.  The reason the topology was originally designed to be “flat” is because everyone is connected at high speed and are in the same geographic location.

    Immediately after upgrading to E2K3 SP2, when a Microsoft Outlook client computer connects to a Microsoft Exchange Server, the DSProxy service may not direct the client to the Global Catalog server that it should. Instead, the DSProxy service directs the client to a domain controller that is in a different domain. (e.g. a user in DOMAIN-A is directed to a GC  in DOMAIN-B) Since individual workstation access is being denied by most firewalls on the campus, an Outlook user may experience a timeout when attempting to attach to an Exchange server because it cannot contact a GC server in another domain.

    We have contacted Microsoft PSS multiple time and each time they state that we need to deploy a Site & Services (S&S) topology. I agree with the S&S setup but my frustration is in understanding exactly how DSAcess and DSProxy actually work in a Post E2K3 SP2 implementation with Outlook 2002/2003 clients:

    It is my understanding that DSAccess on each Exchange server discovers the Active Directory topology, detects domain controllers and Global Catalog servers, and maintains a list of valid directory servers that are suitable for use by Exchange components. In addition, DSAccess maintains a cache to minimize the load on AD by reducing the number of LDAP requests. DSProxy is an Exchange 2003 component that provides an address book server to Outlook clients. In Outlook 2002/2003 clients, DSProxy refers the Outlook clients to a Global Catalog (GC) server. These clients connect to the user’s home Exchange server and send a request to the RFR service on the Exchange server. RFS works with DSAcess to determine the name of a qualified GC server and returns that name to the Outlook client. The Outlook client then sends its NSPI requests directly to the selected GC.

    If the above is true, then DSAccess through automatic discovery will know about all the GCs in the environment (no firewall blocks between DCs and Exchange). When DSProxy works with DSAccess for Outlook GC discover, it will pick a GC that a client might not have access to (e.g., a DOMAIN-A Outlook user can not access a DOMAIN-B GC because of a firewall block). So, my question still remains; will Sites and Services fix the root issue since individual workstation access to DCs in different domains is being denied by most firewalls on campus?

    Joe Mancuso

    Univ. Of Md, Baltimore

  9. Terabanitoss says:

    Hi all!

    You are The Best!!!

    G’night

Comments are closed.