How does Outlook Anywhere work (and not work)?

[The_Exchange_Team]

EDIT 8/22/2008: Corrected a typo in a "DisabledComponents" registry key name. It's been a while since I've been thinking of writing a blog post about various aspects of Outlook Anywhere that people have been asking questions about. Somehow, I keep getting myself caught up in one thing or another, and have consequently delayed writing this blog post by almost 4 months. Ugh. Better late than never I figure. Given how long this blog post is overdue, I plan to cover a lot of topics, from frequently asked questions to common misconceptions to problems with Outlook Anywhere to suggested solutions for different problems. How does Outlook Anywhere work? I won't cover details on the cmdlets that enable and change settings for Outlook Anywhere. There is already a bunch of documentation on it. Instead, let's do a slightly deeper dive than the cmdlet documentation provides. The values that you provide to Outlook Anywhere settings can be classified into 2 types of properties - client facing and server facing. Examples of client facing properties are ClientAuthenticationMethod, External Host Name. Examples of Server facing properties are IISAuthenticationMethods, SSLOffloading. Client facing properties are picked up by Autodiscover and supplied to Outlook to configure client access to the Outlook Anywhere service. Server facing properties are picked up by a servicelet called RpcHttpConfigurator which runs as part of the Microsoft Exchange Service Host service. This servicelet runs every 15 mins by default, but the interval can be adjusted by changing the value of the HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MSExchangeServiceHost\RpcHttpConfigurator\PeriodicPollingMinutes regkey. Note that setting this value to 0 turns off the RpcHttpConfigurator. When the RpcHttpConfigurator runs, it picks up the IISAuthenticationMethods and SSLOffloading values from the AD and stamps it on the \rpc vdir settings in the IIS metabase - overwriting any previously set value. This means that if you manually change the settings on this vdir, you should expect to be run over pretty shortly by the RpcHttpConfigurator (unless you have set the reg key to 0). Ok, so that's just part of what the servicelet does. Outlook Anywhere depends on the RPC/HTTP Windows component to do the marshalling and unmarshalling of the RPC packets from the client to the CAS server. A client side RPC component is responsible for marshalling every RPC packet into an HTTP tunnel and sending it over to the \rpc vdir on the CAS server. RPCProxy is an ISAPI extension that unmarshals the RPC packet, retrieves the RPC endpoint that the client wants to talk to and forwards the packet to the endpoint. But imagine if you were able to connect to any server in the organization if you were able to auth against an IIS box running RPCProxy. By the weakest link theory, all you'd need to do would be hack into a single IIS server and you'd have free access to all servers in the org. Ouch ! To alleviate this problem, RPCProxy only allows connections to be made to servers and ports that are in a trusted list. This list is maintained through the HKEY_LOCAL_MACHINE\Software\Microsoft\Rpc\RpcProxy\ValidPorts regkey and contains all the servers/ports that RPCProxy is allowed to talk to. So, the other part of what the RpcHttpConfigurator servicelet does it that is queries the AD for all mailbox servers and stamps them in the ValidPorts regkey allowing access to ports 6001, 6002, 6004 for both FQDN and Netbios access. So, you will typically see something like mbx1:6001-6002;mbx1:6004;mbx1.contoso.com:6001-6002;mbx1.contoso.com:6004 as the value for the key. As new mailbox servers are added to the org, they will be picked up when the servicelet runs and be added to the key. Again, if you manually change this regkey, you should expect to be bulldozed by the servicelet. Note that the ValidPorts key is only used by RPCProxy as a filter to disallow communication with unlisted server ports. It is not used to determine which server to send requests to. For the same reason, the order in which servers are listed in this key does not matter. I just thought I'd clarify this since I was recently told that there was confusion on what this key accomplished. Ok, simple enough, now that all the configuration is done, how does Outlook Anywhere actually establish its connections. The following diagram may help: As you see above, the client specifies the VIP of the Load balancer (or direct CAS FQDN if the CAS is exposed to the Internet) as the HTTP endpoint and the mailbox server as the RPC endpoint. The query string is somewhat like this: http://nlb.contoso.com/rpc/rpcproxy.dll?mbx1.contoso.com:6001 This tells the RPCProxy on CAS1 that the client is trying to connect to server mbx1.contoso.com on port 6001. RPCProxy looks up the ValidPorts key and if mbx1.contoso.com:6001 is listed there, it allows the connection to go through. The blue and red arrows above represent the 2 different connections spawned by the RPC/HTTP client component to represent a single RPC connection. This is done because HTTP connections are half duplex (i.e. they either allow you to send information or receive information, not both at the same time). In the case of RPC, connections need to be long lived and full duplex, so the RPC_IN_DATA connection acts as the sending half duplex connection, while the RPC_OUT_DATA connection acts as the receiving half duplex connection. Since HTTP requires that each connection be given a max length, each of these connections are 1GB "long" and are terminated when this limit is reached. Each of these connections is tagged with a client session id. When the RPC server component receives the RPC_IN_DATA and RPC_OUT_DATA with the same client session id, it knows that for any request received on the RPC_IN_DATA connection, it must reply back on the RPC_OUT_DATA connection. That's magic. Ok, so you already know this, but I'll reiterate - the mailbox server has 3 ports that are used for RPC/HTTP: port 6001 is used for Mail connections, port 6002 is used for directory referral, port 6004 is used for proxying directory connections to AD. The Referral Service running on port 6002 and DSProxy running on port 6004 are part of the same mad.exe process, and the Referral Service just refers clients back to DSProxy to establish their Directory connections. If you Ctrl+Right Click the Outlook icon and click on Connection Status, it will tell you what connections exist (Mail vs. Directory), what server they are going to and what protocol they are using (HTTPS vs. TCP(direct Exchange RPC connection)). I have conveniently omitted any discussion around certificates, since that can take up another few blog posts. As some would say, that is beyond the scope of this article and is left as an exercise to the reader. How do I know Outlook Anywhere is working? Simple... when no one is complaining! Seriously though, it is preferable is to run diagnostics on Outlook Anywhere before subjecting it to thousands of users. The one tool that works pretty well in most cases is rpcping. Yes, it has a lot of parameters and is confusing, but it does provide pretty good diagnostic information and as long as you have the KB open, you can figure out where problems lie. Start by pinging just the RPCProxy by using the -E option. Once that works, move onto testing the mailbox server endpoints by removing the -E and adding -e 6001 instead. Similarly for 6002, 6004. A typical command line would be something like this. Refer to http://support.microsoft.com/kb/831051 for usage details rpcping -t ncacn_http -o RpcProxy=cas1.contoso.com -P "user,domain,password" -H 1 -F 3 -a connect -u 9 -v 3 -s mailbox.contoso.com -I " user,domain,password " -e 6004 How does Outlook Anywhere not work? Unfortunately, there are some cases where Outlook Anywhere does not work without requiring manual tweaks. This is the part I wish I had blogged about earlier. I'm sure there are poor folks out there that have hit these issues and wasted their time figuring out what I had already learned... DSProxy and IPv6 As of E12 SP1, Outlook Anywhere on Windows 2008 requires that IPv6 be manually turned off on the CAS server. This is because the DSProxy component that listens on port 6004 (mad.exe) for directory connections does not listen on the IPv6 stack. If you do a netstat -ano | findstr 6004, you will see only 1 LISTENING entry - the one that corresponds to the IPv4 stack. Contrast this with ports 6001 and 6002 that have 2 entries. (As most of you already know, if you are running your Mailbox role on the same machine as a DC, lsass.exe not mad.exe listens on port 6004, so this problem will not surface since lsass.exe listens on both protocol stacks.) How do you turn off IPv6 ? It depends on whether you are running CAS and Mailbox on the same server or different ones. If you're in a multi-server scenario where the RPCProxy is not on the same server as the Mailbox, then you need to do the following:

1. Unselect IPv6 from the properties of your NIC (on the RPC-over-HTTP Proxy machine); that will force the RPC-over-HTTP Proxy to use IPv4 to talk to Exchange and everything will be fine. In most cases, this step suffices. If it does not, continue with steps 2 and 3.
2. Under the regkey HKLM\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters, add a 32 bit DWORD with the name DisabledComponents and value 0xFF
3. Reboot the machine

If you're in a single-server scenario where the RPCProxy and Mailbox are on the same machine, then the above does not work since the loopback interface still uses IPv6. In this case, you need to make the following changes in the system32\drivers\etc\hosts file:

1. Comment out the line ":::1    localhost"
2. Add the following two lines: <IPv4 address>    <hostname of the computer> <IPv4 address>    <FQDN of the computer>

Thanks to Kevin Reeuwijk and others for finding and reporting the issue and solution. A fix (make DSProxy listen on the IPv6 stack) is on the way and is expected to be available in Exchange 2007 SP1 RU4 in Q3 2008. DSProxy and Split RPC_IN_DATA, RPC_OUT_DATA connections In the diagram above, you will notice that I have used a Source IP Loadbalancing layer. This ensures that the RPC_IN_DATA and RPC_OUT_DATA connections coming from a single Outlook instance are always affinitized to the same CAS server. However, there are some legitimate scenarios where Source IP affinity is not viable for customers. A typical example is when a large number of end users are behind NAT devices causing all connections to end up with the same IP and hence the same CAS server... yay load balancing! Outlook Anywhere does not support cookies, so cookie based Load balancing cannot be used either. The only way of spreading load across the server farm is to use with no affinity or SSL-ID based affinity. However, this has the problem that the RPC_IN_DATA and RPC_OUT_DATA connections could (and most likely would) end up on different CAS servers as shown in the diagram below: If you've been reading closely, you'll remember my earlier mention that the RPC server component is well aware of client session IDs and can reply on RPC_OUT_DATA for any requests on RPC_IN_DATA. And if that's the case, we should still be fine since Outlook always specifies the mailbox server as it's RPC endpoint. Well, almost. We are fine for ports 6001 and 6002 which are real RPC end points. The issue is with port 6004 where DSProxy pretends to be an RPC endpoint, but is just a proxy as the name implies. DSProxy only proxies client connections through to the DC. In the example above, RPC_IN_DATA is proxied to DC1 while RPC_OUT_DATA is proxied to DC2. The DCs are the real RPC endpoints. However, now that the 2 connections have been split, neither of the DCs is aware of the other connection and requests sent on RPC_IN_DATA are lost in oblivion. We call this split connectivity and it is a problem surfaced by SSL-ID or no affinity load balancing. While I would recommend not using these configurations if avoidable, it is clear as described earlier that these may be the only alternatives. Think hard if this is the case since the workaround that I am describing below will be tedious to maintain. The goal of these steps is to eliminate the possibility of split connectivity by (1) having clients bypass DSProxy wherever possible and (2) constrain DSProxy to talking to a single DC for any requests to DSProxy. First off, you need to avoid using DSProxy wherever possible. Normally, the Referral Service running on port 6002 refers clients to DSProxy on port 6004. By setting the following regkey, you instruct Referral Service to not send clients to DSProxy, but instead give them a referral to a DC for directory connections. So, instead of client connections going from Client to RPCProxy to DSProxy to DC, the path would be from Client to RPCProxy to DC. Note that the client is not directly connecting to the DC, so it is not required to publish the DCs to the internet or open any new firewall ports. See KB http://support.microsoft.com/kb/872897 for details:

On the Mailbox servers: a DWORD  entry needs to be created on each Mailbox server named "Do Not Refer HTTP to DSProxy" at HKLM\System\CCS\Services\MSExchangeSA\Parameters\ and the value set to 1

Next, as indicated earlier, the RPCProxy will block access to the DC servers unless there servers are included in the ValidPorts regkey. So, set the following on the Client Access Servers

1. The ValidPorts setting at HKLM\Software\Microsoft\RPC\RPCProxy needs setting so that the entries referring to 6004 point to DC servers in addition to the mailbox server.
2. The PeriodicPollingMinutes key at HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MSExchangeServiceHost\RpcHttpConfigurator\ needs setting to zero to prevent RpcHttpConfigurator from updating the Valid Ports key automatically.

Finally, you need to make sure that the DCs are listening on port 6004:

On the Global Catalog servers: a REG_MULTI_SZ  entry needs to be created on each GC named NSPI interface protocol sequences at HKLM\System\CCS\Services\NTDS\Parameters\ and the value set to ncacn_http:6004

These fixes will make sure that all directory connections bypass DSProxy and terminate at the DCs, thereby allowing the DC RPC server side component to receive both the RPC_IN_DATA and RPC_OUT_DATA connections. There is 1 last thing to deal with in this SSL-ID load balanced configuration. Outlook profile creation hard codes a call to DSProxy on 6004. Which means that we can get split connectivity during profile creation. To deal with this minimal volume of traffic, there is 1 final regkey that should be set on the mailbox servers:

On the Mailbox Servers - set the HKLM\System\CCS\Services\MSExchangeSA \Parameters key "NSPI Target Server" to the FQDN of the DC that profile creation should use.

By using only 1 DC for profile creation, all DSProxy calls will be proxied into that single DC, once again avoiding split connectivity. That's it folks! Of course, subsequent releases will provide cleaner solutions for such topologies, but for now, rest assured that having gone through the above steps multiple times, I feel your pain. That's pretty much it. I hope that adds some clarity to how Outlook Anywhere works and hasn't succeeded in confusing everyone even more. Until the next post - Hasta Luego! - Sid