So for many weeks I've debated about what to post here. My colleagues and I have worked pretty hard since the release of Exchange 2000 to document the SMTP Transport stack. So I figured by now everyone knew everything there was to know about setting up SMTP domains for inbound and relay. We covered the topic in the Greenbooks (http://www.microsoft.com/technet/prodtechnol/exchange/2003/Library/default.mspx), web casts, KB articles (http://support.microsoft.com/?id=260973, http://support.microsoft.com/?id=268838), and more. But the truth is, it is complex, and is difficult to troubleshoot without understanding how it all works. So here's everything you ever wanted to know on how to configure SMTP domains for inbound and relay.
First, you have to know what inbound and relay domains are. Clearly, it would be bad for Exchange to accept email for every SMTP domain. Either one of two things would happen, we'd either be generating a lot of non-delivery reports (NDRs) for emails we never should have accepted, or we'd be forwarding email to a lot of different domains around the world. Thus, we limit what we accept for inbound and relay. Inbound domains (authoritative) are domains that Exchange accepts mail for, and if there is no match in the directory (AD), we generate an NDR. Relay domains are domains that we accept mail for, check to see if there is a match in our directory, and if not, forward it using either DNS or to a smart host.
So, out of the box, Exchange accepts email for just one authoritative inbound domain, your default domain. The Exchange Metabase Update agent (also known as DS2MB) replicates this domain from the default recipient policy object into the IIS metabase, where SMTP reads it. A good chunk of the SMTP related configuration is replicated in this manner. So this means that you may have to wait on the local Config_DC to complete replication, but shortly thereafter, DS2MB will notice the change and replicate them to the metabase. Even better (especially those who remember 5.5), few changes require any sort of restart of services.
New installations will have the default domain of their AD namespace, which is typically not what you want to leave it as. You can of course change this default domain to be anything you wish, whether it resolves in DNS or not. In fact, that is one way that you can for all practical purposes prevent inbound email. Next you may want to configure it such that Exchange accepts email for other domains. Simply make sure that SMTP domain appears in a recipient policy. It need not be the default, although it can be, and you need not use the Recipient Update Service (RUS) to stamp all user accounts. For example, if you only had two users that need to accept email for the secondary domain, then you would simply create a new recipient policy with a blank LDAP filter (assuming you put the addresses on the two accounts manually) or create a filter that selects just those two users. In either case, the RUS would not stamp the rest of your accounts. Yes, there is no "reroute" functionality like Exchange 5.5, but this provides nearly identical results. You can add up to about 1000 domains total before you need to start thinking about performance and our "hosted Exchange" solution. Speaking of Exchange 5.5, if you are in a mixed site/AG, although you can add SMTP domains to any recipient policy, if you DO want the RUS to stamp addresses on the users, you will need to use the special recipient policy for that site. Yes, this policy supports multiple SMTP domains. Just make sure you're running the latest service pack and you should be able to do this.
Now let's say you want a relay or non-authoritative domain. Since relay is considered mail not destined for the local organization, you do not configure this on the recipient policy. What you do is create a new SMTP connector with a specific address space (e.g., "myrelaydomain.msft") and check the box for "Allow messages to be relayed to these domains" on the same tab. You do not ever want to do this on a connector with address space of "*" because that would open you up to allow relay to all domains, which would clearly be bad. Selection of the bridgehead of this connector should be done with care. This checkbox is what allows you to receive mail for this domain, so this is one of the only places where a connector not only serves an outbound purpose, but an "inbound" (well, relay) as well.
On the topic of RUS and relay/non-authoritative domains: you may wish to stamp SMTP addresses of the non-authoritative domain on users in your directory. This is where the checkbox for "This Exchange Organization is responsible for all mail delivery to this address" on the recipient policy comes in. This check box tells DS2MB not to replicate this domain to SMTP, not to accept the domain as inbound (thus limiting acceptance to the bridgehead with the connector mentioned in the previous paragraph), and also not to generate NDRs if no match is found for a particular address in the directory. A few words of warning here: First, never configure an authoritative domain for relay by adding a connector. This makes no sense to DS2MB and it will probably warn you with an event in your event log. You may also get unpredictable results. Secondly, ALL recipient policies should match on this setting.
Complex? Sure, though the most common scenarios are actually easier than in 5.5 where you had to go to both Site Addressing AND the IMC Routing tab. In addition to getting into trouble by not following my words of warning above, there is at least one other major limitation.
Allow me to help this be less complex as well - nothing related to this should ever have you modifying your SMTP VS settings. Just your recipient policies for inbound, and SMTP connectors for relay.
<sidebar>By the way, under very few circumstances would I recommend using the setting "Forward unresolved recipients to smart host", even though we've presented it as an option in KB321721 (http://support.microsoft.com/?id=321721). Certainly, you cannot mix the "Method 1" with "Method 2" as they are mutually exclusive. </sidebar>
Troubleshooting this comes down to looking at the NDR you're getting back:
5.7.1 means that you haven't either accepted the domain as inbound or relay
5.4.6 means loop -- that the server is configured non-authoritative for this domain, but then has no place to forward the message and is trying to deliver it back to itself (particularly if you are using contacts where the target SMTP address is a shared domain and/or DNS is resolving the domain back to the sending server)
5.1.1 means that the server couldn't find a match for the user in AD, and the server is authoritative for this particular domain
It is certainly possible, on a server that ran Exchange 2000 prior to SP3 (even if it has SP3 or later now) that DS2MB is confused and cannot replicate. Usually you get events in your event log from MSExchangeMU and you're getting NDRs, even though you're positive that you configured this correctly. Optionally, you can perform the following steps:
1. Backup the metabase
2. Stop the SMTP service
3. Use Metaedit or MBExplorer to delete all subkeys under LM/SMTPVS/X/Domain
4. Use the same tool to delete the LM/DS2MB key
5. Restart the Exchange System Attendant & SMTP services
If you aren't sure if this is your issue, or if you need help, please call Support Services and do not attempt this on your own. Editing the metabase is just like editing the registry and you should only proceed if you know what you are doing. It should take much less time to pick up the phone than it would to reinstall and re-patch Exchange and IIS.
In conclusion, if you would like to pick my brain on other Transport related topics, please post your suggested topics feedback here: http://blogs.msdn.com/exchange/archive/2004/08/11/213211.aspx.