Understanding how Outlook, CDO, MAPI, and Providers work together


When diagnosing problems with Exchange, particularly Exchange clients like outlook, or API consumers like server applications, it is common to encounter confusion about which components are actually doing the work for any particular operation.  It's common to hear descriptions of connection problems, address resolution problems, and even calendaring issues as "MAPI problems."  First, some terms.  For purposes of this discussion, MAPI will generally mean MAPI32.DLL, or the APIs contained therein.  CDO references CDO.DLL, not CDONTS, CDOSRV, CDOSYS, CDOEX, EXCDO, or CDOEXM.  EMSMDB refers to the Exchange Transport, EMSMDB32, and EMSABP means the Exchange Address book provider, EMSABP32. 


MAPI is a glue API, implemented largely in MAPI32.DLL (Messaging Application Programming Interface, 32 bit build).  In MAPI, developers implement a series of Providers.  Providers come in three major types, Transport Providers, Message Store Providers, and Address Book Providers.  Providers are DLLs which implement a specific pseudo COM API, such as IMessageStore, and the underlying required interfaces, such as IMAPIProp.  One example of a provider is EMSMDB2, the "Electronic Messaging System Microsoft Data Base, 32 bit build" provider.  EMSMDB implements both a transport and a message store, and as such is a dual provider.  The transport is the ability to submit messages to Exchange Server, the message store is the ability to read (and possible write) messages to an exchange STORE process.  The other common provider involved with Exchange is EMSABP, which is an Address Book provider (Electronic Messaging System Address Book Provider, 32 bit build).  EMSABP implements IAddrBook and allows access to the Exchange Global Address list via NSPI (Name Service Provider Interface).


A MAPI application simply means any app that uses the MAPI calls.  MAPI applications work against sets of data sources grouped into Profiles.  Profiles are created, saved, and deleted by the Profile Provider.  The Profile Provider is implemented in MAPI32.DLL in two forms:  the temporary profile provider, which stores profile data in temp files, and the normal profile provider, which stores profile data in the registry.


An empty profile gives a MAPI app no data access, so MAPI applications add Providers (also called Services) to the profile.  A common thing for MAPI Applications to do is add services for EMSMDB and EMSABP to a profile, granting the ability to access data from Exchange.  Providers do all of the actual work in MAPI.  When you submit a Message, while the call might pass through MAPI32, it is EMSMDB (or your provider) which does the actual work for sending.  When saving a message, it is not MAPI32 that saved the message, it's the message store provider (usually EMSMDB for Exchange).  When you resolve a recipient, it is not MAPI32 that searches for the user in the address book. It is the Address Book Provider, implementing the IAddrBook interface (probably EMSABP).


CDO and Outlook are both MAPI applications, meaning that they use MAPI APIs to access their data, and therefore, the underlying providers.  CDO and Outlook share a common set of Properties that taken together define different classes of messages and different functionality.  This works similar to the schema in a database, however, MAPI32 does not implement the storage.  The Message Store Provider does.  In the Exchange case, this is EMSMDB and Exchange is mostly agnostic about the semantics of data applications choose to store.  Exchange 5.5 has almost no knowledge of individual named prop meanings.  Exchange 2000 and 20003 are considerably more aware of the significance of particular named props on different message classes, in particular calendar items.  This logic is implemented in EXOLEDB and EXCDO.  In general, it is up to users of a particular set of props to agree on what those props mean, and how they are manipulated. 


CDO implements an API to manipulate sets of properties together to accomplish tasks, such as scheduling a meeting.  Outlook also implements APIs to manipulate sets of properties, and does presentation of that data to the end user.  If CDO and Outlook don't agree on what particular props mean, or on the mechanisms used to control processing logic (for instance, when an appointment has been updated), then potential for problems interoperating between Outlook and CDO exist.  While Outlook continues to be updated and expanded with every version, CDO.DLL is more or less static at the same level of functionality it has possessed for years.  Newer libraries for manipulating calendar data, such as CDOEX, are considerably more up to date. 


A chain of communication


It is important to understand how the components communicate with each other. For example, an application (scheduleme.exe) uses CDO, with the intention of scheduling a meeting.  Here is a high level overview of what happens. Please note that this is very much abbreviated, there are numerous interactions at each point not relevant to the overall goals of this discussion.



1. Application uses MAPI32's profile services to create a profile and add EMSMDB/EMSABP to it.
2. Application asks CDO to log on to that profile.
3. CDO uses MAPI APIs (MapiLogonEx) to log on to the session.  CDO is also capable of calling the appropriate APIs to create a profile and populate it with Exchange Services, dependent on the parameters used to log on.
4. Application asks CDO to create a meeting request
5. CDO uses MAPI interfaces implemented by EMSMDB to access storage for what we will call "a message".
6. CDO uses MAPI interfaces (IMapiProp) implemented by EMSMDB to set a set of default properties that Outlook and CDO agree constitute "a meeting request".
7. Application supplies the appropriate values for subject, location, time, ect.
8. CDO uses MAPI Interfaces (IMapiProp) to set the properties that Outlook and CDO agree represent those data bits, as well as any intrinsic properties needed to support this.
9. Application calls CDO API to send the meeting request.
10. CDO uses MAPI Interfaces (IMessage::Submit) implemented by EMSMDB to send the message, in the process using APIs implemented by EMSABP (ResolveNames) to resolve the recipients.
11. EMSMDB32 uses a private RPC interface to submit the message.
12. Application releases CDO objects.
13. Application deletes profile via MAPI APIs (CDO will do this if it created it).
14. MAPI32 deletes profile.


Knowing when it makes sense to update binaries


Change control is essential for good server management, and most administrators do not apply random fixes to servers for experimentation purposes.  However, when troubleshooting certain classes of problems, updating the involved binaries makes a lot of sense (we will ignore issues of memory leaks and or heap corruption, in those cases it is unlikely that an argument can be made against updating if any such fixes have been made in more recent builds.) 


Take for instance, deadlocks.  Deadlocks could be caused by anything in the application chain.  The application, the top level API (CDO), the glue layer (MAPI32), or the Providers (EMSMDB/EMSABP in this instance).  Updating MAPI32 (the dll for the glue layer), CDO.DLL (the DLL for the top level API), the providers (EMSMDB or EMSABP), or the application would be appropriate.


Next, consider disappearing profiles, which often manifest as failure to open information stores.  Referencing the data above, you should see that profile services are implemented in MAPI32.  Therefore, MAPI32 would be a possibility for updating, as would be the application, since the application might be deleting its own profile.


What about profiles not being deleted?  Profile deletion might be the work of the program (if it calls MAPI directly), the top level API (CDO), or on rare occasions the provider (MAPI).  In this case, the application, CDO.DLL, and MAPI32.DLL are all good candidates for updates.


Next, consider problems with context specific data like Meeting requests, free/busy, or appointments.  Updating MAPI32, the glue layer that provides interfaces to the providers, but does not store or interpret data is not likely to resolve the issue.  EMSMDB, the agnostic storage provider is also not a likely candidate.  What about CDO?  CDO implements the sets of properties that define specific classes of messages such as meeting requests.  It has to agree with Outlook on how these properties work, the behavior can be unpredictable.  In this case, updating CDO.DLL makes sense.  Finally, there's the application.  A number of applications have attempted to operate directly against the calendar data, often with disastrous results.  Always consider the application.


What components communicate with each other?


Anything that touches in this diagram talks directly to the next layer.  This drawing is a generalization, not a flow diagram.  For instance Outlook can use CDO, and Outlook can in later versions communicate directly with EMSMDB.  The vital things to note here are that only the providers actually communicate with the server, and that these providers do not talk directly to their counterpart DLLs on the server.  That means that CDO on the client does not communicate with CDO on the server, nor does MAPI, EMSMDB, or EMSABP.



Where do I update these?


Application DLL should be updated wherever they are used by the application.  For example:


An application uses CDO. 
- If CDO needs to be updated, update CDO where the application runs. 
- CDO uses MAPI32. 
- If MAPI32 needs to be updated, update MAPI32 where it runs, which is by definition "Where CDO runs", which means "Where the application runs".


Applications use the providers via MAPI32.  If the providers (EMSMDB/EMSABP) need to be updated, do so where MAPI32 runs. 
Providers RPC to the Server interface implementations.  For EMSMDB32, this means it talks to STORE.EXE. 


Common Update questions



Q. My application doesn't run on the server, do the server binaries need to be the same version as the client?
A. EMSMDB32 on the client does not communicate with EMSMDB32 on the server. It communicates with STORE.EXE.


Q. Does MAPI32 on the client communicate with MAPI32 on the server? 
A. No, it does not.  MAPI32 does interface with the providers and the application


Q. Does CDO on the client communicate with CDO on the server? 
A. No, it does not.  CDO interfaces with MAPI and the applications using it.


Q. What if I have a CDO application that runs on the server?
A. Update the binaries "Where the application runs."  If your application runs on the server, then update it there.


Q. "But I also have it running on another machine." 
A. Update where the binaries run.  If that is the client, then update on the client.  If that is on the server, let's update on the server.  If the application runs on both, update both.


Q. "I have a 3rd machine that runs this." 
A. See above.


Client and Server versions of CDO and providers


The area of client and server versions of the Exchange Providers, MAPI32, EMSMDB, EMSABP and MSPST32 almost always causes confusion.  The Exchange Server group wrote each of these DLLs originally, and the Outlook client used the server versions directly for years.  However, as time passed, the DLLs diverged to meet different goals, even though they retain the same name.  Starting with Outlook 2000, Outlook shipped its own client optimized builds of MAPI32, EMSABP, and EMSMDB. 


EMSMDB32 on the client was modified to support RPC canceling instead of hanging waiting on the server as well as RPC/HTTP and Kerberos authentication.  EMSMDB32 on the server went through a series of multithreading changes to improve performance for server applications.  EMSABP32 on the client gained the ability to rebind to GCs, where the server version originally did not.  MAPI32 was further adapted to work well with 3rd part providers, while the server version received additional fixes to the temp profile provider. 


CDO is almost exactly the same code on the server as the client, but CDO.DLL on the server does not have any security warning dialogs, as you would not want your system service throwing up dialogs asking if you want to access data.  CDO.DLL on the client, on the other hand, does, because email worms are bad.  Finally, MSPTS32, the PST provider, has grown far, far richer under Outlook than the original server version.  The Outlook version supports Unicode PSTs, PSTs larger than 2 Gigabytes, and sundry other improvements.  The server version is still the original.  People often ask why Exmerge doesn't create PSTs larger than two Gigabytes.  The answer is that the server version of the DLL is not capable of it.  The usual question to follow is "Why don't' you just copy the DLL over?"  The answer is, it's not that simple.  Solving that problem remains an area we are investigating.


Thanks to Steve Griffin and Dana Birkby for helping restructure my original ramblings into a more friendly format.


- Jason Nelson

Comments (14)
  1. Alexander Zammit says:

    Great article thanks. It is nice to see good old MAPI being discussed. At the time of Ex2000 release this technology was too much treated as "legacy" despite being indispensable for many applications.

    Question: How do you see the future of MAPI? Are we to expect major changes with Exchange 12?

  2. Mike V says:

    How about OWA ?

    DOes OWA use WEBDAV ?

    And is WEBDAV essentially an Application in your diagram ?

    Thanks

  3. Jason Nelson says:

    Sorry, I can’t make statements about what will or will not happen in a future version.

  4. Jason Nelson says:

    Mike V, OWA is a mix of DAV/Internal store APIs. It does not appear in this diagram, but if it did, it would be an integrated block with the STORE process at the bottom of my strangely colored diagram.

  5. Juicy says:

    This was a very good blog, Jason! A great read. I have one question: Does Microsoft always recommend making sure that the build version of the CDO.DLL, EMSMDB32 and MAPI32.DLL match? Is there any issue with having (for example) CDO.DLL 6.0.6249, EMSMDB32 6.0.6249 and MAPI32.DLL 6.0.6603.0 on the same machine, if an application uses all three?

    Thanks.

  6. Juicy says:

    This was a very good blog, Jason! A great read. I have one question: Does Microsoft always recommend making sure that the build version of the CDO.DLL, EMSMDB32 and MAPI32.DLL match? Is there any issue with having (for example) CDO.DLL 6.0.6249, EMSMDB32.DLL 6.0.6249 and MAPI32.DLL 6.0.6603.0 on the same machine, if an application uses all three?

    Thanks.

  7. Jason Nelson says:

    There is no requirement that a particular version of CDO be used with a particular build of EMSMDB, EMSABP, or MAPI. There’s also nothing that ties a particular build of CDO to a particular version of the store, which is why the version of CDO that ships with the latest Outlook can run against Exchange 5.5, Exchange 2k, or Exchange 2k3.

  8. Anonymous says:

    Jason Nelson, one of our core Sustained Engineering guys, just posted an excellent article on the nature…

  9. Dan Christensen says:

    What role does the transport provider play when the message store supports submits directly? Our message store provider reads and writes information directly to an online database server. As such, calling IMessage::SubmitMessage() triggers the backend to do the send of the message without involving the transport provider. Similarly, when new mail arrives it is automatically delivered into the user’s mailbox on the backend, triggering a table update in Outlook. The transport provider isn’t really involved in "downloading" new mail like it would be in a POP or IMAP account. Yet a message service must have a transport provider defined to even be considered an "email account" by Outlook. I imagine that the Exchange providers operate in a similar manner. Could you elaborate on the interaction between the exchange message store and transport provider? I’ve also noticed that exchange includes two transport providers in it’s message service: both a normal and a remote transport provider. How does the remote transport provider interact with the message store?

    Thanks,

    Dan

  10. Dan Christensen says:

    What role does the transport provider play when the message store supports submits directly? Our message store provider reads and writes information directly to an online database server. As such, calling IMessage::SubmitMessage() triggers the backend to do the send of the message without involving the transport provider. Similarly, when new mail arrives it is automatically delivered into the user’s mailbox on the backend, triggering a table update in Outlook. The transport provider isn’t really involved in "downloading" new mail like it would be in a POP or IMAP account. Yet a message service must have a transport provider defined to even be considered an "email account" by Outlook. I imagine that the Exchange providers operate in a similar manner. Could you elaborate on the interaction between the exchange message store and transport provider? I’ve also noticed that exchange includes two transport providers in it’s message service: both a normal and a remote transport provider. How does the remote transport provider interact with the message store?

    Thanks,

    Dan

  11. Jason Nelson says:

    A discussion of the full interaction between tightly coupled providers (like Exchange’s EMSMDB) and offline providers is beyond the scope of this blog. I highly recommend the book "Inside MAPI". While I"m not sure if it’s still in print, it is far and away the best book on this subject (and the sample provider code runs more of the 3rd party providers you might expect).

  12. Dan Christensen says:

    Yeah, I suppose it was wishful thinking to get that kind of response from a blog :) I do have a copy of "Inside MAPI". It is still available from http://www.insidemapi.com . Unfortunately, there is very little material in this book or elsewhere that discusses tightly coupled message store and transport providers.

    The little information that is available talks about the role of the message store in the tightly coupled relationship. Even less (or no) information is available on the role of the transport provider in the tightly coupled relationship.

    Thanks anyway, it was worth a shot :)

  13. Jason Nelson says:

    The reason it isn’t well documented is because it isn’t well defined, that’s the key to being "tightly coupled". That’s a code word for "I can bypass the normal spooler and do things directly if I please". This is why VSAPI hooks via splhook don’t work (I implemented this once). Tightly coupled implies carnal knowledge and shared address space/hierarchy in most cases, making it the easiest implementation by far.

  14. Anonymous says:

    Jason Nelson, one of our core Sustained Engineering guys, just posted an excellent article on the nature…

Comments are closed.

Skip to main content