Named Properties, X-Headers, and You

[The_Exchange_Team]

EDIT 8/10/2010: Please note that this blog post covers namely the history of named properties in Exchange. For updated and current guidance of how to deal with named properties issues, please see this blog post.

Coming soon to a roll up near you is a set of changes to the way that Exchange Server handles x-headers and named properties, specifically named properties that represent x-headers. This represents a change in behavior that has been this way for nearly a decade, so it bears explanation of the old way, the new way, and how it will impact (or not) you. First though, for those new to the table I offer a history of named properties.

Property Archeology

In the beginning was MAPI. Ok, maybe the earth cooled first, and then the dinosaurs came, but you can bet that one of the first problems the dinosaurs faced was a proliferation of ways to communicate, and the need for a common API to "glue" a disparate set of transports together. For Microsoft, that glue API was called MAPI, or "Messaging API". MAPI represents almost everything as a "Property". And how do you know what properties are what? Property Identifiers, or PropIds, for the typing phobic. Property identifiers were a set of numeric values ranging from 1 to the nearly unreachable 0xFFFF. Amazing, really. This vast 64k landscape represented a nearly unlimited area for all the properties a man could ever want. Somewhere, I'm sure Bill Gates reviewed it and said "64k worth of properties ought to be enough for anyone." And all was well.

Now the early architects of MAPI saw fit to divide up the properties into ranges, each range a vast area of properties wherein we could arrange values by logical groups. For instance, we had a range of properties called "Transmittable", meaning that Exchange would pack them up and send them with a message. Another range called "Internal," meaning "You don't set these, we do." We had a range called "Non transmittable," meaning non infectious - the properties simply wouldn't hit the wire when we bundled up a message to send it out into the world.

Name that Property

And right at the middle of the range we drew the biggest line of them all. Properties below 0x8000, those would be fixed properties, for us to assign on the server and apply at will. We thought (and hoped) that other people would want to use MAPI as well. And we didn't want to be in the "Property Assigning Business." Imagine if every time an independent developer wanted to get a MAPI property for his application to use, he had to go through Microsoft? Never mind. Don't imagine that. It's too scary. The nice thing is, they didn't have to. Through the magic of Named Properties, we would create a mechanism by which developers could ask the server to reserve a PropId for their use when deployed. And all was well.

Now named properties came in two major flavors: Properties whose names were actually numbers (like the ones Outlook uses), which are nice when you have a list of them #defined in a source file somewhere, and properties whose names are, well, names. These types are called "String Named Properties", because their names are strings (Logical, is it not?). In addition each named property had a guid associated with it, allowing developers to divide their named properties into "Property Sets", kind of like Monopoly, except that no one would ever land on your space, no matter how many hotels you built unless you gave them your guid and property names. To start people out we even created some nice "starter" namespaces, with enticing names like "PS_PUBLIC_STRINGS." Most people avoided PS_PUBLIC_STRINGS like it was a housing project and rolled their own guids.

Then Came the Internet

MAPI, as you may recall from above (and if you do not, please seek medical attention now), stored everything as property sets. Messages sent on the internet are sent in a format called message/rfc822, a text format designed to be both flexible and nearly impossible to create a canonical parser for. Rfc822 messages are plain text with a set of key and value pairs in their headers. Of course, the people who contributed to rfc822 made a fundamental flaw, and that was thinking that the people interpreting their document would be rational. If an acceptable range of values for a header was "1, 2, or 3," it never occurred to the authors to include lengthy paragraphs explaining that 1, 2, or 3 were good values. 4 was not be included as a value, nor was it only to include 2, unless it also supported three. Five was right out. Now the rfc822 included support for a set of properties called x-headers (for eXperimental, or eXtended). So now we needed a way to communicate between MAPI and rfc822. And we called it Imail. Imail took the key value pairs in headers and translated them (where possible) to MAPI properties. THAT process is the substance of an article I have no desire to write, but keep in mind that Imail converts headers to properties (and vice versa).

The road to Hell

When Exchange 2000 (may it rest in peace) came onto the scene, we rewrote large portions of Imail (a process we would repeat for Exchange 2007). Part of the rewriting included the addition of what were called "Ad-hoc" headers. In other words, headers we would preserve because we thought at some point they might be useful (or just so we could emit them when we vomited forth the message for internet clients). X-headers fell into the bucket of ad-hoc headers, values we would preserve in case someone needed them. As you hopefully recall (because I just covered it, again, see the doctor if not) MAPI properties are identified by "IDs" and X-headers, these could be created by any moron with a text editor. If he added "x-williamrobert:yeeeeeehaaaaaaw" to his message, we'd preserve it. Now, we couldn't possible know the value of every x-header to ever be created (or else I would have immediately done a GetProps() on PR_TOMORROWS_LOTTO_NUMBERS), we used named properties to represent these X-headers. Specifically, we used String Names for the Named properties, because, well, we had the name of the x-header, and as you may have noticed it is a string. So, if an inbound message contained X-headers, we would create named properties to represent those values and store them on the message. In fact we published another "Starter Namespace", PS_INTERNET_HEADERS. The astute among you may immediately derive that properties in this namespace were internet headers. Those of you who could not derive this, leave now, no questions will be asked of you. Even "better", MAPI applications could actually emit X-headers should a message ever be condemned to the internet by setting a tring named property in the PS_INTERNET_HEADERS namespace.

X-haustion

X-headers, as covered before, are a (seemingly random) set of characters that theoretically represent valuable data. Named properties are a method to reserve an ID for a given piece of data. Named properties fall into the range of 0x8000 and up. So there is a definite "upper bound" to the number of named properties that can ever exist on a MDB. Once a named property is allocated, it's gone for good, forever reserved to that name and guid combination. Through the years people have often asked why we can't recover them. The answer is that it's technically possible. If the database were dismounted one could crawl every message, making a list of every property, every guid and every id. One could then combine the list of allocated named properties and determine which ones cannot be in use. In practice this is nearly impossible if you ever want the mail store to be available again.

One of the biggest consumers of named properties is Outlook (who have a nigh endless array of properties and names for them neatly arranged). Should Outlook be unable to map a named property, bad things happen. So we created a quota system by which the STORE process would begin warning you when named property exhaustion was fast approaching. And at a second threshold it would refuse to give out new named property IDs. And we clamped down the number of x-headers we would allow any message to create. Of course, we still allowed some to be created, and that means that "bad things will happen." On a survey I did of real production databases at Microsoft, I was amazed at the sheer amount of crap that was being stored in x-headers. Specifically, we have at least one mailer out there that adds x-<today'sdateandtime>:True to every message. In the world of the internet, where anyone and his dog can create x-headers, allowing an unrecoverable resource to be consumed by anonymous submissions is not acceptable.

Changes Coming

So this is changing. If you like it, good, it's changing. If you don't like it, sorry, it's changing. If it enrages you, seek counseling, and it's still changing. We are putting the clamp down on x-header promotion and putting it hard. When we say that what it actually means is that we are changing the way that new x-headers are promoted. X-headers in E12 are database wide, meaning that if user A maps a named property on a database, user B on that database will get the same ID for the same name and guid combination. If a named property for a give X-header already has been promoted on a database, i.e. we've already created a mapping from x-williamrobert to 0x803f, we will continue to preserve the data for x-williamrobert, and we'll do it in the property with the id 0x803f. If a message comes in with x-billybob, and that header has never been promoted on the database before, the new rules take effect. The new rules control who gets to consume named properties to store x-header values. When the new promotion rules go into effect, many scenarios that once created new named properties to preserve X-header data will not:

Anonymous submissions: Sorry, anonymous mail sender from the internet. I do not know you and you do not get to consume precious resources on my server. If I've already saved off a named property for x-somerandomheader then I don't mind preserving the data, but if not, the value of x-somerandomheader will not be saved as a property on the message.

Embedded messages: This feature was created as an interop feature for MAPI applications to be able to set envelope or top level properties. Embedded messages with your six hundred voice over ip x-headers, no new named properties for you.

Journal Messages: This one will likely prove controversial but journal messages represent the biggest concentration of x-header consumption that is possible. Consider if you have 100 databases in your organization (really 101). Each MDB is exposed to 100 new x-headers from the internet per week. The journal mdb will be subject to 100x100 new x-headers a month. It will go through them faster than my son does a bag of microwave popcorn. Journal messages will not promote new x-header mappings. Journal content - that still gets journaled. Journal reports, yep, still saved. Your messages? Still in the journal mailbox. X-headers being mapped to named properties where they did not exist before: No.

So who gets to do this?

Well, obviously this doesn't remove the ability to map x-headers entirely. That's not a bad idea, but it isn't the case at the moment. Authenticated users can still create messages whose x-headers will be promoted. MAPI applications can also create named properties (and should those lie in the PS_INTERNET_HEADERS namespace, those would be x-headers). EWS applications using property mappings can create named properties which might be x-headers. Remember - once one person maps the named property on a database, the property will then be preserved for any inbound message. What we are controlling is who gets to decide to consume the named property.

If you have an application that consumes x-headers from MAPI, be not afraid, for I bring you tidings of "your application will still work." If you wrote your application correctly then it already asks for the ids of the named property combinations (PS_INTERNET_HEADERS and "x-whateverstringitisyouareafter"). And if you are using the MAPI_CREATE flag on that request, your application will automatically add the x-header to the list of x-headers to preserve on a database. Similarly, if your application was setting an x-header from MAPI by creating a named property mapping in the PS_INTERNET_HEADERS namespace, then you are already creating a mapping for that X-header, and it will be preserved inbound (and emitted outbound).

If your application does not use MAPI_CREATE when asking for named property ids, and you are asking for names that fall in PS_INTERNET_HEADERS, and it is not a property that any well written client would already have asked for, your application will break. At least until the first authorized (or MAPI) client asks for that x-header by name. Then it works.

Conclusions

This is the sort of change we hate to produce in mid product cycle. It's the kind of change people hate to hear about, even if it won't affect them one whit. It's the kind of change that leads to phantom support calls as "Well, my application relies on property x-contoso, and I can't be bothered to map it. I swear it wasn't here before, and now it is." For Exchange 14 we are making further changes in this same area. We're moving the named property resources to be at a mailbox level, not a server level, and including this change as part of the product baseline.

Jason Nelson