Public Folder Replication Troubleshooting – Part 4: Exchange Server 2007/2010 tips


Two years ago, I posted a three-part series on troubleshooting public folder replication. Part 1 discussed the replication of new data, Part 2 discussed the replication of existing data, and Part 3 discussed the replica deletion process and some common problems we saw with Exchange 2003. With this post, I want to update the series for Exchange 2007.

In Exchange 2007, public folder replication works basically the same way it always has. The troubleshooting steps in the first three parts of the series all still apply. However, the admin tools have changed and the common problems we see with Exchange 2007 are a little different, so that's what I want to cover here.

Changes in the Admin Tools

The event log is still your best tool for narrowing down a replication problem to a particular point of failure. In Part 1, I suggested turning up logging on Replication Incoming and Replication Outgoing to Maximum. That still applies, except that with Exchange 2007 you'll be using the Set-EventLogLevel cmdlet to set "MSExchangeIS\9001 Public\Replication Incoming Messages" and "MSExchangeIS\9001 Public\Replication Outgoing Messages" to the Expert level.

In Part 2, I described how to use the Synchronize Hierarchy and Synchronize Content options in ESM to force a status message and to timeout all outstanding backfill entries. You can still do this in Exchange 2007 via the Update-PublicFolderHierarchy and Update-PublicFolder cmdlets. These are also available in Sp1's public folder management tool, appearing as Update Hierarchy when the Public Folders root is selected and Update Content when a particular public folder is selected. Because you can use these from the command line, they are a lot more flexible than the Exchange 2003 options. For instance, it's now very simple to time out backfill entries for every folder that has a replica on your Exchange 2007 server with a simple "Get-PublicFolderStatistics | Update-PublicFolder" command. That wasn't possible in Exchange 2003 without a lot of clicking.

In Part 3, I described how to use the Public Folder Instances view to see if the deletion of a replica has completed. In Exchange 2007, you use the Get-PublicFolderStatistics command to see that same information.

Common Problems in Exchange 2007

The most common symptom I've seen so far is a store driver failure. For instance, a backfill response will be sent to an Exchange 2007 server, but if you look at the application log on the 2007 side you never see the incoming replication event. Message tracking shows that the replication message got to the hub transport server and then failed in the store driver.

This can happen for a number of reasons, and fortunately it usually is not that hard to troubleshoot. Your best troubleshooting approach in this case is to use the Pipeline Tracing and Content Conversion Tracing available on the hub transport server. If you run "Get-TransportServer | fl" you'll see a few settings related to this:

PipelineTracingEnabled : False
ContentConversionTracingEnabled : False
PipelineTracingPath : C:\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\PipelineTracing
PipelineTracingSenderAddress : SERVER01-IS@contoso.com

To find out why your backfill response is failing in the store driver, set the PipelineTracingSenderAddress to match the SMTP address of the public folder store that's sending the backfill response. Then set ContentConversionTracingEnabled to $true and PipelineTracingEnabled to $true, and reproduce the problem. After doing so, have a look in the folder specified by the PipelineTracingPath. You should find a subfolder called ContentConversionTracing, and an InboundFailures folder within that. In the InboundFailures folder, you'll have an EML file containing the replication message itself and a TXT file containing some information on the failure. The top of the TXT file will often give you a useful clue as to the reason for the failure.

For instance, in a few cases we've seen this output in the TXT file:

Microsoft.Exchange.Data.Storage.PropertyValidationException: Property validation failed. Property = [{00020329-0000-0000-c000-000000000046}:'Keywords'] Categories
Error = Element 0 in the multivalue property is invalid..

In this case it's complaining about the Categories property. This will happen if the public folder in question contains items on which the Categories property is set to be blank - such as with a single space - instead of truly being empty. You can see this by going into Outlook and choosing to view items by Category. You should find that there are two different sets of items with "None". To correct the problem, simply clear the Category on all of the "None" items using Outlook. You may have to set them to some other category and then set them back to None. Once the Categories property is truly clear, you'll only have one set of items that show "None", and the items will replicate successfully.

Another example:

Microsoft.Exchange.Data.Storage.PropertyValidationException: Property validation failed. Property = [{00062004-0000-0000-c000-000000000046}:0x8092] Email2AddrType
Error = Email2AddrType is too long: maximum length is 9, actual length is 35..

In this case its flagging the Email2AddrType property as a problem. We discovered that it had somehow been populated with the full email address on certain contacts, instead of containing only the address type that it normally should, such as 'SMTP' or 'EX'. Fixing that property allowed the items to replicate.

Sometimes the store driver will fail in a manner where it does not identify a specific problem property, such as in this output:

Microsoft.Exchange.Data.Storage.ConversionFailedException: Message content has become corrupted. ---> Microsoft.Exchange.Data.Storage.ConversionFailedException: Content conversion failed due to corrupt TNEF (violation status: 0x00000800)

This is what the failure looks like when you hit the problem described in KB 936000. Applying the fix to the Exchange 2003 server that's generating the replication message will correct that problem.

The important thing to take away from this is that Exchange 2007 does a lot of property validation to keep bad data from getting into the store. Although that's a good thing, it can keep public folder data from replicating from your Exchange 2003 servers until problems with the content are corrected on the 2003 server. ContentConversionTracing can help you identify these problems, and will often point you to the exact property causing the problem.

There's one more common problem you can identify with ContentConversionTracing, but it isn't caused by any actual problem with the content. The TXT file in the InboundFailures folder will look like this:

Microsoft.Exchange.Data.Storage.ConversionFailedException: The content conversion limit has been exceeded.
at Microsoft.Exchange.Data.Storage.InboundMimeConverter.ConvertToItem(MimePromotionFlags promotionFlags)
at Microsoft.Exchange.Data.Storage.ItemConversion.InternalConvertAnyMimeToItem(Item itemOut, EmailMessage messageIn, InboundConversionOptions options, MimePromotionFlags promotionFlags, Boolean isStreamToStream)
at Microsoft.Exchange.Data.Storage.ItemConversion.ConvertAnyMimeToItem(Item itemOut, EmailMessage messageIn, InboundConversionOptions options)

InboundConversionOptions:
- preferredCharset: iso-8859-1
- trustAsciiCharsets: True
- isSenderTrusted: False
- imceaResolveableDomain: contoso.com
- preserveReportBody: False
- clearCategories: True
- userADSession:
- recipientCache: Microsoft.Exchange.Data.Directory.Recipient.ADRecipientCache
- clientSubmittedSecurely: False
- serverSubmittedSecurely: False

ConversionLimits:
- maxMimeTextHeaderLength: 2000
- maxMimeSubjectLength: 255
- maxSize: 2147483647
- maxMimeRecipients: 12288
- maxRecipientPropertyLength: 1000
- maxBodyPartsTotal: 250
- maxEmbeddedMessageDepth: 30
- exemptPFReplicationMessages: True

First, notice the top line says, "The content conversion limit has been exceeded." Normally a public folder replication message is exempt from size limits and such, so why would this message be failing in this way? Notice that "isSenderTrusted" is False. This means the message came over an SMTP connection that was not authenticated. The sending server either failed to authenticate, or never even tried. This is very similar to what I described in the Common Problems section of Part 3, where an authentication failure would cause the XEXCH50 verb to fail in Exchange 2003. Because the sending server did not authenticate, the Exchange 2007 server applies the normal size limits to the message. If this is a content replication message with more than 250 attachments (not uncommon for a content backfill response), then it fails because it exceeds the limit. This often happens because an administrator has created a second Receive Connector which does not allow authentication, but has it configured such that it listens on the internal IP rather than the external IP. If that's not the cause, you may need to use a Netmon capture to identify the problem as I described in Part 3.

Conclusion

That should cover everything you need to know to narrow down public folder replication problems in an environment with Exchange 2007. All the old troubleshooting steps still apply; we just have different administrative tools and a different set of problems. Hope this helps!

- Bill Long

Comments (13)
  1. Oliver says:

    Hello,

    VERY good article!

    I have this replication error you described in your blog – problems with categories.

    Your solution – log on with outlook and change the wrong categories wont work for me!

    There are 2 exchange servers (one old exchange 2000, and one exchange 2007) – how can i logon to the public store on the exchange 2000 server?

    Or is it possible to "ignore" these error in the registry?

    TiA

    Oliver

  2. Priit says:

    Hello, thanks for this very good article, it was exactly what i was looking for. My old server sends the backfill response and it is received on the new e2k7 smtp but according to exchange message tracking it fails with:

    554 5.6.0 STOREDRV.Deliver; Corrupt message content

    or

    554 5.6.0 STOREDRV.Deliver.Exception:ObjectValidationException; Failed to process message due to a permanent exception

    So the debugging steps you described helped me futher. Here is what I get from ContentConversionTracingInboundFailures:

    Microsoft.Exchange.Data.Storage.ObjectValidationException: The object is invalid and cannot be saved.

      at Microsoft.Exchange.Data.Storage.StoreObject.ValidateStoreObject()

    Any ideas what might cause the exception? Or how do I futher investigate this?

    As a solution to other cases you suggest to edit the items on the old server. How exactly should we gain access to these items? Outlook shows me the pf store on the new server. Should I make a temporary account into the mailbox store of the old server and reassociate this mailbox store with this public folder store?

    Thanks in advance!

  3. Bill Long says:

    Oliver: To log on to the public store on the Exchange 2000 server, you’ll need to log on to a mailbox where the mailbox store points to that public store as its default (in ESM, properties of the mailbox database, General tab, "Default public store"). As long as there’s a replica of the folder on that server, the mailbox should access the public folder on its default public store. It’s not possible to ignore the errors in the registry. If the solution I suggested in the article doesn’t work, it’s possible there some other problem with the categories on those messages… though I can’t imagine what that would be. If all else fails, call in and open a case and I’m sure we can figure it out.

    Priit: For an ObjectValidationException, sometimes you’ll find a little more detail in the TXT file, such as:

    Invalid property = [Calc:EndTime] EndTime. Invalid data for that property =

    8/3/2007 3:30:00 PM. Constraint violated = Property [Calc:StartTime] StartTime must

    be LessThanOrEqual when compared to property [Calc:EndTime] EndTime…

    at Microsoft.Exchange.Data.Storage.StoreObject.ValidateStoreObject()

    In a case like that, it’s telling you that the Start and End times on the appointment don’t make sense. Are these appointment items?

    Yes, as far as accessing the public store on the old server, you’ll need to make a mailbox on the old mailbox store that points to the old public store. Logging on to that mailbox, you’ll be able to edit the items, and typically that will correct the problem.

  4. Priit says:

    Bill, thanks for quick feedback. Yes, these are appointment items, but I didn’t find the invalid date issue. I’ll post the complete text file here:

    Microsoft.Exchange.Data.Storage.ObjectValidationException: The object is invalid and cannot be saved.

      at Microsoft.Exchange.Data.Storage.StoreObject.ValidateStoreObject()

      at Microsoft.Exchange.Data.Storage.Item.SaveInternal(SaveMode saveMode, Boolean commit)

      at Microsoft.Exchange.Data.Storage.InboundTnefConverter.ParseAttachDataObject(TnefPropertyReader propertyReader)

      at Microsoft.Exchange.Data.Storage.InboundTnefConverter.ParseTnefProperty(TnefPropertyReader propertyReader, Boolean forceTransmittable)

      at Microsoft.Exchange.Data.Storage.InboundTnefConverter.ParseAttachmentProperties()

      at Microsoft.Exchange.Data.Storage.InboundTnefConverter.ReadAttribute()

      at Microsoft.Exchange.Data.Storage.InboundTnefConverter.ConvertToItem(TnefReader reader, InboundMessageWriter targetWriter, InboundConversionOptions options, ConversionLimitsTracker limitsTracker, String correlationKey, Boolean isSummaryTnef)

      at Microsoft.Exchange.Data.Storage.InboundTnefConverter.ConvertToItem(Stream tnefStream, InboundMessageWriter targetWriter, InboundConversionOptions options, ConversionLimitsTracker limitsTracker, String correlationKey, Boolean isSummaryTnef)

      at Microsoft.Exchange.Data.Storage.InboundMimeConverter.<>c__DisplayClass16.<PromoteTnef>b__10()

      at Microsoft.Exchange.Data.Storage.ConvertUtils.CallCts(Trace tracer, String methodName, String exceptionString, CtsCall ctsCall)

      at Microsoft.Exchange.Data.Storage.InboundMimeConverter.PromoteTnef(InboundTnefConverter tnefconverter, MimePart tnefPart, Boolean isSummaryTnef)

      at Microsoft.Exchange.Data.Storage.InboundMimeConverter.ConvertToItemInternal(MimePromotionFlags promotionFlags)

      at Microsoft.Exchange.Data.Storage.ConvertUtils.CallCts(Trace tracer, String methodName, String exceptionString, CtsCall ctsCall)

      at Microsoft.Exchange.Data.Storage.InboundMimeConverter.ConvertToItem(MimePromotionFlags promotionFlags)

      at Microsoft.Exchange.Data.Storage.ItemConversion.InternalConvertAnyMimeToItem(Item itemOut, EmailMessage messageIn, InboundConversionOptions options, MimePromotionFlags promotionFlags, Boolean isStreamToStream)

      at Microsoft.Exchange.Data.Storage.ItemConversion.ConvertAnyMimeToItem(Item itemOut, EmailMessage messageIn, InboundConversionOptions options, MimePromotionFlags flags)

    InboundConversionOptions:

    – preferredCharset: iso-8859-1

    – trustAsciiCharsets: True

    – isSenderTrusted: True

    – imceaResolveableDomain: domain.de

    – preserveReportBody: False

    – clearCategories: True

    – userADSession:

    – recipientCache: Microsoft.Exchange.Data.Directory.Recipient.ADRecipientCache

    – clientSubmittedSecurely: False

    – serverSubmittedSecurely: True

    ConversionLimits:

    – maxMimeTextHeaderLength: 2000

    – maxMimeSubjectLength: 255

    –  maxSize: 2147483647

    – maxMimeRecipients: 12288

    – maxRecipientPropertyLength: 2000

    – maxBodyPartsTotal: 250

    – maxEmbeddedMessageDepth: 30

    – exemptPFReplicationMessages: True

    Btw, for some other folders I got a punch of "Content conversion failed due to corrupt TNEF" errors. I requested the KB936000 patch and applied it to e2k3 and tried the "Update content" again on e2k7, but I still get the same TNEF errors. What am I doing wrong?

    TIA

    Priit

  5. Oliver Schluga says:

    Hello Bill,

    thanks for your response – your solution works great!

    please dont stop writing these technical articles!

    Greetings

    Oliver

  6. Russ M says:

    I too am getting the ‘object is invalid’ exception.  I believe these are coming from old recurring appointments, but am unsure how to proceed from here.  If there was a way to identify the items in error I could correct them.  As above, here is the complete text.

    Microsoft.Exchange.Data.Storage.ObjectValidationException: The object is invalid and cannot be saved.

      at Microsoft.Exchange.Data.Storage.StoreObject.ValidateStoreObject()

      at Microsoft.Exchange.Data.Storage.Item.SaveInternal(SaveMode saveMode, Boolean commit)

      at Microsoft.Exchange.Data.Storage.InboundTnefConverter.ParseAttachDataObject(TnefPropertyReader propertyReader)

      at Microsoft.Exchange.Data.Storage.InboundTnefConverter.ParseTnefProperty(TnefPropertyReader propertyReader, Boolean forceTransmittable)

      at Microsoft.Exchange.Data.Storage.InboundTnefConverter.ParseAttachmentProperties()

      at Microsoft.Exchange.Data.Storage.InboundTnefConverter.ReadAttribute()

      at Microsoft.Exchange.Data.Storage.InboundTnefConverter.ConvertToItem(TnefReader reader, InboundMessageWriter targetWriter, InboundConversionOptions options, ConversionLimitsTracker limitsTracker, String correlationKey, Boolean isSummaryTnef)

      at Microsoft.Exchange.Data.Storage.InboundTnefConverter.ConvertToItem(Stream tnefStream, InboundMessageWriter targetWriter, InboundConversionOptions options, ConversionLimitsTracker limitsTracker, String correlationKey, Boolean isSummaryTnef)

      at Microsoft.Exchange.Data.Storage.InboundMimeConverter.<>c__DisplayClass16.<PromoteTnef>b__10()

      at Microsoft.Exchange.Data.Storage.ConvertUtils.CallCts(Trace tracer, String methodName, String exceptionString, CtsCall ctsCall)

      at Microsoft.Exchange.Data.Storage.InboundMimeConverter.PromoteTnef(InboundTnefConverter tnefconverter, MimePart tnefPart, Boolean isSummaryTnef)

      at Microsoft.Exchange.Data.Storage.InboundMimeConverter.ConvertToItemInternal(MimePromotionFlags promotionFlags)

      at Microsoft.Exchange.Data.Storage.ConvertUtils.CallCts(Trace tracer, String methodName, String exceptionString, CtsCall ctsCall)

      at Microsoft.Exchange.Data.Storage.InboundMimeConverter.ConvertToItem(MimePromotionFlags promotionFlags)

      at Microsoft.Exchange.Data.Storage.ItemConversion.InternalConvertAnyMimeToItem(Item itemOut, EmailMessage messageIn, InboundConversionOptions options, MimePromotionFlags promotionFlags, Boolean isStreamToStream)

      at Microsoft.Exchange.Data.Storage.ItemConversion.ConvertAnyMimeToItem(Item itemOut, EmailMessage messageIn, InboundConversionOptions options, MimePromotionFlags flags)

    InboundConversionOptions:

    – preferredCharset: iso-8859-1

    – trustAsciiCharsets: True

    – isSenderTrusted: True

    – imceaResolveableDomain: domain.com

    – preserveReportBody: False

    – clearCategories: True

    – userADSession:

    – recipientCache: Microsoft.Exchange.Data.Directory.Recipient.ADRecipientCache

    – clientSubmittedSecurely: False

    – serverSubmittedSecurely: True

    ConversionLimits:

    – maxMimeTextHeaderLength: 2000

    – maxMimeSubjectLength: 255

    –  maxSize: 2147483647

    – maxMimeRecipients: 12288

    – maxRecipientPropertyLength: 2000

    – maxBodyPartsTotal: 250

    – maxEmbeddedMessageDepth: 30

    – exemptPFReplicationMessages: True

  7. Ian Darke says:

    I’ve been trying to troubleshoot my replication problems between my old Exchange 2003 server that I’m hoping to decomission and the new 2007 server.

    I’m getting these messages in the ContentConversion logs:

    Microsoft.Exchange.Data.Storage.ConversionFailedException: Content conversion: Corrupt summary tnef content.

      at Microsoft.Exchange.Data.Storage.InboundMimeConverter.GetMimeMessageClass()

      at Microsoft.Exchange.Data.Storage.InboundMimeConverter.ConvertToItemInternal(MimePromotionFlags promotionFlags)

      at Microsoft.Exchange.Data.Storage.ConvertUtils.CallCts(Trace tracer, String methodName, String exceptionString, CtsCall ctsCall)

      at Microsoft.Exchange.Data.Storage.InboundMimeConverter.ConvertToItem(MimePromotionFlags promotionFlags)

      at Microsoft.Exchange.Data.Storage.ItemConversion.InternalConvertAnyMimeToItem(Item itemOut, EmailMessage messageIn, InboundConversionOptions options, MimePromotionFlags promotionFlags, Boolean isStreamToStream)

      at Microsoft.Exchange.Data.Storage.ItemConversion.ConvertAnyMimeToItem(Item itemOut, EmailMessage messageIn, InboundConversionOptions options, MimePromotionFlags flags)

    InboundConversionOptions:

    – preferredCharset: iso-8859-1

    – trustAsciiCharsets: True

    – isSenderTrusted: True

    – imceaResolveableDomain: zog.com

    – preserveReportBody: False

    – clearCategories: True

    – userADSession:

    – recipientCache: Microsoft.Exchange.Data.Directory.Recipient.ADRecipientCache

    – clientSubmittedSecurely: False

    – serverSubmittedSecurely: True

    ConversionLimits:

    – maxMimeTextHeaderLength: 2000

    – maxMimeSubjectLength: 255

    –  maxSize: 2147483647

    – maxMimeRecipients: 12288

    – maxRecipientPropertyLength: 2000

    – maxBodyPartsTotal: 250

    – maxEmbeddedMessageDepth: 30

    – exemptPFReplicationMessages: True

    I just applied KB93600 but to no avail. Any more suggestions?

    I noticed that there is a SP3 for Exchange 2003 out.. worth trying?

    Should I just give up and try and archive to a PST all of the folders that are missing data that I can find?

  8. Bill Long says:

    For anyone who has run into an "ObjectValidationException" as shown in Priit’s example and Russ’s example above, we’ve discovered this can happen if you have appointment items with inconsistent date properties. We use several different properties to determine the start and end times for appointments, and if those properties are not in sync with each other, this error will occur. You can fix the problem by opening the appointment with Outlook, changing start or end time, and saving it. Unfortunately, the error doesn’t tell you which item is the problem, so it could be a pain to track it down. There may be other causes for this error, but so far I’ve only had one such case come to me, and that was the problem in that case.

    Ian, I haven’t seen the "corrupt summary TNEF content" before. I did find a couple of previous cases with this error. In one case a third-party application running on an Exchange 2003 hub server was modifying messages on their way to Exchange 2007. In another, a third-party scheduling application was creating some invalid meetings.

    If you call in and open a case on it I’m sure we can track it down. Of course, I can’t promise the solution will be any better than going the PST route, but we should at least be able to find the cause of the failure and document it appropriately.

  9. Ian Darke says:

    Bill,

    Thanks for the info again. Unfortunately, I don’t think either of those scenarios is relevant in my case.

    I’ve only had fairly limited interactions with MS support so please excuse/correct me if my understanding is incorrect.

    I run a small business environment @ home so that I can play with newer technologies in a less risky production environment. I don’t think that my Action Pack gets me the ability to open up a case without paying $s. I’m willing to invest my time to help trace down the issue, but I’m not willing to spend $s to trace down bugs in MS software.

    I do also have a MSDN Universal subscription through my work that probably has some incidents available, but I’m not sure I want to burn them on this.

    Any suggestions?

  10. Exchange says:

    Ian Darke,

    Bill will probably reply to you too but I wanted to address one part of your last comment:

    It is Microsoft’s policy that if the support case that you open ends up being a bug in our software, the cost of support incident will be refunded to you. But you do still pay for the case to be opened as there is no way to know that this is going to be the case up front.

  11. Bill Long says:

    Hey Ian,

    As the Exchange team pointed out, if the problem is caused by a bug in our software, then you are refunded the cost of the incident. Of course, we don’t know ahead of time whether it will be a bug or not. In the previous cases I found, that particular error was being caused by third-party products modifying or creating content in strange ways rather than by a bug in Exchange.

    If you’d rather not open a case, your best bet is probably the PST route.

  12. Russ M says:

    Thanks for the response.  I had already come to the same conclusion.  I was having trouble comparing and re-saving some of the items in Outlook.  The solution I have been using now (although cumbersome) is to use OWA, one on 2003 and another on 2007, to compare the calendars and open/change/re-save any items that are inconsistent.  I don’t know how far this will get us becuase it is a lot of work wih no guarantees.  We are currently working on a strategy to migrate away from public folders and avoid the issue altogether.  I like the suggestion of using a PST to transition, I may look into that a bit further.

  13. Alan.Gim says:

    Hi, Bill. About your example 1 on e2k7:

    [{00020329-0000-0000-c000-000000000046}:’Keywords’] Categories

    Error = Element 0 in the multivalue property is invalid

    I got similar error here with your method:

    = [{00020329-0000-0000-c000-000000000046}:’Keywords’] Categories

    Error = Element 1 in the multivalue property is invalid

    Can you give advice on it? Appreciated!!

Comments are closed.

Skip to main content