Fix available to alleviate event ID 9548


No doubt many of you are familiar with the event commonly seen in application logs of Exchange 2000 and 2003 servers, MSExchangeIS event 9548, indicating that the information store came across a disabled user who is missing the msExchMasterAccountSid attribute while processing some various task.  There are many KB articles associated with this event, such as 291151, 326990, 278966, 328880, 316047, and possibly more.  There have also been countless support cases where this design was at least a contributing factor.  Almost every application log from an Exchange 2000 or 2003 server ever seen has likely been littered with 9548 events, to the point where it has become an annoyance event.  For additional information on the event and the related information, I suggest reading this article: http://www.msexchange.org/articles/NoMAS-Tool.html.


 


A CDCR (Critical Design Change Request) was accepted last year to resolve the issue without having to run tools or scripts, and the first version of this fix was released last week.  The KB article is 903158.


 


The problem was that a decision was made during the development of Exchange 2000 that every disabled user account with a mailbox had to have the msExchMasterAccountSid attribute.  This is because in order for a mailbox to function within the store, a SID must be associated with the mailbox.  The logic worked like this:


 


1. If the user account associated with the mailbox is disabled, and has msExchMasterAccountSid, AND msExchMasterAccountSid is not the well-known SELF SID, then msExchMasterAccountSid is the SID that is associated with the mailbox.


 


2. If the user account is enabled, or if it is disabled and has msExchMasterAccountSid set to the well-known SELF SID, then use objectSid.


 


3. If the user account is disabled, and has no msExchMasterAccountSid, OR if we cannot tell if the user is enabled or disabled due to access control on the user object, or if we cannot read the objectSid due to access control, fail the operation and log the 9548 event.


 


In the 3rd case, the vast majority of the 9548 events came from the first part – that is, almost all 9548 events are due to disabled user accounts which are missing msExchMasterAccountSid.  The new logic works as follows:


 


1. If the user account associated with the mailbox is disabled, and has msExchMasterAccountSid, AND msExchMasterAccountSid is not the well-known SELF SID, then msExchMasterAccountSid is the SID that is associated with the mailbox.  (NOTE: No change here)


 


2. If the user account is enabled, or if it is disabled and has msExchMasterAccountSid set to the well-known SELF SID, OR if the user is disabled has does not have msExchMasterAccountSid, then use objectSid.  (Big change here)


 


3. If we cannot tell if the user account is enabled or disabled due to access control on the user object, or if we cannot read the objectSid due to access control, fail the operation and log the 9548 event.


 


So now, the only way you will get the 9548 event is due to a real problem with the user account associated with a mailbox.


 


Alex Seigler

Comments (47)
  1. Scott Bueffel says:

    Thank you, thank you, thank you.  I used to have a one-off script to set the attribute for disabled accounts (since we wanted the mailboxes to still function) until NoMAS came along.  Actually, I have always preferred MsAccntSidFixer, and I use it to this day.  Now I will be able to stop doing that once applied.  Thank you so much.

    Scott.

  2. Anonymous says:

    I’m sure a lot of you have seen event 9548 in your event logs from time to time.

    If 9548 isn’t ringing a bell, maybe this example text will:

    Disabled user /o= Organization Name /ou= Administrative Group Name /cn=Recipients/cn= Computer Name does not

  3. Deji says:

    Well done.

    Now, let’s examine #2 more closely:

    You wrote: If the user account is enabled …then use objectSid.

    No problem with that.

    You wrote:

    if it is disabled and has msExchMasterAccountSid set to the well-known SELF SID…then use objectSid

    My question: Which ObjectSID? The disabled account’s SID or SELF?

    You wrote:

    if the user is disabled has does not have msExchMasterAccountSid…then use objectSid

    Again, a question:

    Which ObjectSID? The disabled account’s SID or SELF?

    The last 2 descriptions are not clear (to me) because, in one instance, there is MAS, and in another, there isn’t. So, which SID are we using? The way I read KB903158 is that Exchange will always assume that MAS is equal to SELF SID and will ALWAYS use SELF SID even when there is no MAS associated with the disabled account. But your description does not make this clear when you mentioned objectSID.

  4. Deji says:

    Forget what I wrote, ladies and gentlemen. The effects of the morning coffee just set in :)

  5. Alex Seigler says:

    For both questions, the answer is the objectSID of the disabled user account.  When the store sees "SELF", it grabs the objectSID and uses that (because that is what SELF means).

    So, for disabled user, if MAS is set to SELF, or if it is not set at all, we use the objectSID of the disabled user.  The only time we would use the SID in MAS as-is is when the attribute is populated and not SELF (like an NT4 SID or a SID from a different forest).

    -aseigler

  6. JamesK says:

    I think I am missing something. I got the hotfix from support. When I run the hotfix it says: "You can install this hotfix on Service Pack 1 only".

    I didn’t see mention of this limitation anywhere.

  7. Beautiful, Andy.  The KBarticle indicates a pre-SP2 version, though, with a February date.  Will there be a post-SP2 version?

  8. That should read "Alex", of course…

  9. aseigler says:

    Correct, the build available right now is for SP1 only.  The fix for SP2 servers will be out very shortly.

    -aseigler

  10. Kevin says:

    Thankyou.

    This has been a major annoyance for us when running daily backup jobs, as backup exec always complains when it finds a mailbox without the SELF permission thats associated to a disabled user. even just one disabled user and the backup job reports as having failed. Thankyou again for finally fixing this up.

  11. Ilja Summala says:

    5 years of waiting…finally. Way things are going AD and Exchange might start to be better together after all.

  12. Michel says:

    Hmm, have to wait until the PostSP2 version arrives, but since we’ve waited a long time for this a few more days/weeks don’t mind.

    I would like a peek on the ‘Accepted CDCR’ list….

  13. Dave Trevallion says:

    Quick question, if we apply the pre SP2 fix onto a SP1 build is it likely that we will then have to apply the post SP2 fix after we apply SP2.

    We are just planning our SP2 rollout and need to decide to apply the fix now or wait for the post SP2 hotfix version.

    Very pleased to see this fix though as it’s a big issue to us

  14. aseigler says:

    Dave Trevallion, that is correct.  I expect the SP2 version to be released any day now.

    -aseigler

  15. Veno Mouse says:

    Hmm. ADModify.Net has always worked well in the past. Maybe it’s just not well known.

  16. Jewel says:

    Total: 6 objects. 5 succeeded, 1 failed.

    Copy Exchange Files

    Completed

    Status: Completed.

    Elapsed Time: 00:34:16

    Organization Preparation

    Completed

    Status: Completed.

    Elapsed Time: 00:20:15

    Bridgehead Server Role

    Completed

    Status: Completed.

    Elapsed Time: 00:02:00

    Mailbox Server Role

    Completed

    Status: Completed.

    Elapsed Time: 00:04:16

    Client Access Server Role

    Completed

    Status: Completed.

    Elapsed Time: 00:01:32

    Unified Messaging Server Role

    Failed

    Installing product E:servereni386SetupServerRolesUnifiedMessagingesenus32.MSI failed. Fatal error during installation. Error code is 1603.

    Fatal error during installation

    Elapsed Time: 00:14:22

  17. Athif says:

    How come the fix is released only for SP1?!!

  18. Alex Seigler says:

    Please read the previous comments, the SP2 fix will be available very soon.

    -aseigler

  19. Yonkey says:

    This wont work on Exchange 2000 SP3 correct?

  20. aseigler says:

    Yonkey:

    Correct, there are no plans at this time to release a fix for Exchange 2000.

    -aseigler

  21. Matt says:

    Alex,

    PSS needs to get it’s act together then. They’ve just told me four times that the fix is included in SP2 (including after talking to an "engineer")

    How are we meant to find out that the SP2 patch is still on its way when PSS don’t know and neither the KB article nor this blog entry say so?

  22. Dan Sheehan says:

    Well done Alex, you are as usual an exceptional coder!

    And I suspected Mr.NOMAS wrote this hot fix when I saw the KB article independant of this post. :)

    Please keep the good mods coming!

    Dan.

  23. Alan says:

    Okay… what I don’t understand is. How am I supposed to get this Hotfix? Why isn’t it just "available" like every other hotfix Microsoft has ever released. Am I supposed to pay $250 to call PSS so I can get this one hotfix… that’s rediculous!! Can’t someone just send it to me, I have Exchange 2003 SP1.

  24. Exchange says:

    Alan,

    if you call us and ask for the hotfix up front – you will not be charged for the call.

    There is actually a pretty good reason for you having to call in for this: this is a hotfix and as such, it did not go through as much testing as rollups or service packs do. So – by calling in, you give us your contact information. If there is a regression or a problem found in the hotfix later, we have a way to contact you and tell you about it and what to do. So – I’d suggest you call in. :)

  25. Anonymous says:

    Today this list is longer, since I hadn’t the time to post it last week. Sorry…

    Exchange on NAS:…

  26. Shannon says:

    PSS do seem to think this fix is *included* in SP2. Apparently, the keyword "kbexchange2003presp2fix" in the KB article means to them that "this fix is included in SP2".

    Could the KB be updated to include a definitive statement about SP2 support?

    I await the SP2 version with bated breath!

  27. White Ninja says:

    BEWARE of hotfix 903158 if you are using BlackBerry handheld devices in your environment.

    I applied 903158 on March 24th and it broke the ability for users to send email from their handheld devices — everything else seemed to work fine; running E3SP1 and BES 4.0.3.11.

    KB 912918 did not fix the problem, had RIM and Rogers — Service Provider on the horn for 8 hours attempting to resolve. End result, removed the hotfix.

    RIM suggested upgrading to 4.0.4 or 4.1; though could not confirm if this would solve the problem.

    Still waiting to hear back from RIM SE’s for a workaround or fix.

    BES = POS

  28. Alex Seigler says:

    The fix has now been released for SP2.  The fix is KB916783.  The article is not ready, but the content will be identical to 903158.

    Thanks,

    -aseigler

  29. Jad says:

    Thanks for the update. Any comments on the message by White Ninja for BES compatibility ?

    Thanks !

  30. aseigler says:

    I heard of one other instance of BES issues, and it was resolved with 912918.  This fix definitely falls into the category in the "Cause" section of 912918.

    It doesn’t explicitly say so in 912918, but I believe that in order for the changes it recommends to take affect immediately, you must restart the store.  Otherwise, you must wait for the cache to flush, similar to what is discussed in 179065 and various other articles.

    -aseigler

  31. jdonaldson says:

    Installed the SP2 version with no problem for my Blackberry users.  I had previously gone through the steps in 912918 and am running BES 4.0.4.

    -jdonaldson

  32. BGT says:

    Where can you get the Post SP2 fix?  I have tried to download it from premier support but it states that it is not currently available.

  33. aseigler says:

    BGT:  You have to contact support and request KB916783.  It is not available for general download.

    -aseigler

  34. OliverM says:

    Hi there,

    Hi you guys at M$ – just so you know this patch breaks Blackberry Enterprise Server…

    I’m guessing you need to finetune it some more or Blackberry have some work to do.

    Oliver

  35. vwebb says:

    attempts to contact PSS for 916783 have failed spectacularly. is there a secret password i can use to get the hotfix? thanks!

  36. Exchange says:

    vwebb,

    What happened? There is no secret password needed for you to get this patch :). When you call in, just be clear that all you need was a hotfix. I am not sure what happened but maybe try again?

  37. vwebb says:

    called into PSS support noted i needed a hotfix for exchange 03, gave the article number. the liasons(?) were adamant on a few different occasions the aritcle couldn’t be found and without an article there is no hotfix. even tried sighting this blog but no one would bite.

  38. Oldeman says:

    When I installed the hotfix (E2003 SP1, Bes 3.6), Blackberry users could no longer send.  KB912918 doesn’t help because the BES Administrator already had Send As permission. I removed BES Admin and added using the KB article, but there was no change.

  39. Oldeman says:

    When I installed the hotfix (E2003 SP1, Bes 3.6), Blackberry users could no longer send.  KB912918 doesn’t help because the BES Administrator already had Send As permission. I removed BES Admin and added using the KB article, but there was no change.

  40. ff says:

    Hi Oldeman,

    Ensure that the BES admin account does not have a deny on send-as.  Also where is do you see that he has the send-as permission? Examine one of your users and look at the security tab and see if he has the send-as permission.  Also make sure it is set to apply to user objects.

    ff

  41. Paradigm says:

    Yup i installed it on my 2003 SP2 server and it did indeed

    break my BES server. I called MS and told them about it,

    and they said "it shouldnt have", to which i replied "oh but it

    did" lol. So am i to understand that upgrading BES will fix

    the problem ? m currently on 4.0.3

  42. Shannon says:

    I think this is a place to ask – how does 916783 relate to MS06-029/912442?  06-029 also updates store.exe, but with a version number lower than 916783?  It *seems* that 916783 includes MS06-029 as it also has a much later build date/time (despite MS06-029 being released months after 916783.)  Thanks hugely for this patch!

  43. Robert says:

    I have confirmed that MS06-029/912442 breaks BES. We’re on 3.6 still, but I imagine it would break any version. Same issue as mentioned in 912918. Have others run that script with success. I ran it and the output was daunting. We have 16,000 mbxs. I’m just not sure where to start in prepping our environment for the post 7650.23 store.exe world. Seems like there should be a utility put out that just "does it". Is this script it? I haven’t studied it in depth yet.

  44. Anonymous says:

    (Egentligen skulle jag bara posta om att jag hittat NoMAS-verktyget för publik nerladdning, men det känns…

Comments are closed.