Event 1035, MSExchangeTransport – Inbound authentication failed with error UnexpectedExchangeAuthBlob for Receive connector


Scenario

Ran into an interesting case today where mail flow had quit working in one direction between sites. The environment was on Exchange 2010. Basically, Site A and Site B could not send mail to an exchange server in Site C. They could send to each other, and mail was flowing out from Site C, but inbound mail to Site C was queueing up.

The error in the Application event logs looked as follows:

Log Name:      Application
Source:        MSExchangeTransport
Date:          8/24/2016 1:21:43 PM
Event ID:      1035
Task Category: SmtpReceive
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:     SiteCserver.contoso.com
Description:
Inbound authentication failed with error UnexpectedExchangeAuthBlob for Receive connector Default SiteCServer. The authentication mechanism is ExchangeAuth. The source IP address of the client who tried to authenticate to Microsoft Exchange is [IP of Site A exchange server (or site B)].
Event Xml:
<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event”>
  <System>
    <Provider Name=”MSExchangeTransport” />
    <EventID Qualifiers=”32772″>1035</EventID>
    <Level>3</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime=”2016-08-24T17:21:43.000000000Z” />
    <EventRecordID>10489441</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SiteCserver.contoso.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>UnexpectedExchangeAuthBlob</Data>
    <Data>Default SiteCServer</Data>
    <Data>ExchangeAuth</Data>
    <Data>[IP of Site A exchange server (or Site B)]</Data>
  </EventData>
</Event>

There was no time skew in the environment and no backpressure events were showing up (Event IDs 15001 -15007) in the Application event logs.

Resolution

Normally, I would spend a great deal of time working through Protocol logs and network traces to figure out what is happening here, but luckily for me, I have an engineer named Miguel Ortiz on my team and he has seen this before.

In this particular case, what was happening was, the Kerberos token from Site A and Site B destined for Site C had gotten corrupted. Site C had suffered a severe outage and we believe it was during this time that the Kerberos tokens became corrupted. We went through the following steps to remediate the issue:

    1. On the Site C domain controllers, we restarted the Kerberos Key Distribution Center service (KDC).
    2. On the Site C Exchange server we purged the Kerberos tickets using the KList Purge command from an administrative command prompt
      1. klist purge
    3. Next, we restarted the Netlogon service on the Site C Exchange server.
      1. Net Stop Netlogon && Net Start Netlogon
    4. Then we restarted the Transport Service on the Site C Exchange server.
      1. Net Stop MSExchangeTransport && Net Start MSExchangeTransport
    5. After transport came back up on Site C, we then repeated steps 2 through 4 on the Site A and Site B servers.

This process basically clears out the Kerberos tickets and forces Site A and Site B to re-send their Kerberos request to Site C. In other words, they cleared the corrupted token and requested a new one.

Once this process completed, mail flow returned to normal and no further 1035 events were thrown.

Special Thanks

Thanks to Miguel Ortiz for helping me get through this case quickly and working out the steps to resolve it.

 

Comments (4)

  1. Scott Schering says:

    We ran into this exact issue today..
    I found we can skip the netlogon restart (that forces a information store restart)
    We did “klist -li 0x3e7 purge” to clear the system account tickets then just a transport restart..

    1. Gus Gallows says:

      Excellent find Scott. I appreciate the update.

  2. Arthur P. says:

    I ran to this issue , and i still have it … the symptoms for this error those it include lost connection with exchange i mean the virtual machine were its hosted “freezes”for a minute and comes back …

    Error that fallow 1035:
    106 general,msexchange common
    2937 ada acces validation

    1. Gus Gallows says:

      Arthur, If I am understanding you properly, you are saying that you are having the issue where the error, Inbound authentication failed with error UnexpectedExchangeAuthBlob for Receive connector, is reported? And I assume you followed the steps as laid out in the article above. If that is the case then you need to check to ensure you are not having backpressure events and if you are, troubleshoot those. They will show up as Event IDs 15001 -15007 in the Application Event logs. Also make sure you do not have a time skew issue. Your Exchange servers and your Domain Controllers all need to be on the same time (with the time zone adjustments of course). If they are not, you need to get them into sync. Hopefully this helps. If you are still having the issue and you have a premier account, I think it would be in your best interest to call in and open a ticket.

Skip to main content