Tales of the weird: OAB distribution failing

From 'Tales of the Weird and Unusual', we recently worked a long running incident where at customer was having issues with their OAB in Exchange 2010, and our customer was hoping we could do a blog article on it.  What we were finding is that the generation was working as expected, however it was the distribution point from the generation server that was failing.  (Basically the OAB's were not updating between CAS01 and CAS02 in the diagram below)

In this instance, we are not using public folder distribution, but rather web distribution.  We could see on CAS03 that our OAB's were getting generated, it's just the updated files were never making it to CAS01 or CAS02.  After ruling out network as being an issue, this left us wondering what was going on and we were finding that we couldn't browse via SMB to the fileshares.  We would set to C:\ for our fileshare, but we weren't even able to browse to it, but we could see where it was configured:

We then had the customer patch the system, and make sure it was up to date 100% as the customer was missing some windows patches.  This still did not fix it.

This far along in our troubleshooting, we've eliminated the following:

  • -Firewall/Network.
  • -Proper OAB configuration.
  • -Windows Server patching.
  • No Antivirus/third party.  (Uninstalled as part of the troubleshooting process, as we were going to have to take an iDNA and AntiVirus + iDNA do not behave well, as discussed here. )

It didn't leave us many other options, and we ended up taking an iDNA of explorer.exe to see if we could determine what was going on and where.  In the iDNA, we saw there we were referencing a very odd registry key!  The provider name under [HKLM\System\CurrentControlSet\Control\NetworkProvider\HwOrder] was referencing LanmanWorkstationt not LanmanWorkstation

vs. What we should have saw:

 

We then had the customer go back through their change control (and you are doing your change control like this, right?) and discovered some 3rd party software they were using had updated this registry key as part of it's patching process.

Once we reverted the change and implemented the correct key, restarted the system, the OAB was being distributed to their CAS, and in turn was being sent to their clients.