Odd behavior with OWA when sending data through a load balancer
One of my colleagues, Danny Pastuszynski and I, recently had the privilege of working a very odd case for a customer of ours, and after discussing with the customer, they suggested I write a blog post on it. So, here we go!
Our customer was spinning up a new Exchange environment with multiple sites with a load balancer to help manage their traffic load. Everything from what they could tell was configured correctly, however what they were running into was when sending the traffic for a cross-site call for OWA, they were getting an error similar to below:
Or:
Sorry, we can't get that information right now. Please try again later. If the problem continues, contact your helpdesk.
X-ClientId: XXXX-XXXX-XXXX-XXXX
X-OWA-Error: ClientError;exMsg='$SERVER' is undefined;file=ReferenceError: '$SERVER' is undefined
at _h.$Fy.prototype.$Des (https://domain.contoso.com/owa/prem/15.0.1263.5/scripts/boot.0.mouse.js:82:1710720)
at _h.$1az.prototype.$988 (https://domain.contoso.com/owa/prem/15.0.1263.5/scripts/boot.0.mouse.js:82:1622688)
at _a.$Fj.prototype.$6O3 (https://domain.contoso.com/owa/prem/15.0.1263.5/scripts/boot.0.mouse.js:82:383225)
at _a.$H8.prototype.$390 (https://domain.contoso.com/owa/prem/15.0.1263.5/scripts/boot.0.mouse.js:82:385523)
at _a.$H8.prototype.$1U (https://domain.contoso.com/owa/prem/15.0.1263.5/scripts/boot.0.mouse.js:82:385302)
at _brp.$AW.prototype.$2Pk (https://domain.contoso.com/owa/prem/15.0.1263.5/scripts/boot.1.mouse.js:2:13
X-OWA-Version: undefined
X-FEServer: FrontEnd.Contoso.Com
X-BEServer: undefined
Date: 6/30/2017 3:14:08 AM
Basically, the X-OWA-Error: ClientError;exMsg$MachineName is not defined;file=ReferenceError :$MachineName is not defined.
Then, we would see a series of java scripting errors.
However, when we would pull the data in a network capture between the front and and back end Exchange servers, everything looked good. We would see the cert exchange, and traffic would not look abnormal. (i.e. TTL lowered to somewhere in the 60's, TCP resets, etc.)
We then reviewed our HTTP Proxy logs between the front and and back end servers - we would see http status codes of 200 between our front end and back end Exchange servers. This would indicate the communication was successfully happening, and that there was something with the rendering happening between the Exchange server front end and the client. However, since the message was indicating something was going wrong with the rendering, a fiddler capture would not be helpful to us here.
Resolution:
Our next troubleshooting step was to bypass the customers load balancer. This is a fairly easy step where you configure a hosts file or just type in the server FQDN of one of the Exchange servers to no longer point to your VIP name. Whenever we did this, we would never get the above error. This just helped to add in more pieces to the puzzle. We were able to engage the NLB vendor and the customer's network team, where after comparisons to their dev site and their production site, we were able to determine there was a slight configuration change between the two sites, in that the customer had enabled an analytics feature within their load balancer which was injecting java script into the OWA calls, which ended up breaking the OWA render.