Lync Server: Event 41029 LS Data MCU – No connectivity with the Lync Web App

In a recent support case we were working on an issue where sometimes the users couldn’t use the Lync Web App.

The troubleshooting started in the Event Viewer > Lync Server, we notice that we had a few errors:

event41029-01

Log Name: Lync Server
Source: LS Data MCU
Date: 01/03/2017 15:00:43
Event ID: 41029
Task Category: (1018)
Level: Error
Keywords: Classic
User: N/A
Computer: lync2013fe01.gears.lab
Description:
No connectivity with the Lync Web App. Affected Web browser clients cannot use Web Conferencing modality.

Server Machine FQDN: lync2013fe01.gears.lab, Port:8061
Server Type: External-WebApp-Edge [HTTP side error:Unable to connect to the remote server]
If the problem persists this event will be logged again after 20 minutes
Cause: Service may be unavailable or Network connectivity may have been compromised.

Another error was mentioning an issue HTTP connectivity:

event41029-02

Log Name: Lync Server
Source: LS User Services
Date: 01/03/2017 15:04:57
Event ID: 30988
Task Category: (1006)
Level: Error
Keywords: Classic
User: N/A
Computer: lync2013fe01.gears.lab
Description:
Sending HTTP request failed. Server functionality will be affected if messages are failing consistently.

Sending the message to https://lync2013fe01.gears.lab:444/LiveServer/Replication failed. IP Address is 172.20.13.21. Error code is 2EFD. Content-Type is application/replication+xml. Http Error Code is 0.

Cause: Network connectivity issues or an incorrectly configured certificate on the destination server. Check the eventlog description for more information.
Resolution:
Check the destination server to see that it is listening on the same URI and it has certificate configured for MTLS. Other reasons might be network connectivity issues between the two servers.

Both events showed that the server could not establish a connection himself.

Then we check if the server was listening on that port:

netstat -anp TCP
https://technet.microsoft.com/en-us/library/bb490947.aspx

event41029-03

Get-NetTCPConnection -State Listen -LocalPort 80,8080,443,444,4443,8061 | ft -AutoSize
https://technet.microsoft.com/itpro/powershell/windows/tcpip/get-nettcpconnection

event41029-04a

The HTTP/HTTPS bindings were only on 127.0.0.1 and this is the loopback address.

Then we run the same on a working server in the same pool:

event41029-05

event41029-06

Note: For Get-NetTCPConnection :: is any available IPV4/IPV6 address.

So, in a working server the binding was on any available IP address, while the non-working was only on the loopback address.

Initially, we thought the issue was in IIS/certificate bindings, but both were properly configured:

Get-WebBinding | ft -AutoSize
https://technet.microsoft.com/en-us/library/hh867866(v=wps.630).aspx

event41029-07

netsh http show sslcert
https://msdn.microsoft.com/en-us/library/windows/desktop/cc307236(v=vs.85).aspx

event41029-08

After checking other parameters available in netsh we found that the non-working had the loopback address configured in the HTTP IP Listen List:

netsh http show iplisten

event41029-09

While the working server we didn’t had any IP address configured:

event41029-10

This was causing the wrong binding, to fix it we only had to remove the loopback address from the list:

netsh http delete iplisten 127.0.0.1

After this change the server started to listen in the correct IP address/ports:

netstat -anp TCP

event41029-12

We also confirmed in the Event Viewer that the Lync Web App was starting:

 event41029-13