In my Previous Article we discussed about the detailed call flow when a Skype for Business Desktop Client tries to sign in.
Its time to jump to the troubleshooting phase, where we are going to discuss on step by step approach, Collecting required logs and Using different tools that could help in identifying the issue cause.
Typical sign in issue would be that users not being able to sign in and getting one of below errors:
And many other errors, depending on what’s causing the issue:
Depending on the Error, issue cause could be in one of below:
- SFB Client or Computer issues
- Authentication or provisioning issues
- Network/Connectivity related issues
- Server related issues
Troubleshooting, Step by Step Approach:
Below approach could help us isolate the issue cause in a systematic manner.
1 : Verify if Server discovery is succeeding or not:
Manual vs Automatic Sign in
Simpler way to identify if DNS Records are the problem is by testing Manual Sign in vs Automatic Sign in method. To do that, we can edit the Advanced settings and manually provide Skype for Business Server/Pool name like below:
Skype for Business Client Settings -> Tools -> Personal -> Advanced
For Manual Sign in, provide the Skype for Business Pool name or Server name and the Port number manually in Internal or External Server name Box (depending on whether we are testing internal Sign in or External Remote Connectivity)
Internal Server Name : Front End Pool name & Port
External Server Name : External Access Edge FQDN & Port
If manual sign in worked, we know that Client is able to sign in when its connecting directly to Server/Pool using Manual settings, it’s just the Automatic sign in that is not functional, basically client is not able to determine where to connect to sign in.
For Automatic sign in to work, there are certain DNS Records (refer to Server Discovery step in previous article) that needs to be configured so that client can make DNS Query and get the Pool Info.
Skype for Business Client Log
Since we know from above step that automatic sign in is failing, we can look at client logs to see what all DNS Queries it made and whether it’s able to resolve DNS records or not.
If we have Logging Enabled on the client, we can open client Side Log ‘Lync-UccApi.Uccapilog’ present in Location “C:\Users\Mouli\AppData\Local\Microsoft\Office\1X.0\Lync\Tracing” using a notepad or any editor, below are sample log where client queries DNS for
<Info><![CDATA[Discovery request sent to URL https://email@example.com, txn (10A4EBE8), action id(1), type(InternalUrl)]]></Info>
<Info><![CDATA[Discovery request sent to URL https://firstname.lastname@example.org, txn (10A4EB68), action id(2), type(ExternalUrl)]]></Info>
09/12/2016|19:30:07.502 1098:E10 INFO :: QueryDNSSrv - DNS Name[_sipinternaltls._tcp.contoso.com]
09/12/2016|19:30:07.502 1098:E10 INFO :: QueryDNSSrv - DNS Name[_sip._tls.contoso.com]
09/12/2016|19:30:07.502 1098:112C TRACE :: SIP_MSG_PROCESSOR::OnDnsResolutionComplete[0D2F4C30] Entered host lync2010-se.contoso.com
09/12/2016|19:30:07.502 1098:112C TRACE :: SIP_MSG_PROCESSOR::OnDnsResolutionComplete get DNS result server: lync2010-se.contoso.com IP: 192.168.2.50:5061
Host A Records:
09/12/2016|19:34:00.56 1064 ERROR ResolveHostNameUsingGetAddrInfo - getaddrinfo(sipinternal.contoso.com)
09/12/2016|19:34:00.567 1064 ERROR :: ResolveHostNameUsingGetAddrInfo - getaddrinfo(sip.contoso.com)
09/12/2016|19:34:00.567 1064 ERROR :: ResolveHostNameUsingGetAddrInfo - getaddrinfo(sipexternal.contoso.com)
We could also make use of windows Builtin tool Nslookup to check the DNS Resolution.
Nslookup Lyncdiscover.contoso.com [For SRV Record we need to run ‘Set type=SRV’ ]
From above check we will know if DNS Records exists or not, based on that we could configure DNS Records to fix the issue.
Host File Entries:
Sometimes even if DNS Record is resolvable, automatic sign in might still be failing, as DNS Records might be pointing to a Load Balancer or Reverse Proxy or Firewall or different Skype for Business Pool which is having issues.
In such situation, we could use host file entries (admin rights is needed) to override DNS and point user to hit the Front end or Edge Server IP Address directly and test it out. Host file entries overrides DNS Queries, I, e, Client prefers host file entries over DNS query response.
By default, host file is located in “C:\Windows\System32\drivers\etc”
At the End of this stage, we will be in a state where we will know if the issue is specific to user's home server/Pool or whether its with different pool and due to the Redirection failure, based on the outcome we should take necessary actions to fix the issue.
2 : Verify if Network Connectivity is succeeding or not:
Now that we confirmed that DNS isn’t the problem, or takes steps to fix the DNS Issue, if sign in is still failing, next thing to check if whether client is able to make network connection to the Skype for business server and whether it was able to establish a TLS Session successfully or not.
We could make use of Telnet or Port Query tool (needs to be installed) to check if required ports are open and whether client is able to make successful connection. (if telnet is successful, you will see a Blank window in cmd prompt)
Telnet Sip.contoso.com 443
If telnet or Port Query is blocked on firewall (due to security restrictions), we can collect a Network trace from client to see whether connectivity check is succeeding or not.
If it’s able to connect, then we should see TCP 3 Way Handshake (A -> A,S -> A), if not we would see Retransmits, where client to trying to connect over the required port & its failing.
Based on the outcome, we could proceed further or we could work with Security team to get the required ports accessible for Client to be able to make connection to Server.
Other than above basic connectivity check, for client to be able to connect to Skype for Business Server, TLS Check is necessary. After successful Port connectivity check, client will try and establish a secure TLS connection, for this to work, below are some of things to consider:
Client should trust the Certificate issued on the Server (Client should contain the Root/Intermediate Certs)
FQDN/ URL that we are trying to access should be part of Certificate SAN entry on the Server where we are connecting to.
Required TLS Protocol & TLS cipher suites should be allowed in Firewall (If any)
3 : Check for Authentication or Server side Failures:
If we have surpassed above steps, then we should see SIP REGISTER Request being sent by client to initiate sign in process. This is another trick to identify, if all the above checks are successful is by opening UCCAPI Logs using Snooper Tool, if we see any REGISTER Messages being sent, which means client has performed all previous steps and then only sending SIP Requests.
Using Snooper Tool (Needs to be installed) we can open the Skype for Business Client side logs located in “C:\Users\Mouli\AppData\Local\Microsoft\Office\1X.0\Lync\Tracing” to see what's happening at the application level.
Depending the issue cause, sometimes Client logs might reveal the exact cause of the issue (for Ex : "Destination URI either not enabled for SIP or does not exist") and sometimes it would just give a generic Error (For Ex : 500 Internal Server Error )
Once we see which server is the source for generating this Error, we could take a look at server side Logs to see what's going on. Some of the Logs that we can look on the Server side are below:
Lync Server Event Logs (for any Server side or user specific Error, if logged)
Security Event Logs (for any Authentication failures, failure security audit)
Skype for Business CLS Logs (for detailed information on what's happening at SIP level)
Centralized Logging Service Logging is something that we can use to collect Server side logs while reproducing the issue. we could use the CLSLogger or the CLS Logging commands and collect the logs for IMAndPresence scenario (SIPStack & User Services) to get detailed information on what's going on when server gets the REGISTER request from the Skype for Business Client.
CLS Logging commands:
Start-CsClsLogging -Scenario "IMAndPresence" -Pools "SFB-SE.Contoso.com"
Stop-CsClsLogging -Scenario "IMAndPresence" -Pools "SFB-SE.Contoso.com"
Search-CsClsLogging -Pools "SFB-SE.Contoso.com" -Computers "SFB-SE.Contoso.com" -Components "SIPStack,UserServices" -StartTime "10/14/2016 6:09:51 PM" -EndTime "10/14/2016 6:14:51 PM" -LogLevel "All" -MatchAll -OutputFilePath "C:\Logs\SFBLog.txt"
4 : Troubleshooting Tools:
If we are suspecting issues where users is not able to connect to HTTPS services like Lyncdiscover or Certificate Provisioning web services, to see client side request and response, we can collect Fiddler trace from client machine when we try to sign in.
If we are suspecting Network connectivity or TLS Session establishment issue, we can use Network monitor Tool to collect network capture. Sometimes simultaneous capture from client and server would help identify if there are any firewall between blocking the packets
To view Client side UCCAPI logs or Server side CLS logs, we could make use of Snooper Tool to see SIP requests and responses.
Hope the above information helps, Happy Troubleshooting !
Pragathi Raj S
Premier Field Engineer - Microsoft