The purpose of the VoiceRx series is to give you confidence in the audio quality of your Lync Enterprise Voice deployments. In this NextHop blog series, Microsoft IT and the Lync Product Group share their findings from our own corporate Lync deployment. We describe how we measure audio quality and use reporting to monitor the deployment. We also describe specific issues we have faced and how we remediated those issues in our own environment.
Today’s article comes from our esteemed colleague, Rob Pittfield. Rob is a Microsoft Senior Service Engineer for Lync.
Quality of Service (QoS) is a networking technology that enables organizations to optimize the end-user experience for audio and video communications. QoS identifies, marks, and prioritizes audio traffic at the packet level. This process helps ensure that packets travel faster through the network, which results in improved audio and video quality. QoS is commonly used when network bandwidth is limited. This article explains how to validate that traffic is correctly marked at the endpoints, after policies are set.
Author: Rob Pittfield, Microsoft Senior Service Engineer
Publication date: December 19, 2012
Product versions: Office Communications Server 2007 R2, Lync Server 2010, Lync Server 2013
When adding QoS to your infrastructure it is essential to plan, tag, and prioritize traffic correctly. This article assumes that Lync Server is deployed and that QoS is configured correctly. For in-depth information on planning, configuring and maintaining QoS download the white paper Enabling Quality of Service with Microsoft Lync Server 2010.
During a Microsoft corporate Lync Server deployment we encountered an issue where some network devices (routers, wireless access points, switches) were either striped (set to 0) or the audio Differentiated Services Code Point (DSCP) markings were changed from Expedited Forwarding (EF) 46 to another value. These settings were originally applied to clients using Group Policy Objects.
The process below shows how we determined that changes had occurred, and after correction, shows how we validated that the correct DSCP markings were preserved end to end.
When device settings are incorrect or are not consistent across the entire environment, network devices in the path don’t know which packets should be prioritized. This causes packet loss or jitter which results in degradation of the audio quality.
Validating QoS Policies
To validate QoS policies, collect network traces from the receiving servers or clients. Then verify the DSCP markings are correct.
Traffic from several locations and directions must be validated:
1. Servers: Front End to Edge Server and Mediation Server to Front End.
NOTE: The server side network capture validates that the sending client or other server GPO is implemented properly—see the Enabling Quality of Service with Microsoft Lync Server 2010 white paper for details on how to create QoS policies.
2. Clients: P2P, Client to Front End, Client to Mediation Server, and client to Edge Server.
NOTE: The client side network capture is necessary when there may be issues with DSCP marking on the network packets from a Lync Server or from another Lync client over a specific network path. Remember, these markings could be stripped or modified by any network device in the network path, it’s important to validate all paths that may seem suspect.
To validate this traffic collect network traces at each end. This ensures that traffic is being marked in each direction and that it’s traversing the network without being stripped somewhere between the endpoints.
To validate QoS policies are functioning as expected:
To begin a capture, open Network Monitor, click New Capture, and then click Start.
Figure 1. Start a new capture in Network Monitor
Make a call using the Lync Client. Verify you have two way audio between the caller and callee and then hang up.
To end the network trace click the Stop button on Network Monitor
Use Snooper to open the latest Communicator-uccapi-*.uccapilog and search for the SIP traffic for the call and locate the ports involved.
NOTE: The Communicator-uccapi-*.uccapilog is located in %userprofile%\tracing. If client logging isn’t enabled see the article Turn logging on in Lync. Snooper is part of the Microsoft Lync Server 2010 Resource Kit Tools.
To locate the information:
In Snooper type in the callee name or number and press Enter. Snooper filters the results and delivers only conversations with the callee.
Identify the final INVITE and 200 OK for the call, see the green lines in Figure 2.
Figure 2. Snooper output of a call from a client with the final INVITE and 200 OK marked
Next, find two sets of candidates which determine the IP/port combinations in use for an audio session:
1. Finding candidates in the INVITE.
2. Finding candidates in the 200 OK.
Finding candidates in the INVITE
Generally there are two INVITEs to start a call. The first establishes a long set of candidates (IP/Port combinations) that are tested to establish an audio session. The second (shown in green in Figure 2) selects the most optimal successful candidates for audio. To see what the candidate lines look like see INVITE examples below. If one of the parties is a remote caller, the actual lines will vary slightly from the ones shown below. Lines always start with a=candidate and have an IP/port listed.
When you've identified the final INVITE (green highlighted INVITE in Figure 2) review the two candidate lines as shown in Figure 3 below. You will see the details of the SIP traffic on the right pane.
Find the ports in use from the caller to determine where to look in the network trace for audio traffic. Look for the two lines that start with a=candidate. If you see more than two a=candidate lines for an audio only call, you’re not seeing the final INVITE. Go further down in the log file in Snooper until you see the INVITE that has only two a=candidate lines for the call you’re trying to understand. Figure 3 shows sample INVITE lines.
a=candidate:1 1 UDP 2130706431 192.168.0.11 24134 typ host
a=candidate:1 2 UDP 2130705918 192.168.0.11 24135 typ host
Figure 3. Candidate lines from final INVITE
192.168.0.11 is the IP address in use and 24134 is the SRTP port used for audio in this call. 24134 is the UDP port used by the client for RTP audio. 24135 is the UDP port used for RTCP in this call. The client IP is 192.168.0.11. We want to look at the traffic headed to and from the client on UDP port 24134.
Finding candidates in the 200 OK
To see which ports the server is using, look at the 200 OK for two similar lines from the server or other client in a P2P call. Use Snooper to highlight the final 200 OK to show the SIP protocol details in the right pane. See Figure 2.
a=candidate:1 1 UDP 2130706431 192.168.0.22 17994 typ host
a=candidate:1 2 UDP 2130705918 192.168.0.22 17995 typ host
Figure 4. Candidate lines from final 200 OK
192.168.0.22 is the IP address in use and 17994 is the port used for this call. 17994 is the UDP port used by the callee for RTP audio. 17995 is the UDP port used for RTCP in this case. The callee IP is 192.168.0.22. The combination of these ports and IP addresses shows how audio is flowing.
To validate the traffic on the receiving end, filter for the first port in the network traces. This validates the DSCP markings are making it through the network unchanged and are not being dropped.
When using the calling party client, remember that the INVITE and 200 OK can be going in opposite directions if the client is the one receiving the call, instead of initiating it. Example: Alice initiates a call to Bob. Alice sends the INVITE and Bob sends back the 200 OK when he answers it.
For more information on understanding call setup and negotiation, see How Communicator Uses SDP and ICE To Establish a Media Channel. This article talks about Edge connectivity but the same principles apply for session establishment.
Watch Thomas Binder’s TechEd presentation Lync Deep Dive: Edge Media Connectivity with ICE for an in-depth explanation of this topic.
Next, go back to Network Monitor and expand communicator.exe in the left pane.
Look for entries with the callee IP address and expand them to find the conversation that has the relevant ports in use.
Figure 5. Network Monitor with relevant audio traffic conversation highlighted on the left pane
After selecting the appropriate conversation there are two things to find:
- Audio from the caller to the callee.
- Audio from the callee to the caller.
It’s important to validate the appropriate direction of the audio to ensure the policies you’re testing are working as expected and nothing in the network path is modifying the relevant traffic.
Confirming caller to callee audio marking
The source IP address should be the caller or client in this case. The destination IP address should be the callee. Locate an audio packet in Network Monitor and highlight it to show the Frame Details see Figure 4.
In the Frame Details window, when you expand IPv4, you should see the following: DifferentiatedServicesField: DSCP: 46, ECN: 0. This shows that traffic from the caller to the callee is being marked—in this case with DSCP 46 (Expedited Forwarding). This is our default recommended policy.
Confirming callee to caller audio marking—the opposite direction
The source IP address should be the callee or destination in this case. The destination IP address should be the caller. Using Figure 4 as an example, ensure the IP addresses are now reversed. If there doesn’t appear to be any traffic from the callee, verify there was two way audio in the call. Locate an audio packet and highlight it to show the Frame Details information below.
In the Frame Details window expand IPv4 and you should see the following: DifferentiatedServicesField: DSCP: 46, ECN: 0.
This shows that traffic from the callee to the caller is being marked—in this case with DSCP 46 (Expedited Forwarding). This is our default recommended policy; however, it is possible to change this option using the GPOs for QoS.
When we were troubleshooting the Microsoft deployment, we determined the markings weren't set as expected and we looked at the configuration of the devices between the clients and servers. In some cases we found some settings that weren't quite right, and in another case—with a certain model of Wireless Access Point—we found an issue that required an update from the manufacturer. When we got those issues sorted out, we retested and to ensure the markings were happening as expected. This exercise also caused us to review our strategy for monitoring network devices to account for configuration drift to ensure a consistent configuration over time.
NOTE: audio traffic can travel over multiple network paths. To ensure QoS is applied appropriately, all paths should be validated. It’s also possible to mark traffic for Video, Desktop Sharing, File Transfer, and some people even mark the SIP signaling traffic. The strategy outlined above can be used to validate any type of traffic that you decide to prioritize with Diffserv DSCP marking.
Ensuring good network audio quality is a challenging task and a good QoS strategy is a small component of the work required to keep voice running smoothly. This article describes a method to determine that Lync endpoints are marking traffic appropriately, and that nothing on the network is interfering with the correct end to end markings. Of course, that’s only one part of the process. You also need verify that network devices are correctly prioritizing audio traffic to ensure it gets through the network as quickly as possible.
- Microsoft Lync Server 2010 Quality of Service (QoS) Deployment Guide
- Creating a Client Quality of Service (QoS) Policy
- Lync 2010 Bandwidth Calculator
- Troubleshooting Lync Server 2010 with Snooper
- Turn logging on in the Lync client
- TechNet Lync 2010 QoS Configuration guide
- Network Monitor 3.4
- Lync Server 2010 Resource Kit Tools
Lync Server Resources
- Lync Server 2010 Documentation Library
- DrRez blog
- NextHop blog
- Lync Server and Communications Server resources
We Want to Hear from You
Keywords: QoS, DSCP, Diffserv, Network Monitor, Tracing.