Lync 2010 response group members phones keep ringing even if the remote party hangs up the phone

In this blog post, I’ll be talking about a response group problem where the response group members phones keep ringing even if the user calling from pstn side hangs up the phone. I would like to talk about how I troubleshooted this issue:

 

We first collected ETL traces and network traces from the mediation server to better understand the problem (actually mediation server and frontend server were running on the same machine). Some more details about the response group:

 

Response group number: 8900

Response group members: user1@contoso.com / user2@contoso.com / user3@contoso.com / user4@contoso.com / user5@contoso.com

 

In the S4 ETL trace, we can see that once the call is made to response group number, response group application starts ringing each response group member with 15 seconds intervals while it keeps playing music on hold to the caller:

 

 

Since none of the response group members answer, response group application goes through each member in a loop fashion. (actually the exact behavior here is identified by response group charateristics like workflow, routing method etc)

 

Interestingly, even the user who initiated the call to response group hangs up the phone, the rtp traffic from voip gateway keeps coming towards the mediation server:

 

  98312 2012-11-23 08:59:05.383 0.015 172.16.1.7 172.17.1.100RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x3FF62C47, Seq=19584, Time=964130050

  98313 2012-11-23 08:59:05.388 0.004 172.17.1.100172.16.1.7 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x6ED4D5BE, Seq=33285, Time=1945495369

  98320 2012-11-23 08:59:05.403 0.015 172.16.1.7 172.17.1.100RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x3FF62C47, Seq=19585, Time=964130210

  98322 2012-11-23 08:59:05.408 0.004 172.17.1.100172.16.1.7 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x6ED4D5BE, Seq=33286, Time=1945495529

  98323 2012-11-23 08:59:05.423 0.015 172.16.1.7 172.17.1.100RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x3FF62C47, Seq=19586, Time=964130370

  98324 2012-11-23 08:59:05.428 0.004 172.17.1.100172.16.1.7 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x6ED4D5BE, Seq=33287, Time=1945495689

  98326 2012-11-23 08:59:05.443 0.015 172.16.1.7 172.17.1.100RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x3FF62C47, Seq=19587, Time=964130530

 

So we should have got a SIP BYE from VoIP gateway side when the remote caller hung up the phone...

 

This looks like a VoIP gateway side problem. We then made some internal testing to verify the expected behavior. Here are the results from the internal testing:

 

Expected behavior:

===============

Note: The test response group has one member only so the response group application keeps ringing the same user

 

- The remote party calls the Response group number

- Response group answers the call and then it starts ringing the response group member (joe@contoso.com)

- Since the agent doesn’t pick up the phone for 15 seconds, it drops that session with a SIP CANCEL to the group member and it starts another session with the same agent (because there’s only one agent)

 

- The person calling from behind the VoIP gateway hangs up the phone after 23 seconds or so after the initial call

- So the second dialog with the agent is terminated

 

- The second call to the agent is made at 13:36:17

- The remote caller hangs up the phone at 13:36:22 (23 seconds after the response group answers the call)

- And the mediation server gets a SIP BYE from VoIP gateway since the remote caller hangs up the phone:

 

 

TL_INFO(TF_PROTOCOL) [0]0C4C.1410::11/23/2012-13:36:22.847.0002b91f (S4,SipMessage.DataLoggingHelper:sipmessage.cs(686))[3311788563]

<<<<<<<<<<<<Incoming SipMessage c=[<SipTlsConnection_386DD23>], 192.168.1.20:5071<-192.168.1.20:50807

BYE sip:napool.contoso.com:5071;grid;ms-fe=nafe02.contoso.com SIP/2.0

FROM: <sip:18006934501;phone-context=DefaultProfile@contoso.com;user=phone>;epid=8D443A6D50;tag=928547717d

TO: <sip:17042563016;phone-context=DefaultProfile@contoso.com;user=phone>;epid=14B9F3DDE9;tag=872d3a4ad0

CSEQ: 3828 BYE

CALL-ID: d9789202-c026-4661-88d8-adfc12c37995

MAX-FORWARDS: 69

VIA: SIP/2.0/TLS 192.168.1.20:50807;branch=z9hG4bK6FF51B0B.C998AF162BD50886;branched=FALSE

VIA: SIP/2.0/TLS 192.168.1.15:54616;branch=z9hG4bK97e1a6aa;ms-received-port=54616;ms-received-cid=726C00

CONTACT: <sip:nafe01.contoso.com@contoso.com;gruu;opaque=srvr:MediationServer:gJSXgywGXl2uoBnZnm4R7QAA;grid=1792a8d837324735acd180c81887b8b0>;isGateway

CONTENT-LENGTH: 0

SUPPORTED: ms-dialog-route-set-update

SUPPORTED: gruu-10

USER-AGENT: RTCC/4.0.0.0 MediationServer

P-ASSERTED-IDENTITY: "PBX 01"<sip:18006934501;phone-context=DefaultProfile@contoso.com;user=phone>

ms-diagnostics: 10027;source="nafe01.contoso.com";reason="Normal termination response from gateway after the call was established";component="MediationServer"

ms-diagnostics-public: 10027;reason="Normal termination response from gateway after the call was established";component="MediationServer"

ms-endpoint-location-data: NetworkScope;ms-media-location-type=intranet

 

------------EndOfIncoming SipMessage

 

Right after the SIP BYE is received from VoIP gateway, response group stops ringing the agent with a SIP CANCEL request:

 

 

We can see the same behavior in network trace as well:

 

=> The call starts and RTP traffic starts flowing:

 

 

=> At 15:36:23, VoIP gateway (Asteriks PBX) sends a SIP BYE to mediation server since the remote caller hangs up the phone, and the RTP traffic also stops at this point:

 

 

Summary:

========

We advised our customer to further check their VoIP gateway to better understand why it doesn’t send a SIP BYE to mediation server when the remote caller hangs up the phone (and because the SIP BYE is not received, response group member phones keep ringing)

 

Hope this helps

 

Thanks,

Murat