After an in-place upgrade to Skype for Business Server 2015 Response Group Workflows Fail to work


In a peculiar case, we noticed that after an in-place upgrade from Lync Server 2013 to Skype for Business Server 2015, Response Group Workflows stopped working. The initial troubleshooting involved verifying (i) that the Response Group Service is starting, (ii) the work-flow is defined in Skype for Business Server 2015 after the upgrade, (iii) the AD Object for the workflow exists.

To troubleshot further, as a simple step, we deleted and re-created the workflow, and could reproduce the error. From the logs on client, we could see 500 Internal Server Error, so I wanted to first make sure the permissions on AD are set correctly based on Issue: Calls to certain Response groups fail with an Error “500 Internal Server Error” . Once I could verify that the issue was indeed not to do with AD permissions, I started to investigate further.

Looking through the built-in scenarios, a scenario called RGS exists, but I wasn’t sure, if the components would suffice.

Description Centralized Logging Tool Scenario Logging Tool Components
Response Group Service RGS RgsClientsLib

I decided to create a custom scenario, call it RGSCustom and perform logging. I used the below commands to create my custom scenario

$SIPStack = New-CsClsProvider -Name “SIPStack” -Type “WPP” -Level “Debug” -Flags “All”
$S4 = New-CsClsProvider -Name “S4” -Type “WPP” -Level “Debug” -Flags “All”
$RGSClientLib = New-CsClsProvider -Name “RGSClientLib” -Type “WPP” -Level “Debug” -Flags “All”
$RgsCommonLibrary = New-CsClsProvider -Name “RgsCommonLibrary” -Type “WPP” -Level “Debug” -Flags “All”
$RgsDatastores = New-CsClsProvider -Name “RgsDatastores” -Type “WPP” -Level “Debug” -Flags “All”
$RgsDeploymentApi = New-CsClsProvider -Name “RgsDeploymentApi” -Type “WPP” -Level “Debug” -Flags “ALL”
$RgsDeploymentLibrary = New-CsClsProvider -Name “RgsDeploymentLibrary” -Type “WPP” -Level “Debug” -Flags “ALL”
$RgsDiagnostics = New-CsClsProvider -Name “RgsDiagnostics” -Type “WPP” -Level “Debug” -Flags “All”
$RgsHostingFramework = New-CsClsProvider -Name “RgsHostingFramework” -Type “WPP” -Level “Debug” -Flags “ALL”
$RgsMatchMakingService = New-CsClsProvider -Name “RgsMatchMakingService” -Type “WPP” -Level “Debug” -Flags “All”
$RgsDBSyncAgent = New-CsClsProvider -Name “RgsDBSyncAgent” -Type “WPP” -Level “Debug” -Flags “ALL”

New-CsClsScenario -Identity “Global/RGSCustom” -Provider @{Add=$SIPStack,$S4,$RGSClientLib,$RgsCommonLibrary,$RgsDatastores,$RgsDeploymentApi,$RgsDeploymentLibrary, $RgsDiagnostics, $RgsHostingFramework, $RgsMatchMakingService, $RgsDBSyncAgent}

Next it was time to start logging and while we were logging to restart the Response Group Service, so that I would have any failure encountered during service state captured

Start-CsClsLogging -Scenario “RGSCustom” -Pools SFBSTD01.contoso.com
Stop-CsClsLogging -Scenario “RGSCustom”
Search-CsClsLogging -Pools SFBSTD01.contoso.con -OutputFilePath “C:\Windows\Temp\RGSCustom.txt” -StartTime (get-date).AddMinutes(-30)

 

As I was investigating the CLS logs, I noticed the a peculiar error Message: CALLCONTROL: Call declined because CallControl is not started

TL_INFO(TF_COMPONENT) [SFBSTD01\SFBSTD01] 208D8.4700::05/03/2016-19:46:01.484.0000A42D (RgsCommonLibrary,RgsLogMessage.ReportMessageInternal:rgslogmessage.cs(452))
_rgs_message_begin_
Direction: Incoming
From:
sip:sritodi@contoso.com
To: sip:sampleworkflow@contoso.com
Message: CALLCONTROL: Call declined because CallControl is not started
WorkflowId: c338e8ba142042b9b30023269d29daa0
_rgs_message_end_

I decided to look through the event logs when the service was starting and noticed EVENT ID 31067

clip_image002

This sounded a little strange as I was aware that every Lync Server 2013 pool has 2 Application Contacts, one RGS Presence Watcher and another RGS Announcement Service. This event was speaking about 3 RGS Presence Watcher Contact objects.

I decided to query the number of RGS Presence Watcher contact objects

Get-CsApplicationEndpoint | Where-Object DisplayName -eq “RGS Presence Watcher” | Ft Identity, DisplayName, RegistrarPool

This could also be accomplished by running

Get-ADObject -Filter ‘msRTCSIP-OwnerUrn -eq “urn:application:RGS”‘ -SearchBase ‘CN=Configuration,DC=contoso,DC=Com’ -Properties msRTCSIP-PrimaryUserAddress, displayName | ft msRTCSIP-PrimaryUserAddress, displayName –AutoSize

 

I quickly realized that for the pools that were upgraded from Lync Server 2013 to Skype for Business Server 2015 and where calls to response group were failing, only had a single RGS Presence Watcher contact object

Solution: We decided to republish the topology by running

Enable-CsTopology -Verbose

Once topology was republished, we could see 3 RGS Presence Watcher contact objects. A service restart to the RGS service, and calls started to work as expected.

Comments (2)

  1. Dr.Lync says:

    Maybe a similar case.
    I had more than 2 presence watchers after an upgrade and was able to use adsiedit to delete the redundant ones.
    Topology worked fine after I deleted them.
    This seems only to happen if you have rgs workflows when upgrading

    1. Sri Todi says:

      After an in-place upgrade, you should have 3 unique RGS Presence Watcher contact objects. AD wouldn’t even allow for duplicates to be created. Can you please provide some specifics.

Skip to main content