getting error " 502 proxy error. The requested resource is in use. (170)." on the upstream proxy TMG servers

This one is about a case, where Admin had configured Two TMG 2010 upstream proxy servers and he had two TMG 2010 downstream proxy servers. Users started reporting that they intermittently get error

" 502 proxy error. The requested resource is in use. (170)." while accessing internet.

At the time of issue, even if you try to access it from the Upstream TMG server itself, you will see this error on the browser. To avoid the problem ,Admin was bypassing these upstream TMG proxy servers. So it appeared to be a Load issue.

Troubleshooting and Data analysis

During the course of troubleshooting, I collected TMG data packager while doing repro of the issue i.e. when issue occurred. In the TMG traces I found following.

  
 ************************************************ 
ERROR:Cannot create more than 3000 concurrent inspection sessions. 
ERROR:spScanner->CreateSession failed, hr=0x800700aa(ERROR_BUSY) 
ERROR:CreateInspectionSession() failed, hr=0x800700aa(ERROR_BUSY) 
ERROR:Accumulate failed, hr=0x800700aa(ERROR_BUSY) 
ERROR:pRequestContext->OnClientSendRawData failed, hr=0x800700aa(ERROR_BUSY) 

Warning m_spProtocolHandler->GetExpectedSize failed, hr=0x80070015(ERROR_NOT_READY) 
Info Parsing response headers hasn't completed yet, cannot check for filename. 
ERROR:pRequestContext->OnClientSendRawData failed, hr=0x800700aa(ERROR_BUSY) 

ERROR:SendResponseHeaders failed 170(ERROR_BUSY) 
Info WPPISAPUBLIC:Returning error text "The requested resource is in use. " for error code 170(ERROR_BUSY) 
********************************************* 
 We can see from above trace that we are getting error
 "ERROR:Cannot create more than 3000 concurrent inspection sessions. " in the trace, which means we are hitting the maximum 
 threshold for concurrent inspection sessions. 
 Then as a follow through we get message that server was busy. or Requested resource is in use.
  
 After doing some research on this behavior I suggested following :
   
 1. To increase the default value of 3000 via this RAT flag in registry 
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\RAT\Stingray\Debug\Web Filters

 

 \EMP_MAX_CONCURRENT_INSPECTION_SESSIONS 

2. To add another upstream server i.e. scale up the number of servers to share
  the Load further to avoid reaching this limit of concurrent inspection sessions. 

After we configured the reg key with a value of 5000 on the TMG servers, 
 we did not see issue occurring again. 

 

 We can change this value upto our requirements but better idea would be to scale up the
  number of servers as we may have to keep doing this if user base would keep increasing.