I was trying to setup my OCS 2007 R2 lab environment using windows 2008 R2. It's a simple lab which consist one standard edition server and one edge server. The OCS internal user login was working fine, but the problem was with the remote user access. I can't sign-in from remote environment thru my edge server. I'd been trying to get it to work until I saw the network traffic using wireshark that I realized there's connection reset (RSET) coming from edge server to the internal server. I knew that the problem was with the SSL certificate failure. But it was not really easy to find out where the problem was. I checked my entire certificate and global settings were configured correctly.
When I captured the SIP log from the edge server, I saw the following error message from edge server saying that the Outbound TLS negotiation failed. It means that even on doing SSL handshake/established was already failed.
TL_ERROR(TF_CONNECTION) 07B4.0814::04/08/2011-21:50:58.091.00000852 (SIPStack,SIPAdminLog::TraceConnectionRecord:SIPAdminLog.cpp(157))$$begin_record
Text: Outbound TLS negotiation failed
Result-Code: 0x80004005 E_FAIL
I found the KB 982021 which describes the supportability of running OCS 2007 R2 on windows Server 208 R2.
Supportability is available for Office Communications Server 2007 R2 member server role on a Windows Server 2008 R2 operating system
I went through the document and found there's one patch that I missed to install on my server which relates to TLS/SSL negotiation failure on Windows Server 2008 R2/Windows 7 OS, described as follow:
975858 (http://support.microsoft.com/kb/975858/ ) An application or service that calls the InitializeSecurityContext function together with the ISC_REQ_EXTENDED_ERROR flag may encounter a TLS/SSL negotiation failure on a computer that is running Windows Server 2008 R2 or Windows 7 operating system
I Downloaded the patch, installed, and rebooted the server, after that the remote user access worked well 🙂