Scenario: You are publishing your Microsoft Exchange services (OWA, ActiveSync, Outlook Anywhere) through Web Application Proxy in Windows Server 2012 R2. Using a web browser for OWA works as expected but clients using Outlook for Mac 2011 cannot get their mail. The error you receive on the Outlook for Mac client is "bad username or password." There may be other clients not working correctly as well including, but no limited to, Android phones using ActiveSync.
Cause: Outlook for Mac is a "non-SNI capable" client. Web Application Proxy utilizes the Server Name Indication (SNI) feature of SSL TLS and clients that do not support this will not work correctly. The quickest and easiest way to tell if your client is non-SNI capable is to get a network trace of the SSL traffic that the client is sending. If getting it from the client is not possible you can also get a network trace from the WAP server.
Look for the following pattern in the SSL traffic from client to server.
Frame Source Destination Protocol
4681 3:19:14 PM 7/23/2014 75.8088450 10.1.1.1 10.1.1.254 TCP TCP:Flags=……S., SrcPort=49633, DstPort=HTTPS(443), PayloadLen=0, Seq=893461645, Ack=0, Win=65535 ( Negotiating scale factor 0x4 ) = 65535
4682 3:19:14 PM 7/23/2014 75.8096090 10.1.1.254 10.1.1.1 TCP TCP:Flags=…A..S., SrcPort=HTTPS(443), DstPort=49633, PayloadLen=0, Seq=2972027426, Ack=893461646, Win=8192 ( Negotiated scale factor 0x8 ) = 2097152
4683 3:19:14 PM 7/23/2014 75.8099690 10.1.1.1 10.1.1.254 TCP TCP:Flags=…A…., SrcPort=49633, DstPort=HTTPS(443), PayloadLen=0, Seq=893461646, Ack=2972027427, Win=8235 (scale factor 0x4) = 131760
4684 3:19:14 PM 7/23/2014 75.8099860 10.1.1.1 10.1.1.254 SSL SSL:SSLv2RecordLayer, ClientHello (0x01)
4685 3:19:14 PM 7/23/2014 75.8104840 10.1.1.254 10.1.1.1 TCP TCP:Flags=…A.R.., SrcPort=HTTPS(443), DstPort=49633, PayloadLen=0, Seq=2972027427, Ack=893461703, Win=0 (scale factor 0x8) = 0
As you can see in the network traffic above the client at 10.1.1.1 completes the TCP handshake to the server at 10.1.1.254 and then sends the Client Hello. Immediately after the Client Hello the server sends a Reset (RST). A closer look at the frame where Client Hello is being sent would reveal that that it does not contain a ClientHelloExtension for Server Name.
Workaround: The good news is that there is a simple workaround that can be employed to get your non-SNI clients working. Ian Parramore wrote a very detailed and informative blog post that goes into much more depth than I have and explains how to implement the workaround. Ian will also be updating his blog as we discover more types of clients that have this issue. The blog can be viewed at http://bit.ly/1qqNOnx
For more detailed information on the SSL Protocol Version 3.0 or a quick cure for insomnia please see the official RFC at http://tools.ietf.org/html/rfc6101