Using WireShark to Trace Your SharePoint 2010 to Azure WCF Calls over SSL

One of the interesting challenges when trying to troubleshoot remotely connected systems is figuring out what they're saying to each other.  The CASI Kit that I've posted about other times on this blog (http://blogs.technet.com/b/speschka/archive/2010/11/06/the-claims-azure-and-sharepoint-integration-toolkit-part-1.aspx) is a good example whose main purpose in life is providing plumbing to connect data center clouds together.  One of the difficulties in troubleshooting it is that case is that the traffic travels over SSL so it can be fairly difficult to troubleshoot.  I looked at using both NetMon 3.4, which has an Expert add in now for SSL that you can get from http://nmdecrypt.codeplex.com/, and WireShark.  I've personally always used NetMon but had some difficulties getting the SSL expert to work so decided to give WireShark a try.

WireShark appears to have had support for SSL for a couple years now; it just requires that you provide the private key used with the SSL certificate that is encrypting your conversation.  Since the WCF service is one that I wrote it's easy enough to get that.  A lot of the documentation around WireShark suggests that you need to convert your PFX of your SSL certificate (the format that you get when you export your certificate and include the private key) into a PEM format.  If you read the latest WireShark SSL wiki (http://wiki.wireshark.org/SSL) though it turns out that's not actually true.  Citrix actually has a pretty good article on how to configure WireShark to use SSL (http://support.citrix.com/article/CTX116557), but the instructions are way to cryptic when it comes to what values you should be using for the "RSA keys list" property in the SSL protocol settings (if you're not sure what that is, just follow the Citrix support article above).  So to combine that Citrix article and the info on the WireShark wiki, here is a quick run down on those values:

• IP address - this is the IP address of the server that is sending you SSL encrypted content that you want to decrypt
• Port - this is the port the encrypted traffic is coming across on.  For a WCF endpoint this is probably always going to be 443.
• Protocol - for a WCF endpoint this should always be http
• Key file name - this is the location on disk where you have the key file
• Password - if you are using a PFX certificate, this is a fifth parameter that is the password to unlock the PFX file.  This is not covered in the Citrix article but is in the WireShark wiki.

So, suppose your Azure WCF endpoint is at address 10.20.30.40, and you have a PFX certificate at C:\certs\myssl.pfx with password of "FooBar".  Then the value you would put in the RSA keys list property in WireShark would be:

10.20.30.40,443,http,C:\certs\myssl.pfx,FooBar.

Now, alternatively you can download OpenSSL for Windows and create a PEM file from a PFX certificate.  I just happend to find this download at http://code.google.com/p/openssl-for-windows/downloads/list, but there appear to be many download locations on the web.  Once you've download the bits that are appropriate for your hardware, you can create a PEM file from your PFX certificate with this command line in the OpenSSL bin directory:

openssl.exe pkcs12 -in <drive:\path\to\cert>.pfx -nodes -out <drive:\path\to\new\cert>.pem

So, supposed you did this and created a PEM file at C:\certs\myssl.pem, then your RSA keys list property in WireShark would be:

10.20.30.40,443,http,C:\certs\myssl.pem

One other thing to note here - you can add multiple entries separated by semi-colons.  So for example, as I described in the CASI Kit series I start out with a WCF service that's hosted in my local farm, maybe running in the Azure dev fabric.  And then I publish it into Windows Azure.  But when I'm troubleshooting stuff, I may want to hit the local service or the Windows Azure service.  One of the nice side effects of taking the approach I described in the CASI Kit of using a wildcard cert is that it allows me to use the same SSL cert for both my local instance as well as Windows Azure instance.  So in WireShark, I can also use the same cert for decrypting traffic by just specifying two entries like this (assume my local WCF service is running at IP address 192.168.20.100):

10.20.30.40,443,http,C:\certs\myssl.pem;192.168.20.100,443,http,C:\certs\myssl.pem

That's the basics of setting up WireShark, which I really could have used late last night.  🙂   Now, the other really tricky thing is getting the SSL decrypted.  The main problem it seems from the work I've done with it is that you need to make sure you are capturing during the negotiation with the SSL endpoint.  Unfortunately, I've found with all the various caching behaviors of IE and Windows that it became very difficult to really make that happen when I was trying to trace my WCF calls that were coming out of the browser via the CASI Kit.  In roughly 2 plus hours of trying it on the browser I only ended up getting one trace back to my Azure endpoint that I could actually decrypt in WireShark, so I was pretty much going crazy.  To the rescue once again though comes the WCF Test Client. 

The way that I've found now to get this to work consistently is to:

1. Start up WireShark and begin a capture.
2. Start the WCF Test Client
3. Add a service reference to your WCF (whether that's your local WCF or your Windows Azure WCF)
4. Invoke one or more methods on your WCF from the WCF Test Client.
5. Go back to WireShark and stop the capture.
6. Find any frame where the protocol says TLSV1
7. Right-click on it and select Follow SSL Stream from the menu

A dialog will pop up that should show you the unencrypted contents of the conversation.  If the conversation is empty then it probably means either the private key was not loaded correctly, or the capture did not include the negotiated session.  If it works it's pretty sweet because you can see the whole conversation, or only stuff from the sender or just receiver.  Here's a quick snip of what I got from a trace to my WCF method over SSL to Windows Azure:

• POST /Customers.svc HTTP/1.1
• Content-Type: application/soap+xml; charset=utf-8
• Host: azurewcf.vbtoys.com
• Content-Length: 10256
• Expect: 100-continue
• Accept-Encoding: gzip, deflate
• Connection: Keep-Alive
• HTTP/1.1 100 Continue
• HTTP/1.1 200 OK
• Content-Type: application/soap+xml; charset=utf-8
• Server: Microsoft-IIS/7.0
• X-Powered-By: ASP.NET
• Date: Sat, 19 Mar 2011 18:18:34 GMT
• Content-Length: 2533
• <s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" blah blah blah

So there you have it; it took me a while to get this all working so hopefully this will help you get into troubleshooting mode a little quicker than I did.