Hardware Load Balancer Health Checks and Web Application Proxy / AD FS 2012 R2


 

Hi everyone, this blog took a bit longer to get out than we’d planned but we hope it’s worth the wait.

A common question we’ve been seeing from various sources is how to configure Hardware Load Balancer health checks to work correctly with Web Application Proxy. There are some key considerations you need to be aware of when configuring your Load Balancer health checks, but if you know about these things, then it should be pretty straight-forward.

We will also look at new functionality introduced in the August 2014 Windows Server 2012 R2 Update rollup (KB.2975719) to provide HTTP Health Check endpoints for your AD FS 2012 R2 and Web Application Proxy servers.

 

Key Properties of a Health Check request

First, let’s look at the different properties at various protocol layers that we are interested in for a health check request and the recommended values.

 

Protocol Level

Property

Recommend Value

TCP/IP

IP address

Dedicated IP of each Web Application Proxy farm member

TCP/IP

Port

Port of service being queried e.g. 80, 443

SSL

SNI Header

FQDN of service being health checked

HTTP

Host Header

FQDN of service being health checked (not IP address)

HTTP Uri

URL path to test

While some of these are self-explanatory let’s drill into the details of what each of these mean and the considerations to take into account. The main ones we have seen some confusion and issues around are SNI Header and Host Header.

 

1) TCP level – the IP address and port the health check request will be sent to

Probably an obvious one but You will want your health requests to be sent to the dedicated IP address of each of the Web Application Proxy farm members.

 

2) TCP level – the port the health check request will be sent to
The health check requests should be sent to the relevant port that the service/application is listening on - most likely port 443 in a Web Application Proxy / AD FS scenario.

3) SSL level – does your load balancer support Server Name Indication (SNI) and if not what are your options?

If you are making a health check to an SSL endpoint, after the TCP connection has been established, the Load Balancer will send an SSL Client Hello to initiate the establishment of the SSL tunnel.

Web Application Proxy and AD FS 2012 R2 both use Server Name Indication (SNI) http.sys certificate bindings. SNI allows multiple certificates to be used on the same IP/port pair and http.sys uses the SNI header in the SSL Client Hello to determine which certificate should be served. This relies on the client sending the Server Name extension in the SSL Client Hello.

If the SSL Client Hello does not contain the SNI header then http.sys is unable to determine which certificate to serve and will reset the connection.

You therefore need to determine if your Load Balancer supports SNI in its health check requests. If it does support SNI then ensuring that the Load Balancer sends the SNI header in the SSL Client Hello is our recommendation.

Unfortunately there seems to be limited information available around configuring SNI with the common hardware load balancer solutions.

The following F5 article has some good details though especially around configuring the F5 Health Checks to send an SNI header - https://devcentral.f5.com/articles/big-ip-and-adfs-part-5-working-with-adfs-30-and-sni

Note– while this article has some great information on SNI and Health Check requests it also recommends using SSL Bridging which can break Web Application / AD FS traffic especially if bridging is used between Web Application Proxy and the AD FS 2012 R2 servers. Please see the following blog for more details on when and why SSL termination can cause issues:-

http://blogs.technet.com/b/applicationproxyblog/archive/2014/07/04/ssl-termination-with-web-application-proxy-and-ad-fs-2012-r2.aspx

For more details on SNI, how to detect if SNI headers are being sent and for options if your load balancer does not support SNI then please see the following blog article:-

http://blogs.technet.com/b/applicationproxyblog/archive/2014/06/19/how-to-support-non-sni-capable-clients-with-web-application-proxy-and-ad-fs-2012-r2.aspx

Using the 0.0.0.0 fallback binding is something we are seeing needed in a number of deployments although our recommendation would be to use SNI where you can.

 

4) HTTP Host Header – this should be the FQDN of the service being tested

This might seem quite an obvious one, but we have seen a number of issues where the Hardware Load Balancer is sending the dedicated IP address of the farm member as the Host Header. This can cause health check failures due to the following:-

- A Web Application Proxy published application will register the external application FQDN with http.sys and will not have a registration for the local dedicated IP address

- AD FS Proxy on the Web Application Proxy will not handle requests where the host header is the local dedicated IP address

- When SNI is used, http.sys will validate that the host header in the request matches the SNI header used during the SSL negotiation

It is therefore very important to ensure that your Load Balancer sends the correct service FQDN as the Host Header in the health check request e.g. if you are health checking the AD FS Proxy service you would send the ADFS Service FQDN as the host header e.g.

Host: adfs.contoso.com

For a Web Application Proxy published application you would set the host header to the External FQDN of the published application.

Not specifying the Host Header or, sending the node IP address as the host header will lead to unpredictable and/or failing health check behavior.

The F5 article mentioned in the SNI section above has the correct host header behavior so if you are using F5 load balancers it’s definitely the place to start. The following Cisco forum discussion talks about setting the Host header in a Cisco health check probe:

https://supportforums.cisco.com/discussion/10406021/setup-ace-probe-http-host-headers-multiple-sites-rservers

It’s important to note that when you are health checking either a Web Application Proxy published application or the AD FS Proxy service, that content is served from the respective back-end server. You are therefore health checking not just the availability of the local service, but also the communications to, and availability of, the relevant back-end service. A failure at the back-end could therefore potentially cause your load balancer to mark a Web Application Proxy node as offline even if other applications are available depending on your HLB configuration.

UPDATE -  as Jaydee points out in his comment if you are using pre-authentication for your Web Application Proxy published application(s) then you will get a 307 Redirect back when you health check your published application URL. This is expected and is Web Application Proxy redirecting to AD FS for authentication.

5) Which URL path to specify in the health check request

As mentioned above, neither Web Application Proxy, or the AD FS Proxy service serve content themselves and content will be retrieved from the back-end service.

For a Web Application Proxy published application, the path specified should be a relevant and valid path on the published application that will return a known response that the hardware load balancer can use for validation e.g. https://extapp.contoso.com/test.html

For health checks  against the AD FS Proxy service (or in fact AD FS itself) the current recommendation from the AD FS Product group is to health check against the following URL:-
https://adfsservicefqdn/FederationMetadata/2007-06/FederationMetadata.xml

Does it really have to be this hard?

While we’ve covered a lot of technical detail in the above the health check issues we’ve been seeing people hit generally come down to one or two of the above areas:

- Does the Hardware Load Balancer support SNI?
If it does then make sure the HLB is correctly configured to send an SNI header. If it does not then using the 0.0.0.0 fallback binding on your AD FS / Web Application Proxy server should provide a workaround.

- Ensuring that the correct Host Header is defined in the health check probe config
Host headers have been in use for many years as a way for a server to differentiate between requests for different sites. All HLB’s should be able to have a host header defined in their health check probes. If you are health checking against AD FS / AD FS Proxy then make sure you send the AD FS Service FQDN as the host header.

If you are aware of these 2 points and ensure these are taken into account when configuring your Hardware Load Balancer then getting your deployment up and running should be a lot smoother.

Disclaimer– while we have provided a couple of links out to HLB vendor sites the best people to advise you how to correctly configure your Hardware Load Balancer will be the relevant vendor. If you have any questions or doubts about how to correctly configure your Hardware Load Balancer please contact your vendor for advice.

 

No, I said “Does it really have to be this hard?”

Well, it is a good question even if you did have to ask it twice 🙂

We’ve definitely heard the feedback that configuring Hardware Load Balancer probes for Web Application Proxy and AD FS 2012 R2 can be tricky. The issues generally boil down to a couple of key points and SNI and hardware load balancers particularly seems to fall into the realm of the mysterious and unknown.

We’re therefore pleased to be able to tell you about some new functionality that shipped in the August 2014 Windows Update rollup for Server 2012 R2 - http://support.microsoft.com/kb/2975719/en-us

 

*NEW* - HTTP based Health Check Probe functionality

The August 2014 Windows Update rollup (http://support.microsoft.com/kb/2975719/en-us) shipped new HTTP based health check probe functionality in ADFS 2012 R2 and the AD FS Proxy service that runs as part of the Web Application Proxy server role.

This avoids any issues with SSL certificates and SNI. It also supports the use of both FQDN’s and IP addresses in the probe helping avoid any potential host header issues. The new HTTP probe can be accessed over HTTP using the path /adfs/probe e.g.

http://<Web Application Proxy name>/adfs/probe

http://<ADFS server name>/adfs/probe

http://<Web Application Proxy IP address>/adfs/probe

http://<ADFS IP address>/adfs/probe

The response to these probe endpoints is an HTTP 200 OK and is served locally with no dependence on back-end services and allows the availability of the local AD FS and AD FS Proxy services to be checked. Health checks for Web Application Proxy published services would still work as detailed in the earlier part of this blog.

These probe endpoints are available by default but on a Web Application Proxy server you will need to create an Inbound Windows Firewall rule to allow the port 80 traffic in to the probe e.g.:

image

AD FS 2012 R2 already has an Inbound Port 80 rule so you do not need to create a manual Windows Firewall rule to use the HTTP probe against AD FS.

While resolving issues with health-checking over SSL due to SNI or host headers issues is relatively straight-forward, it does require that the issues are understood and an accompanying level of information about the associated Hardware Load Balancer configuration.

We hope that providing an HTTP based health check probe option gives an alternate solution that has less complexity to ease your deployments.

 

Summary

Hardware Load Balancer health check requests and AD FS 2012 R2 and Web Application Proxy should be fairly straight-forward but we have seen some challenges in a couple of specific areas:-

- Does the Hardware Load Balancer support SNI?

- Ensuring that the correct Host Header is defined in the health check probe config

Understanding these and taking them into account during your deployment is important to avoid health check request failures. If you are unsure about whether your load balancer supports SNI or, how to configure your load balancer to meet the above requirements, please do reach out to your HLB vendor. We’ll try to share what we can here but the best people to advise on how to configure your HLB correctly is your HLB vendor.

If you come across any good info on configuring any of the Hardware Load Balancer variants out there or links to relevant vendor configuration steps please feel free to share and comment below.

We have also added new functionality in the August 2014 Windows Update rollup for Server 2012 R2 (http://support.microsoft.com/kb/2975719/en-us) to provide HTTP based health check probes to provide another health check option, which avoids some of the complexities such as SNI and host header configuration.

We’ll look forward to any feedback, comments and questions you have and we’ll be back with more content in the near future.

 

Ian Parramore, Senior Escalation Engineer, Web Application Proxy support team

Comments (10)

  1. Hey Jaydee,

    Thanks for the comments and for sharing your config. What HLB was this for and do you know if it used SNI or did you use the 0.0.0.0 fallback cert to get this to work?

    A HEAD request should be fine as long as your HLB is configured to see the 404 as a successful response. Did you try sending a HEAD for the federationmetadata.xml as at the moment you’re just sending a root request for / which is probably why you’re getting
    a 404.

    Good point about the 307 when using pre-auth at WAP – I’ll update this in the blog as it’s something I overlooked.

    Hopefully the new HTTP probe endpoints will make things a lot easier although you can still use SSL based probes if this meets your needs better.

    Thanks,

    Ian

  2. jaydee says:

    Hi Ian,

    thank you for a great post again!

    We also thought to use the federationmetadata as probe but we currently have used a simple HEAD to / since much less data needs to be transferred (compare the content-lengths below). The drawback is, that you only get a 404 http status code, but I assume this
    is sufficient to test, if the service is up.

    Provided your federation service URL is https://adfs.contoso.com and the IP of the server (WAP or ADFS) is 1.2.3.4, you can test your loadbalancer probes via openssl:

    HEAD-probe Content-Length: 315:

    openssl s_client -connect 1.2.3.4:443 -servername adfs.contoso.com
    HEAD / HTTP/1.1
    host: adfs.contoso.com

    HTTP/1.1 404 Not Found
    Content-Length: 315
    Content-Type: text/html; charset=us-ascii
    Server: Microsoft-HTTPAPI/2.0
    Date: Fri, 17 Oct 2014 08:15:12 GMT
    Connection: close

    Federationmetadata probe – Content-Length: 71968:

    GET /FederationMetadata/2007-06/FederationMetadata.xml HTTP/1.1
    host: adfs.contoso.com

    HTTP/1.1 200 OK
    Content-Length: 71968
    Content-Type: application/samlmetadata+xml
    Server: Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0
    Date: Fri, 17 Oct 2014 08:15:21 GMT

    When probing your published applications, the returned http status code depends on the preauthentication settings of the WAP endpoint. You will get a 307 Temporary redirect for applications with preauthentication ADFS:

    Preauthentication ADFS:

    openssl s_client -connect 1.2.3.4:443 -servername publishedapp.contoso.com
    HEAD / HTTP/1.1
    host: publishedapp.contoso.com

    HTTP/1.1 307 Temporary Redirect
    Content-Length: 0
    Location:
    https://adfs.contoso.com/adfs/ls?version=1.0&action=signin&realm=urn%3AAppProxy%3Acom&appRealm=de673a0d-ac18-e411-80db-00505686504d&returnUrl=https%3A%2F%2Fpublishedapp.contoso.com%2F&client-request-id=36E19C2B-E72D-0000-4381-E2962DE5CF01
    Server: Microsoft-HTTPAPI/2.0
    Date: Fri, 17 Oct 2014 08:16:43 GMT

    Cheers,
    JD

  3. jaydee says:

    Hi Ian,

    we use F5 LB and have configured the SNI-monitor based on the F5 article you mentioned above (just configured different expected http status codes).

    I also thought about a HEAD to the federationmetadata as the probe/monitor but this only gives a 500 http status code:

    HEAD /FederationMetadata/2007-06/FederationMetadata.xml HTTP/1.1
    host: adfs.contoso.com

    Response from WAP

    HTTP/1.1 500 Internal Server Error
    Content-Length: 0
    Content-Type: text/html
    Server: Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0
    Date: Fri, 17 Oct 2014 13:56:40 GMT

    Response from ADFS

    HTTP/1.1 500 Internal Server Error
    Transfer-Encoding: chunked
    Content-Type: text/html
    Server: Microsoft-HTTPAPI/2.0
    Date: Fri, 17 Oct 2014 14:01:18 GMT

    Although it is good to know that there are the HTTTP probe endpoints, I still prefer the SNI probes (if the loadbalancer supports these), since this really probes the same endpoint the user is accessing.

    JD

  4. turbomcp says:

    Thanks
    great stuff

  5. Anonymous says:

    In the previous blog post on comparing sign-in protocols, the goal was to give you a primer on the terminology

  6. Anonymous says:

    In the previous blog post on comparing sign-in protocols, the goal was to give you a primer on the terminology

  7. Anonymous says:

    In the previous blog post on comparing sign-in protocols, the goal was to give you a primer on the terminology

  8. Anonymous says:

    The last blog was about planning for ADFS and what questions you should be asking when deploying it.

  9. Thomas Brown says:

    Hello,

    since WAP is now also listeing on http for health probes. What is the recommended way to do http to https redirects for URLs? Via IIS, will that not interfere with the port 80 health probe.
    Thanks,
    Thomas

  10. shubham arora says:

    We are using http based probes for health monitoring of our web application proxy servers. Everything seems to be working fine. However, we could see following errors being logged in HTTPERR log files.

    2017-01-19 07:28:05 xx.xx.xx.xx 49680 xx.xx.xx.xx 443 HTTP/0.9 GET / 404 – NotFound –
    2017-01-19 07:28:06 xx.xx.xx.xx 46051 xx.xx.xx.xx 443 HTTP/1.1 GET /adfs/probe 503 – N/A –
    2017-01-19 07:28:06 xx.xx.xx.xx 58997 xx.xx.xx.xx 443 HTTP/1.1 GET /adfs/probe 503 – N/A –
    2017-01-19 07:28:09 xx.xx.xx.xx 54102 xx.xx.xx.xx 443 HTTP/0.9 GET / 404 – NotFound –
    2017-01-19 07:28:10 xx.xx.xx.xx 32608 xx.xx.xx.xx 443 HTTP/0.9 GET / 404 – NotFound –
    2017-01-19 07:28:10 xx.xx.xx.xx 40583 xx.xx.xx.xx 443 HTTP/1.1 GET /adfs/probe 503 – N/A –
    2017-01-19 07:28:12 xx.xx.xx.xx 62366 xx.xx.xx.xx 443 HTTP/1.1 GET /adfs/probe 503 – N/A –
    2017-01-19 07:28:14 xx.xx.xx.xx 26515 xx.xx.xx.xx 443 HTTP/0.9 GET / 404 – NotFound –

    Is it something we should worry about?

Skip to main content