ADFS Certificate Maintenance - v1
Over the past several weeks, we've celebrated the 1 year anniversary of our ADFS deployment. I say it this way, because the only reason I know this, is that the certificates on the servers keep expiring, and things would break unexpectedly. Yeah, yeah, yeah...I could have, would have, and SHOULD have been a bit more proactive about this, but our use of ADFS internally is somewhat limited until this summer (aka - fiscal year and budgets) when we're going to start onboarding a lot of new ADFS based services for our users.
So, I wanted to get an idea of where I needed to update the certs. Since we use a common cert for both the ADFS trust policy signing certificate, as well as the IIS SSL cert, I needed to make sure I replaced them both. We also use an FS Proxy on the internet, against SSL and policy signing cert required here. This all made sense to me, so off I go:
- Change the SSL cert in the IIS console of the FS
- Change the SSL cert in the IIS console of the FS-P
- Change the token signing cert on the FS using the ADFS MMC
- Change the token signing cert on the FS-P using the ADFS MMC
(note: pet peeve of mine, which I'll probably rant about again... but we can't remotely administer the ADFS servers using the MMC. You have to TS onto the box, or write your own scripts to do things remotely.... grrrrrr... yes, the product group knows, yes I remind them every chance I get...unfortunately no, it didn't make it into R2.)
Ok, so at this point, I'm thinking to myself, "cool, this annual maintenance is done." Probably took 7 full minutes before my phone rang that EVERYTHING was broken. Seems that I forgot that a year ago when we set this up, I had to send a copy of the token signing cert to the federation partners. Their FS-R's need to be able to validate the cert. Ok, drop that in e-mail, follow-up with a phone call...everything's working from internal now (ie. users connecting directly to the FS server). RDP'd back to my home computer, so I could get a view of this thing from the internet though and it's still failing.
Perusing the event logs on the FS-P (which look like a Christmas tree...if Christmas colors were red and yellow) I find:
The Federation Service Proxy encountered an exception when it called a Federation Service Web method.
Federation Server URL: https://corp.sts.microsoft.com/adfs/fs/FederationServerService.asmx
Web method: GetProxyTrustConfiguration
Proxy certificate thumbprint:
<snipped by brian>This may cause a user request to fail.
User Action
The exception details may give an indication of the precise problem.Check network connectivity between the Federation Service Proxy and the Federation Service.
Ensure that the Federation Service is running.
Ensure that the Federation Service Proxy client authentication certificate has been added to the list of proxy authentication certificates in the Federation Service trust policy.
Ensure that the Federation Service Proxy client authentication certificate chains to a root that is trusted by the Federation Service.
Ensure that the Federation Service Internet Information Services (IIS) Secure Sockets Layer (SSL) server certificate chains to a root that is trusted by the Federation Service Proxy.
Ensure that the Federation Service Uniform Resource Locator (URL) that is configured in the Federation Service Proxy web.config uses the name that is the subject of the Federation Service IIS SSL server certificate.
Additional Data
Exception details:
System.Web.Services.Protocols.SoapException: System.Web.Services.Protocols.SoapException: Server was unable to process request. ---> System.UnauthorizedAccessException: Attempted to perform an unauthorized operation.
at System.Web.Security.SingleSignOn.FederationServerService.MethodInvocationCheck(MethodAuthenticationLevel DesiredAuthentication)
at System.Web.Security.SingleSignOn.FederationServerService.GetProxyTrustConfiguration(VersionInformation proxyVersion, VersionInformation& fsVersion, ProxyInformation& proxyInformation, TrustConfigurationData[]& trustConfig)
--- End of inner exception stack trace ---
at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
at System.Web.Security.SingleSignOn.FederationServerSoapProxy.GetProxyTrustConfiguration(VersionInformation proxyVersion, VersionInformation& fsVersion, ProxyInformation& proxyInformation, TrustConfigurationData[]& trustConfig)
at System.Web.Security.SingleSignOn.LSPersistentState.GetPolicy(VersionInformation& fsVersion, ProxyInformation& proxyInformation, TrustConfigurationData[]& data)For more information, see Help and Support Center at https://go.microsoft.com/fwlink/events.asp.
I've bolded the important part above. When I zipped right back to the FS server where I found the following event:
Description:
The Federation Service failed a privileged Web method call because the caller's client authentication certificate is not configured as a Federation Service Proxy certificate.
Certificate thumbprint: <snipped by Brian>User Action
Ensure that the trust policy is properly configured with all valid Federation Service Proxy certificates.For more information, see Help and Support Center at https://go.microsoft.com/fwlink/events.asp.
Back to the ADFS snap-in, and in the trust policy there is an FSP Certificates tab. Added the token signing cert here as well so everything matched, and voila! auth started working again.
There were a couple of reasons I wanted to jot this down:
- As the ADFS service scales, managing the certificates is going to become significantly more important and require a lot of advanced planning. While I'm a HUGE fan of documenting and testing operational processes, etc... The fact that this will require coordination between the FS-A and the FS-R provider, and that this once-a-year function requires so many touch points, is just scary.
- I wanted to point out the quality of the events that are being logged by the service. The ADFS team put a lot of thought into getting the "right" information into the logs, so an administrator can quickly figure out what's going on with the service. There is still granular debug logging available, but you shouldn't really need to use it that often. IMHO, too many admins are jaded against the event logs anyways (rightfully so?), but I still believe that if eventvwr isn't your first stop, then you're probably not doing your job right. For federation services, this is definitely the case. ADFS Product Team - THANK YOU.