IIS on ISA, just one more reason to avoid this type of setup

There are many reasons for me to not enjoy the idea of having IIS on the same box as ISA Server. First and foremost you need to remember that ISA is a Firewall so you shouldn’t use as web server anyway (put your web server behind ISA). Another reason is the resource allocation failure that happens when ISA tries to grab a port that is being use by IIS. Thanks to ISA product team for showing a red alert on ISA Console when it detects that (see below).

 

This week I got another good reason for not using this setup and here a brief story about this:

“Once upon a time, there was a system admin that just got hire to work for Fabrikam. He was happy with his new job when he found out that he inherited an environment that was about to explode on his hand. One day he restarted his ISA Server as part of a maintenance window and when it came back ISA Server services were not starting. No internet access, no inbound or outbound emails, no OWA…chaos”

Troubleshooting

Reviewing the event viewer it was possible to see the following sequence on system log from 10/4:

Event Type: Error

Event Source: Service Control Manager

Event Category: None

Event ID: 7023

Date: 10/4/2009

Time: 7:36:10 PM

User: N/A

Computer: ISASRVSTD

Description:

The IIS Admin Service service terminated with the following error:

Access is denied.

Event Type: Error

Event Source: Service Control Manager

Event Category: None

Event ID: 7001

Date: 10/4/2009

Time: 7:36:10 PM

User: N/A

Computer: ISASRVSTD

Description:

The HTTP SSL service depends on the IIS Admin Service service which failed to start because of the following error:

Access is denied.

Event Type: Error

Event Source: Service Control Manager

Event Category: None

Event ID: 7001

Date: 10/4/2009

Time: 7:36:10 PM

User: N/A

Computer: ISASRVSTD

Description:

The World Wide Web Publishing Service service depends on the IIS Admin Service service which failed to start because of the following error:

Access is denied.

Clearly we have IIS on this box although system admin didn’t know why this box has IIS and why it was broken before and now is working just fine. For day 10/06 we have the following failure from ISA and all was clear from IIS side:

Event Type: Error

Event Source: Service Control Manager

Event Category: None

Event ID: 7024

Date: 10/6/2009

Time: 7:36:10 PM

User: N/A

Computer: ISASRVSTD

Description:

The Microsoft Firewall service terminated with service-specific error 2148073494 (0x80090016).

The application log also has entries about ISA Services failure to start:

Event Type: Error

Event Source: Microsoft Firewall

Event Category: None

Event ID: 14060

Date: 10/6/2009

Time: 7:35:38 PM

User: N/A

Computer: ISASRVSTD

Description:

ISA Server could not load the application filter Web Proxy Filter ({4CB7513E-220E-4C20-815A-B67BAA295FF4}). FilterInit failed with the error code 0x80090016. To attempt to activate this application filter again, stop and restart the Firewall service.

Event Type: Error

Event Source: Microsoft Firewall

Event Category: None

Event ID: 14001

Date: 10/6/2009

Time: 7:35:38 PM

User: N/A

Computer: ISASRVSTD

Description:

Firewall Service failed to initialize. Previous event log entries might help determine the proper action.

The current status now is:

· IIS up and running.

· ISA down and crying out loud.

The event 7024 from Microsoft Firewall was triggering the error 0x80090016, which means: object already exists. Using this error code I found the article below:

You receive error message 0x80090016 or error message 0x8009000f when you try to schedule a task

That’s a very interesting point, because ISA does read the Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys folder as explained in the troubleshooting setup article:

“Before removing ISA Server, be sure to close ISA Server Management and ISA Server Performance Monitor.

If the storage is corrupted, as part of the procedure, you will also have to specify the array membership and Internal network configuration. When you install a certificate, a private key container is created on the Configuration Storage server in the Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys folder. The account running ISASTGCTRL service (by default, the Network Service account) must have appropriate permissions to the private key container. Certificates typically have an expiration period, usually no more than one year. ISA Server cannot use an expired certificate. Be sure to renew your certificates before they expire, so that ISA Server can continue to function.”

I went to this folder to see what we have there and found something weird:

Very odd! However after many questions and investigations, myself and the system admin found out that the previous admin used the KB884872 to fix an IIS issue and ended up breaking ISA. The content of MachineKeys folder just had 1 file on it while the MachineKeys.old has all the other keys and by default MachineKeys folder stores certificate pair keys for both the computer and users.

Note: This is a very sensitive folder and before playing around with it beware that you understand the consequences of changing anything on it. Read KB278381 for more information about usability and permissions needed for this folder.

The solution was quiet simple: rename the MachineKeys folder to MachineKeys.tmp and rename the original MachineKeys.old to MachineKeys, after that Firewall Service started just fine. Now you are probably asking: so you broke IIS again by doing this procedure? Yes, I did (on purpose). The bottom line here was that this ISA didn’t need IIS anyway so we ended up uninstalling IIS from this box (YAY).

Conclusion

From now on, instead of thinking twice before install IIS on ISA, think ten times before doing that. Much better having another Windows Server 2003 box dedicated to be Web Server.