IPSEC Domain Isolation: A Test Study

The Purpose Hello,

Our names are David Pracht and Steve Martin.  As Networking Support Professionals at Microsoft we support IPSec but historically it has not been a high call generator.  We designed this lab to explore an increasingly popular scenario – IPSec Domain Isolation. While it can be the most difficult scenario to deploy it is also very tempting to have the ability to protect all the traffic in your network without requiring specific application support.  The reality is somewhere in between and we wanted to see if we could identify where people might encounter issues and document in a series of posts any problems we uncover while attempting to setup this scenario.

Domain Isolation vs. Server Isolation

IPSec provides technological support to implement a number of scenarios that improve enterprise network security:

■ Secure Server to Server: IPSec can be used to encrypt traffic between two servers.  An example of this is Outlook Web Access and Exchange.  All communications between the OWA server and the Exchange server could be authenticated and encrypted.

■ Server Isolation: IPSec can be used to isolate a server from unauthenticated (and possibly rogue) clients.  A good example of this is a line of business application server.  The application server would only grant access to machines that belong to the domain.  All other clients would not be able to even establish a TCP connection; guaranteeing the application server is isolated from the unknown clients.

■ Domain isolation: IPSec can be used to isolate domain members from non-domain members.  All domain members would be able to connect to each other securely.  Non-domain members would not be able to connect to any domain machine, as they are not successfully authenticated.  However, domain members may be able to connect to non-domain servers.

Why Domain Isolation is becoming more popular

Despite the historical difficulties in deploying an administering IPSec it has some compelling features and is becoming easier to implement.

Here are some of the benefits provided by IPSec:

■ Defense-in-depth against vulnerabilities in upper-layer protocols and applications.

IPSec protects upper layer protocols, services, and applications.  With IPSec enabled, initial communication packets to access an application or service running on a server, for example, will not be passed to the application or service until trust has been established through IPSec authentication and the configured protection on packets for the application or service have been applied.  Therefore, attempts to attack applications or services on servers must first penetrate IPSec protection.

■ Requiring peer authentication prevents communication with untrusted or unknown computers.

IPSec security requires peers to authenticate their computer-level credentials prior to sending any IP-based data.  By requiring peer authentication using credentials based on a common trust model, such as membership in an Active Directory domain, untrusted or unknown computers cannot communicate with domain members.  This helps protect domain member computers from the spread of some types of viruses and worms being propagated by untrusted or unknown computers.

■ IP-based network traffic is cryptographically protected.

IPSec provides a set of cryptographic protections for IP-based traffic based on your choice of AH, ESP without encryption, or ESP with encryption.  Your IP-based network traffic is either tamper proofed (using AH or ESP with no encryption), or tamper proofed and encrypted (with ESP and encryption).  Requiring cryptographic protection of IP traffic helps prevent many types of network attacks.

■ Applications do not need to be changed to support IPSec.

IPSec is integrated at the Internet layer of the TCP/IP protocol suite, providing security for all IP-based protocols in the TCP/IP suite. With IPSec, there is no need to configure separate security for each application that uses TCP/IP.  Instead, applications that use TCP/IP pass the data to IP in the Internet layer, where IPSec can secure it.  By eliminating the need to modify applications, IPSec can save application development time and costs.

In short if you need security IPSec is the way to protect you network.

Why Domain Isolation is difficult to implement

In the past with Windows Server 2003 and Windows XP, all these scenarios rely on machine-level authentication, which is what the IKE protocol that is supported by these operating systems supports.

Note: In addition to IKE Windows Vista and Windows Server 2008 support a new keying protocol called AuthIP.

IPSec policy configuration in many scenarios, such as server isolation and domain isolation, consists of a set of rules to protect most of the traffic on the network and another set of rules for protected traffic exceptions.

Exceptions are needed for unprotected communication with network infrastructure servers such as DHCP, DNS, and Domain Controllers.  For example: When a computer is starting, it must be able to obtain an IP address, use DNS to find a domain controller, and then log in to its domain before it can begin to use Kerberos authentication to authenticate itself as an IPSec peer.

In some cases, there are dozens or even hundreds of exceptions, which makes it difficult to deploy IPSec protection on a private network and to maintain it over time.  There is an optional feature called “Fallback to Clear” but the 3 second delay it introduced was often too long for networking scenarios like obtaining an IP address to complete.

Note: In Windows Server 2003 and XP this was addressed by the Simplified IPSec Policy Configuration update.

914841 How to simplify the creation and maintenance of Internet Protocol (IPsec) security filters in Windows Server 2003 and Windows XP



That sums up why we are taking on this adventure and hopefully we will be able to provide some insight for other people planning to implement IPSec Domain Isolation.

Next post – We will define our scenario and see what issues come up that we will need to address.

David Pracht – Support Escalation Engineer

Steve Martin – Support Engineer