How to enable multicast routing with ISA Server

  • Article

This blog entry will talk about how you can enable multicast traffic with ISA Server. You face a scenario where you have multicast capable applications in different subnets which are isolated by an ISA Server and you need to allow the multicast traffic to pass the ISA Server. The following drawing outlines such a sample scenario:

 

As the first step you need to make sure that you define a routing rule in ISA Server for Network 1 and Network 2 that has a Route relationship. Secondly you need to know how ISA Server interacts with the OS in this particular case here. In order to allow multicast routing RRAS will need to listen to IGMP messages. ISA will rely on RRAS to handle this part of the communication and does not interfere here at all.

Here are the Steps I used:

1. enabled VPN Client Access in the ISA Server MMC ( this is just to let ISA Start RRAS )

2. Enable IGMP Routing in RRAS (see also http://technet.microsoft.com/en-us/library/cc775881.aspx)

a. open the RRAS MMC - go to - In the console tree, open Routing And Remote Access,
the server name, and then IP Routing.

b. In the console tree, right-click General, and then click New Routing Protocol.

c. In the Select Routing Protocol dialog box, click IGMP Router And Proxy, and then
click OK.

d. After the IGMP routing protocol is added, you must add router interfaces by doing
the following:

e. the console tree, right-click IGMP, and then click New Interface.
- in Interfaces, Add all interfaces (external, internal and local host), and then
click OK
Please note that it is also sufficient to select only the interfaces where you are planning to route the traffic. So in this example it would the Interface which is connected to Network1 and the Interface which is connected to Interface2

3. In the next step we need to tell ISA Server to allow IGMP
In the ISA MMC go to Firewall Policy - Tool Box
- Create a new Protocol with the following details:
IGMP - IP Protocol ID 2, send-receive

4. With the new protocol you create a new access rule to allow the IGMP traffic to be processed by ISA server. Please note that you need to configure the network set ‘All Networks (including local host’ as the IGMP traffic is sourced from a either a multicast source IP address or a unicast IP address and the local RRAS instance we have enable in step 1 will need to process this traffic

5. Now we need to tell ISA Server over which protocol the application communicates and create the proper access rules. This depends completely on the application you are planning to use. Please contact the application vendor for the ports used if you are in doubt.
In my example the application used UDP over Port 200:

After you apply the changes to the ISA configuration and the Array is synced with the CSS (in case you are using an Enterprise Edition) your multicast application should work just fine J

Summary:

As you have seen there are a few steps needed to enable multicast routing with ISA Server . Please remember that these steps I have provided are not covered by the overall product testing and the ISA Development team did not had this scenario in the scope of the design specification of ISA server 2004/2006 and therefore may refuse to fix any broken scenarios. So far I am aware of a handful of customers where this works just fine and we do not anticipate any issues to be seen here J

Thomas Detzner

Escalation Engineer

Microsoft CSS Forefront Security Edge Team

 

Thanks to Doron Juster for the technical review.