TMG URL Filtering category precedence

Introduction

Forefront TMG 2010 introduced URL filtering, which enables administrators to create rules that allow or block access to Web sites based on their categorization in the URL filtering database. When a request to access a Web site is received, Forefront TMG queries the remotely hosted Microsoft Reputation Service (MRS) to determine the categorization of the Web site. If the Web site has been categorized as a blocked URL category or category set, Forefront TMG blocks the request.

If a user requests access to a Web site and discovers that access to the Web site is blocked, he receives a denial notification that includes the URL category which resulted in the denied request. In addition, sites can be excluded from HTTPS and malware inspection based on their category.

The Forefront TMG URL filtering mechanism uses URL categorization provided by the MRS Web service. Some URLs have multiple categories, for instance http://finance.yahoo.com is categorized as
Financial, Online Trading and News. Forefront TMG’s policy and its rule engine are based on a single category per URL. This means that in case the MRS responds with multiple categories per URL, Forefront TMG will need to choose one of those categories as the “most relevant” URL category. In order to do that, Forefront TMG uses a pre-defined category precedence list.

Category precedence list

Multiple categories for a single requested URL are sent back by the MRS web service with no concept of prioritization or order. However, Forefront TMG uses single-URL categorization in its policy. Therefore, we need a mechanism to choose “most relevant” category from a set of URL categories provided by MRS. For that task Forefront TMG has a category precedence list, where categories are ordered by significance. The rule of thumb is that more malicious, harmful and non-productive categories have higher precedence.
The list is pre-defined and can’t be changed by administrators. The list for Forefront TMG SP1 is below.   

No.

Category

1

    "Malicious"

2

    "Pornography"

3

    "Botnet"

4

    "Phishing"

5

    "Criminal Activities"

6

    "Hate/Discrimination"

7

    "Anonymizers"

8

    "Spyware/Adware"

9

    "Illegal Drugs"

10

    "Violence"

11

    "Obscene/Tasteless"

12

    "Gambling"

13

    "Spam URLs"

14

    "Dubious"

15

    "Hacking/Computer Crime"

16

    "School Cheating Information"

17

    "P2P/File Sharing"

18

    "Personal Network Storage"

19

    "Remote Access"

20

    "Shareware/Freeware"

21

    "Nudity"

22

    "Mature Content"

23

    "Weapons"

24

    "Alcohol"

25

    "Tobacco"

26

    "Search Engines"

27

    "Financial"

28

    "Online Trading/Brokerage"

29

    "Government/Military"

30

    "Employment"

31

    "Online Communities"

32

    "Digital Postcards"

33

    "Chat"

34

    "Portal Sites"

35

    "Usenet News"

36

    "Web E-mail"

37

    "Web Phone"

38

    "Web-based Productivity Applications"

39

    "Education/Reference"

40

    "Child Friendly Materials"

41

    "Public Information"

42

    "Technical Information"

43

    "Health"

44

    "Art/Culture/Heritage"

45

    "General Entertainment"

46

    "Games"

47

    "Humor/Comics"

48

    "Recreation/Hobbies"

49

    "Special Interests"

50

    "Restaurants/Dining"

51

    "Social Opinion"

52

    "Self Defense"

53

    "Travel"

54

    "Fashion/Beauty"

55

    "Motor Vehicles"

56

    "Shopping"

57

    "Real Estate"

58

    "Legal Services & Reference"

59

    "Non-Profit/Advocacy/NGO"

60

    "Politics/Opinion"

61

    "Religion/Ideology"

62

    "Edge Content Servers/Infrastructure"

63

    "Dating/Personals"

64

    "Sports"

65

    "Free Hosting"

66

    "Internet Services"

67

    "Web Ads"

68

    "Media Sharing"

69

    "Streaming Media"

70

    "Forum/Bulletin Boards"

71

    "News"

72

    "Blogs/Wiki"

73

    "General Business"

74

    "Parked Domain"

75

    "Unknown"

When Forefront TMG receives an HTTP request, it retrieves its URL category from MRS or from internal cache. If the URL has several categories, Forefront TMG applies category precedence rules according to the precedence list. The category with the highest precedence is used by the Forefront TMG rule engine, while all other categories are disregarded.

Let’s see an example. When a user browses to http://msdn.microsoft.com, MRS categorizes that URL as General Business and Technical Information,
as can be seen from the MRS portal at http://www.microsoft.com/security/portal/mrs/

clip_image001

Since “Technical Information” has higher precedence than “General Business”, TMG will use the “Technical Information” category for that URL. The “Technical Information” category will be applied for rules, will appear in log/reports and will be presented to users in denial pages.  It will also be matched to HTTPS inspection and malware protection exemptions categories.
We can use the Forefront TMG UI Category Query tool to validate that.

clip_image003

Summary

In this blog, I showed that although MRS provides several categories for each URL, Forefront TMG rules engine decisions are based on one category only.
This “most relevant” category has the highest precedence in the pre-defined precedence list. Administrators can verify which category was chosen by Forefront  TMG using the Log or Query Category UI.

 

Author: Igor Zarivach
Reviewers: Ori Yosefi, Roman Golubchyck