URL Filtering is Here!

Quick Introduction

Hi Everyone,

We are proud to introduce as part of Forefront TMG Beta3 one of the key elements in the Secure Web Gateway (SWG) role of Forefront TMG – URL Filtering.

URL Filtering allows controlling end-user access to Web sites, protecting the organization by denying access to known malicious sites and to sites displaying inappropriate or pornographic materials, based on predefined URL categories.

The typical use case for this feature includes:

· Enhancing your security.

· Lowering liability risks.

· Improving the productivity of your organization.

· Saving network bandwidth.

The URL Filtering administration experience is pretty straightforward. All you need to do after enabling the feature is add one or more of the predefined URL categories into Forefront TMG policy (you can find some UI snapshots further below). Once this is done, end-users browsing to a Web site included in one of those categories will be blocked and presented with a relevant notification page, which you can customize.

Additional value can be obtained from URL Filtering related reports and log entries. Have you ever wanted to understand how Web usage in your organization is distributed? And how about identifying those users who consistently violate your Web usage policy? You can do those easily now by looking at the built-in URL filtering reports.

Finally, URL Filtering categories can also be leveraged to exclude sites from being inspected by the HTTPS Traffic inspection and the Malware Inspection features. For instance, you may wish to exclude financial sites from HTTPS inspection, due to privacy considerations.

Before going into further details, note that the feature is still in Beta, so we do expect significant improvements in coverage and accuracy by the final TMG release.

URL categorization data, where does it come from?

TMG features over 80 URL categories ranging from security-oriented selections, like Phishing, Malicious and Anonymizers, through productivity-oriented categories such as Games, or Instant Messaging, and ending with liability-oriented categories like Criminal Activities and Pornography. Categories are also grouped into a higher-level hierarchy which we call Category Sets. The latter can also be used in TMG policy to simplify configuration.

As some of you may have noticed, at the RSA 2009 Conference Microsoft announced its new reputation services and its intention to provide these capabilities for our security products and solutions.  Microsoft also announced several key partnerships in the URL filtering space that will be used to support these reputation services. Forefront TMG will be the first system at Microsoft to leverage and utilize Microsoft Reputation Service (MRS).

MRS is a cloud-based object categorization system hosted in Microsoft data centers and designed to provide comprehensive reputation content to enable core trust scenarios across Microsoft solutions. In the case of Forefront TMG, in order to find out the category of a URL, TMG issues an online query to MRS. MRS maintains a database with tens of millions of unique URLs and their respective categories.

Does this mean every end-user request is sent out to the cloud? No it doesn’t. To improve bandwidth utilization and performance, we have implemented a local cache (residing on a TMG server), that stores the recently queried URLs and their respective categories. Cache entries are subject to a time-to-live value, allowing refreshing the entry periodically. This local cache is expected to serve the overwhelming majority of user requests.  The cache is persistent so it doesn't need to be refreshed after each reboot. TMG will query MRS only when a request cannot be served from the local cache.

But that's only the tip of the iceberg. Read on to find out why we think we are building something special with TMG and MRS together.

What is so special about Microsoft Reputation Service (MRS)?

The MRS team wanted to confront an inherent problem with traditional URL Filtering solutions: the problem domain is simply too large for any single vendor to provide a complete solution on its own. As a result, there are multiple vendors out there, each one specializing in a specific area of the solution.

Some vendors specialize in identifying malicious sites and spam URLs; others are rich with productivity related categories. Some specialize in covering the Internet's “long tail”; others are great with quick classification of previously unknown sites. Some use human-based classification where others use machine-based techniques. Some are great with Web2.0 style URLs… OK, I'll stop here as you get the idea by now. Even those vendors who employ several classification techniques and cover multiple categories can't deal with the huge and ever-expanding challenges of today's Web.

MRS team's idea was simple; let's leverage complementary capabilities of different vendors/sources to create a unified database that is best suited to deal with the challenges described above. And so, they have implemented a scalable architecture that allows incorporating multiple streams of data into a merged database. This way – each vendor/source brings its unique strengths to the table into a common solution.

MRS already integrates several data sources and others will be on-boarded in the following months. Some of these data sources are Microsoft internal, and others are the result of collaboration with 3rd party partners. One such agreement, announced during RSA, is an agreement with Marshal8e6.  Other agreements have not been disclosed yet. Expect some surprises...

But the real beauty is that being a Web service, and given its unique architecture, MRS can easily incorporate new DBs completely transparently to the customers. We expect the MRS unified database to expand over time and become the recognized industry leader. TMG customers will benefit naturally from this ongoing upgrade, through our Web security subscription services.

Other interesting aspects – security ,privacy, licensing

Security – Both Forefront TMG URL Filtering and MRS were designed with security in mind, following Microsoft's Security Development Lifecycle (SDL) strict standards and guidelines. Both are resilient to a variety of attacks, and the communication between the two is encrypted.

Privacy – this is a known concern when discussing cloud based services, and therefore the privacy of our customers' data is paramount. We are issuing detailed privacy statements along with the Beta 3 release to provide clarity and transparency on our privacy policies. Make sure to read those.

Licensing – URL Filtering is subscription based, and is part of the Forefront TMG Web Security Service license (together with the Malware Inspection updates).

The small (but important) things

As this is a high-level overview of the feature, we will not dive into all the small details that make for a complete, rich user experience. We will cover some of those in subsequent posts, as we go along. But here are few examples for flexibility you are likely to need/want when working with URL Filtering:

- You can locally override a URL category

- You can query for a URL's category in the TMG UI

- You can customize the block page displayed to end-users, introducing your own HTML tags into the text area.

- You can leverage URL Filtering for ad blocking

- You can use the build-in TMG scripting capabilities to allow non-TMG administrators to locally override a URL (enabling advanced help-desk scenarios)

- You can use URL Filtering related reports to figure out how your organization uses the Internet (which are the top browsed categories for instance)

- You can report classifications issues to Microsoft (this one is not available in Beta3)

A sneak peek at the UI

TMG Web Access Wizard allows you to easily introduce URL categories into your policy:


This is how the policy may look like after completing the Web Access Wizard (viewed from the Web Access Protection node). Note that URL Categories are standard TMG network objects, so you can use the toolbox on the right to drag-drop additional categories into an existing rule, or to create new rules.


You can query for a URL's category (available as a task in the Web Access Protection node)


You can locally override a URL's category (available as a task in the Web Access Protection node)


You can customize the block page presented to end users, introducing your own HTML tags (this is a per-rule setting available from the ‘Action’ tab of the rule’s properties)



Thanks for reading all the way! I hope you have found this post helpful. We are certainly interested in your comments and thoughts!! Feel free to post them below.


Dotan Elharrar, Program Manager, Forefront TMG



Yair Geva, Senior Development Lead

Vladimir Holostov, Senior Program Manager Lead

Nathan Bigman, Content Publishing Manager

Bill Jensen, Senior Product Manager

Yigal Edery, Principal Group Program Manager

David B. Cross, Product Unit Manager

Comments (27)

  1. Anonymous says:

    I have seen this question a few times on both  internal and external lists so I figured I would

  2. Anonymous says:

    Sta procedendo il lavoro del team di sviluppo sulla nuova versione di ISA Server, che prenderà il nome

  3. bayrak says:

    Introducing such a topic you’d like to congratulate you’ve let us know. Have good work

  4. Anonymous says:

    This and similar information-sharing my vocabulary expanded again. Thank you for your writing …

  5. Anonymous says:


    I did a test with URL Filtering and use both proxy and secureNAT clients and it’s working perfect!

    TMG make the name resolution of the secureNAT client request and categories the URL. But why can’t TMG show the URL in the log on the same way as for the proxy client? The TMG makes the name resolution after all.

    Best regards,


  6. We will disclose more details as TMG approaches its release to market.  


    Dotan Elharrar

  7. Anonymous says:

    The MRS only can categorize the URLs (DNS-Name). If a user get blocked for a specific URL he just has to type in the IP address of the URL and the MRS says OK because it’s unknown. Will the IPs be added to the MRS?


  8. Yes. The next version of Windown Essential Business Server will include TMG URL Filtering.


    Dotan Elharrar

  9. Anonymous says:

    Will this be included in Essential Business Server ??

  10. adimcev says:

    And what about the google search(images, videos) porn blocking for example ?

    Either I miss some setting or there isn’t one. And, occasionally, one or two links(with pics or videos) found like so managed to slip through TMG. But usually they were blocked, the search images or "video thumbnails" were displayed though(and they aren’t quite small).

    Also, some normally blocked URLs containing porn can be displayed using the "from cache" option say when searching with google, although the images/videos may not appear, the chats/text or so on those pages are displayed and the user can read that "info".

    Failing to filter search engine images does not affect CIPA compliance ?

    Ignoring these aspects, the porn blocking URL category seemed to work fine.



  11. TMG does not support ‘Safe Search’ enforcement out-of-the-box. However, in some cases this can be easy to apply. For more details, pls see: http://blogs.technet.com/isablog/archive/2009/06/19/bing-safe-search-isa-server-and-forefront-tmg.aspx


    Dotan Elharrar

  12. Thanks for the feedback! In the case of SecureNAT clients, TMG does not perform name resolution for the purposes of the connection. You can check the link below for more details.


    TMG may still perform reverse DNS queries for policy enforcement purposes, however, results of such queries are not reliable or conclusive enough to be placed in the log.


    Dotan Elharrar

  13. Anonymous says:

    Hi! do you have any news about licensing for malware and url filtering? (cost per month, per user or per server …)

  14. Anonymous says:

    This is very good news was well informed that the followers of the issue I am. Thank you …

  15. Anonymous says:

    Thanks a ton for all the hard work

  16. UberTechie says:

    Hi, Can someone please explain why there are TMG categories missing from a standrad installation. So basically, in the MRS error and reporting site it has more categories such as "History" for example but in TMG not!


  17. paul says:

    Why are sites like comedycentral.com, fox.com, abc.go.com which offer streaming videos of TV shows and movies in the Art / Culture / Heritage category??

  18. sepa says:

    when a site is set to over-ride into the unknown category, it still reports the current MRS categorization?

  19. sing says:

    could we get an option to delay categories x amount of seconds 0-120 before display?

  20. boo says:

    why are online reservations at http://www.thecal.com blocked? why is the thecal.com site set as gambling?

  21. jay says:

    why can't we create our own custom URL categories?

  22. oscar says:

    how do we troubleshoot foxnews.com, new category is not blocked, yet the site looks horrible

  23. mike says:

    how do we enable safe search in youtube videos and images?

  24. Team building games for children says:

    It is a really nice instruction. You have a cool blog too.

  25. edward says:

    Hello All! Please help me. How can export only my url category overrides? Please asnwer to my email (eduard.marchenko@miroplast.com)

  26. alex says:

    Well, I guess now we go some bad news: after tremendous success of URL filtering Microsoft now decided to kill TMG and it will be discontinued as per December 31, 2015. So we can only enjoy it for less than 3 years and then switch to another solution. I hate going back to 3-rd party again… This is so bad news…

  27. Saalim says:

    how to block psiphon proxy through TMG 2010

Skip to main content