Common Q&A about TMG URL Filtering database

URL filtering is one of Forefront TMG’s most popular features. The feature makes use of a cloud service, also known as Microsoft Reputation Services (MRS) for URL categorization purposes. In this post we’d like to address some of the more frequent questions we’ve received regarding the URL filtering database and the cloud service.

What is the URL categorization cloud service?

MRS can be thought of as a web service, providing secure access to a huge, cloud-based dynamic repository of URLs and their respective categories. The database features over 70 categories ranging from security-oriented selections like Malicious sites, through productivity-oriented categories such as Online Communities, and ending with liability-oriented categories like Pornography. The database spans tens of millions of unique URLs and their respective categories. Whenever a user behind TMG tries to access a URL, TMG can look up its category by issuing an online query to the cloud service. If you’d like to learn more about URL filtering and the cloud service, check out this post.

What are the sources of the URL filtering database?

The database merges data from several providers. The data providers include internal Microsoft sources as well as 3rd party sources. It is important to note different providers employ different URL categorization techniques. Some employ manual classification while others rely heavily on web crawlers performing automated classification. The highly advanced automated classification techniques ensure TMG URL filtering is as competitive as it gets when it comes to the coverage of the web.

How frequently is the URL filtering database updated?

The database is constantly updated in order to cope with the dynamic nature of the web. New URLs are added, obsolete ones are removed and categories can change based on the page’s content. The update frequency varies per data provider and per URL category.

How good is the coverage for URLs outside the US?

The URL filtering database ensures a global coverage, and is meant to serve TMG customers all over the world. The coverage is regularly monitored against indicators such as Alexa's top Million URL list. In addition, telemetry data gathered from TMG deployments around the world is used to refine the database and improve the coverage for URLs that are popular with our customers.

Is URL categorization performed based on the top level domain only or based on the full URL?

The categorization is based on the full path. This means http://www.contoso.com and http://www.contoso.com/sports could be assigned with different categories in the database.

Can a URL have more than a single category?

Yes. You can learn more about TMG’s support for multiple URL categories in this post.

Can TMG block adult text, images and videos from search results by web search engines?

Yes. You can learn more about TMG’s safe search enforcement support in this post.

Author:

Dotan Elharrar, Senior Program Manager, Forefront TMG

Reviewers:

David Strausberg, Technical Writer, Forefront TMG

Zakie Mashiah, Group Manager, Forefront TMG

Gabriel Koren, Forefrong TMG test team