Adjusting the MOSS ROBOTS meta tag for 3rd party search engines

When developing a public facing website using the publishing features of MOSS it might be required to emit a ROBOTS meta tag that prevents internet search engines from indexing specific pages.

The standard RobotsMetaTag control included in WSS generates the following tag:

<META NAME=”ROBOTS” CONTENT=”NOHTMLINDEX”>

Unfortunatelly most search engines do not understand the NOHTMLINDEX content value but a value of NOINDEX. To resolve this it is required to replace the RobotsMetaTag control in the master page with a custom control that emits the correct value.

The following code will do this:

using System;
using System.Collections.Generic;
using System.Text;
using System.Web;
using System.IO;
using Microsoft.SharePoint;
using System.Web.UI;

namespace StefanG.MossControls
{
    public class RobotsMetaTag : Microsoft.SharePoint.WebControls.RobotsMetaTag
    {
        protected override void Render(HtmlTextWriter writer)
        {
            SPWeb web = GetContextWeb(HttpContext.Current);
            if (web.ASPXPageIndexed == false)
            {
                writer.Write(“<META NAME=\”ROBOTS\” CONTENT=\”NOHTMLINDEX,NOINDEX\”/>”);
            }
        }
    }
}

Here are the required steps to implement this:

  1. compile the above code into a separate DLL (e.g. RobotsMetaTag.dll)
  2. if signed add the DLL to the GAC – otherwise copy it into the bin directory of your web application
  3. add the control to the save controls list in the web.config

          <SafeControl Assembly=”RobotsMetaTag” Namespace=”StefanG.MossControls” TypeName=”*” Safe=”True” />

    (if the DLL is in the GAC ensure to add the strong assembly name instead with version and public key token).

  4. Open the master page in SharePoint designer and register your DLL:

          <%@ Register TagPrefix=”StefanG” Namespace=”StefanG.MossControls” Assembly=”RobotsMetaTag” %>

    (if the DLL is in the GAC ensure to add the strong assembly name instead with version and public key token).

  5. replace the following tag:

          <SharePoint:RobotsMetaTag runat=”server”/>

    with this:

          <StefanG:RobotsMetaTag runat=”server”/>

 

10 Comments


  1. Is there a way to avoid Sharepoint Designer warning us that we are missing the original       <SharePoint:RobotsMetaTag runat="server"/> tag?

    Reply

  2. HotFix Security Hotfix MS07-059 Artikel zum Lesen Why SharePoint Server is Terrible Dazu der Kommentar

    Reply

  3. Hi Stefan,

    Why do you not use a control adapter instead of custom control? Is it for performance improvment?

    Gael

    Reply

  4. Hi Gael,

    nice idea! You are right: this would be the better solution.

    I will post it in a minute as a new blog entry.

    Cheers,

    Stefan

    Reply

  5. When developing a public facing website using the publishing features of MOSS it might be required to

    Reply

  6. I don’t understand.

    If I put this in the Master Page then it means that the WHOLE site isn’t indexed, no?

    I need just a few pages to be excluded.

    Reply

  7. Hi Tal,

    then change the logic in the control and look for a page specific property rather than the site property to decide which robot tags to render.

    Cheers,

    Stefan

    Reply

  8. Thanks stefan.

    how do I check a page property?

    Reply

  9. Hi Tal,

    you would need to open the list item for the current URL and then read the property. Similar to the code above.

    Cheers,

    Stefan

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.