Adjusting the MOSS ROBOTS meta tag for 3rd party search engines


When developing a public facing website using the publishing features of MOSS it might be required to emit a ROBOTS meta tag that prevents internet search engines from indexing specific pages.


The standard RobotsMetaTag control included in WSS generates the following tag:


<META NAME=”ROBOTS” CONTENT=”NOHTMLINDEX”>


Unfortunatelly most search engines do not understand the NOHTMLINDEX content value but a value of NOINDEX. To resolve this it is required to replace the RobotsMetaTag control in the master page with a custom control that emits the correct value.


The following code will do this:


using System;
using System.Collections.Generic;
using System.Text;
using System.Web;
using System.IO;
using Microsoft.SharePoint;
using System.Web.UI;

namespace StefanG.MossControls
{
    public class RobotsMetaTag : Microsoft.SharePoint.WebControls.RobotsMetaTag
    {
        protected override void Render(HtmlTextWriter writer)
        {
            SPWeb web = GetContextWeb(HttpContext.Current);
            if (web.ASPXPageIndexed == false)
            {
                writer.Write(“<META NAME=\”ROBOTS\” CONTENT=\”NOHTMLINDEX,NOINDEX\”/>”);
            }
        }
    }
}

Here are the required steps to implement this:



  1. compile the above code into a separate DLL (e.g. RobotsMetaTag.dll)

  2. if signed add the DLL to the GAC – otherwise copy it into the bin directory of your web application

  3. add the control to the save controls list in the web.config

          <SafeControl Assembly=”RobotsMetaTag” Namespace=”StefanG.MossControls” TypeName=”*” Safe=”True” />

    (if the DLL is in the GAC ensure to add the strong assembly name instead with version and public key token).

  4. Open the master page in SharePoint designer and register your DLL:

          <%@ Register TagPrefix=”StefanG” Namespace=”StefanG.MossControls” Assembly=”RobotsMetaTag” %>

    (if the DLL is in the GAC ensure to add the strong assembly name instead with version and public key token).

  5. replace the following tag:

          <SharePoint:RobotsMetaTag runat=”server”/>

    with this:

          <StefanG:RobotsMetaTag runat=”server”/>

 

Comments (10)

  1. Nicolas Roberge says:

    Is there a way to avoid Sharepoint Designer warning us that we are missing the original       <SharePoint:RobotsMetaTag runat="server"/> tag?

  2. Anonymous says:

    HotFix Security Hotfix MS07-059 Artikel zum Lesen Why SharePoint Server is Terrible Dazu der Kommentar

  3. Gael Duhamel says:

    Hi Stefan,

    Why do you not use a control adapter instead of custom control? Is it for performance improvment?

    Gael

  4. Stefan Goßner says:

    Hi Gael,

    nice idea! You are right: this would be the better solution.

    I will post it in a minute as a new blog entry.

    Cheers,

    Stefan

  5. Anonymous says:

    When developing a public facing website using the publishing features of MOSS it might be required to

  6. Tal Winter says:

    I don’t understand.

    If I put this in the Master Page then it means that the WHOLE site isn’t indexed, no?

    I need just a few pages to be excluded.

  7. Stefan Goßner says:

    Hi Tal,

    then change the logic in the control and look for a page specific property rather than the site property to decide which robot tags to render.

    Cheers,

    Stefan

  8. Tal Winter says:

    Thanks stefan.

    how do I check a page property?

  9. Stefan Goßner says:

    Hi Tal,

    you would need to open the list item for the current URL and then read the property. Similar to the code above.

    Cheers,

    Stefan