Sunday, November 16, 2008

NoIndex NoFollow

I was just using Google to search Stack Overflow and came up with the following search results:




If you click on the second, third, or fourth items in the search results and then search the page for the term jscript you will find that it's not there. That's because when Google indexes the "hottest" part of Stack Overflow the jscript question was on the page but this is a volatile page and the links from this page change frequently.

Personally I find it annoying when I arrive at an dynamic index page on a site (via Google) only to find that the search term has subsequently been bumped off the list.

Should we do anything about this? And if so, what can we do about this?

First off, I think that you should not have Google index this page. If you have this keyword on your site then let Google focus its results on the page that has this keyword and that talks about this keyword. Spreading a keyword widely across a site is going to dilute its effectiveness and degrade user experience when they visit the site looking for that information.

On the Google Toolbar bar in IE there's the option to enable the happy and unhappy faces to vote on a page when you land there. I don't believe that Google uses this information however it does give me satisfaction when I land on an indexed page which doesn't have the search keyword on it to be able to click the unhappy face.

The obvious exception to this is if you have a stable, non-volatile index which will always have the keyword on it. Although this will not be as effective as landing the user on the content page because it will still require the user to find the link on the landing page and click through.

The way that I accomplish this in C# on my ASP.NET web sites is to set the robots meta tag as such:
 <META NAME="ROBOTS" CONTENT="NOINDEX">

I do this by using a common base class for all my pages. Here's a subset of the code from the base class:
public class CommonPageBase : System.Web.UI.Page
{
    public void SetNoIndexNoFollow()
    {
        SetRobots("NOINDEX,NOFOLLOW");
    }
    public void SetNoIndex()
    {
        SetRobots("NOINDEX");
    }
    public void SetNoFollow()
    {
        SetRobots("NOFOLLOW");
    }
    private void SetRobots(string content)
    {
        HtmlMeta robotsMeta = new HtmlMeta();
        robotsMeta.Name = "ROBOTS";
        robotsMeta.Content = content;
        Page.Header.Controls.Add(robotsMeta);
    }
}

Then in the code behind page for the .aspx page the Page_Load function will look something like this:
public partial class MyPage : CommonPageBase
{
    protected void Page_Load(object sender, EventArgs e)
    {
        SetNoIndex();
    }
}

You may of course want to make the call to the SetNoIndex() function conditional on an isVolatileIndex type variable. 

1 comment:

  1. Awesome, just what I was looking for. I needed to set a few forms to NOINDEX, NOFOLLOW and this did the trick. I just copy and pasted it. :)
    Thanks a bunch,
    Tomas

    ReplyDelete