Stopping Google from Indexing Your Site

July 13, 2007

Topics: Google.

All right, this may not be precisely your main goal online. Nonetheless, there are many web sites out there which give the distinct impression that this was they’re specific purpose in creation.

Yet there are, for what it’s worth, very valid reasons to block pages some times. The trick is to make sure you’re only blocking the right documents.

One of the yet-to-come interesting features for stopping Google is the unavailable_after meta tag, announced by Google’s Director of Crawl Systems, Dan Crow at a Search Marketing New England event this week. This is one of the most potentially useful document meta options, although the value may not be immediately apparent.

The point of the unavailable_after meta element is to inform Google that a page should not be indexed after a certain date. This could be used in situations such as:

  • Job Postings with expirations
  • Sale announcements
  • Special offer deals
  • Expired auction listings

Basically, this would be great for any document which expires. From a user perspective, it’s incredibly dissatisfying to arrive at an expired sales page as the result of a search. From a business perspective, at best you’re providing no value; at worst you’re angering the customer. If you remove the page altogether, it may take months before the search engine catches up with you — leaving you with a hefty share of 404 responses. If you could inform the search engine right from the start that your page would cease to be valuable as of a specific date, you could avoid this whole problem.

For when and how the tag will be implemented, of course, we’ll just have to wait and see.

Have something to contribute?




« Read my Comment Policy

Start the conversation!