Appendix C: Browser Compatibility 		index | previous | next

Keep Them Crawlers Offa Me!

Search engines aren't inherently good or evil. On the plus side, they're a (mostly) free way to promote your site. On the minus side, the crawlers that rove the Web looking for new pages can seriously sap system resources if they try to take home the entire site at once.

And then, of course, are the spam crawlers, which sniff the Web for e-mail addresses to sell.

All good search crawlers (also called spiders, sniffers, rovers, etc.) respect the rules of a document called robots.txt. This document is stored in the root level of a Web site and describes what can and cannot be indexed.

Here's where I get to the point. If you have experimental pages or semi-public documents that you don't want to be indexed by a search engine, you can add a META tag to those pages to keep them from being indexed.

For instance: If you have a beta version of a new DHTML toy up, you might not want AltaVista to tell everyone where it is. Or if your soccer team's Web site has a list of contact phone numbers that the team has a URL for but that isn't open to the public, you can also use this method.

In the <head> tag of the top-secret page, place the following line of code:

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

That indicates that sniffers should neither index the page nor follow any links on it.

Creating a Beta Site

(continued from previous page)

If you're posting a corporate site, it's important that you check everything out before the competition does. You don't want to get written up in the Wired gossip column as the world's worst webmaster. (That's your worst-case nightmare, isn't it?)

Creating an entire beta site, or staging server, is the best idea for large Web sites. As the Web designer or webmaster, you may or may not also be responsible for administration of the Web server. If you have a separate site admin team, discuss creating a staging server, and they'll probably know what you mean.

You should also have a beta review team in place that looks at stuff before it goes up. Ideally, this would consist of a copy editor, an editor, a Web guru, and (sigh) a marketing person.

To create a larger beta site:

  1. Set very strict permissions on the FTP server to the main site so that Joe Schmoe in another department can't accidentally post his family vacation photos on top of the home page.
  2. A staging server can be a directory on the main server or a separate machine. Either way, you should alias the site so that its URL is similar to the main URL. If your live URL is www.dorks.com, make the beta URL beta.dorks.com, so that you can eyeball directory stuff without having to add and subtract slashes in your head.

  3. Set up the staging server's directory structure so that everything's in the same place. (There are plenty of ways to automate this; one is to copy over everything on the existing site to the staging server.)

  4. When a new page is ready to preview, post it on the staging server and notify the beta review team.

  5. When everyone gives the thumbs-up, post it on the live server.