Limiting bot rate

[ryanhellyer]

I was recently asked by a colleague, how to go about limiting the amount of traffic a site receives by robots.

It is easy enough to totally prevent them from visiting, a simple noindex meta tag will fix that quite easily. But is it possible to simply reduce the number of visits from bots?

Sitemaps give you the option to specify how often you would like bots to visit, but AFAIK they simply use this as a relative measure. So if you say you want them to visit your home page every hour and your contact form once per year, they may visit your home page more often than your contact page accordingly. But my understanding is that they don't obey your commands per se, just that they use it as a guide to how important your various pages are. Or perhaps I have misunderstood?

Any ideas/suggestions?

In case you wondering why they would want to limit the number of bot visits, it is because the cost of bandwidth within our organisation is quite expensive. The cost per year from robots is about $120 whereas the cost due to humans is only $80. The site is rarely updated, so it seems pointless for the bots to be visiting the site every minute which they seem to be doing at the moment.

[BigAlReturns]

By SiteMaps do you mean Webmaster Tools? I think there is an option in tools to slow down overall bot rate, although I have no idea how effective it actually is.
Beyond this, I don't know of any other methods to be honest.

[ryanhellyer]

By Sitemap, I mean the XML file which search engine spiders can find via your robots.txt file ... Sitemaps - Wikipedia, the free encyclopedia

[BigAlReturns]

Ah OK, I just wondered because Google Webmaster Tools used to be called Google Sitemaps, before they extended its functionality somewhat.
I don't have a tools account to check with, but I'm pretty certain if you log into it then I think there is an option to slow crawl rate.

[ryanhellyer]

Thanks BigAl

[tobbilla78]

Thanks a lot BigA.It really helped me.
--------------

[coder4hire]

ryanhellyer, the easiest solution would be to slow down the speed with which the bots can index your site. For instance, to force them to pause a minute between every fetch, in your robots.txt file, put:

User-agent: *
Crawl-delay: 60

To actually limit the number of times that a particular bot can index your site -- perhaps a certain number of times per month -- would require detecting the IP address (assuming that it does not change over time) and dynamically throttling by keeping track of how many times that bot has already visited your site. That would be much more involved, and would require storing information in a database.

I hope this gives you some ideas.

[ryanhellyer]

That sounds like a great idea coder4hire!

I'll send a link to this topic to my colleague to see.

[chetan1]

Its really helpful.One sitemap is google webmaster tool.Its different from websites sitemap?

[ryanhellyer]

I don't know much about Googles webmaster tool. But the sitemap for a site is just an XML file which you can use to notify search engines when your site is updated and what the most important parts of your sites are (allows SE's to index your site better).