View Single Post
  #7 (permalink)  
Old 03-27-2008, 09:13 AM
coder4hire coder4hire is offline
Junior Member
 
Join Date: Mar 2008
Posts: 20
Default

ryanhellyer, the easiest solution would be to slow down the speed with which the bots can index your site. For instance, to force them to pause a minute between every fetch, in your robots.txt file, put:

User-agent: *
Crawl-delay: 60

To actually limit the number of times that a particular bot can index your site -- perhaps a certain number of times per month -- would require detecting the IP address (assuming that it does not change over time) and dynamically throttling by keeping track of how many times that bot has already visited your site. That would be much more involved, and would require storing information in a database.

I hope this gives you some ideas.
Reply With Quote