Minutes ago I finished reading a new book, "Refactoring HTML", and even though the book mainly focuses on how to create really clean Web pages, and fix pages that you or someone else wrote, it does have some info on search engine bots that I've never seen anywhere else. I learned a lot about how to control bots, monitor how well they are obeying the rules of your robots.txt file, and even how to create a honeypot for catching misbehaving bots, and then exclude them from slowing down your site in the future. There probably are some articles out there on how to do all of this, but I think the author tried to bring it all together "under one roof".
Anyway, if anyone is interested in learning how to catch and control those bandwidth-munching bots that do you no good, that's one resource.
|