Free of charge Resources of available Webmaster Tools and Accessible Online Tests on the Internet
BOTSPOT. A comprehensive list of robots. Go to The Best of the Bots. An in-depth look at some of the smartest intelligent agents available for download. Some are made available for a limited-time trial evaluation basis; others are offered at no charge all.
Electronic Software Publishing Corporation (Elsop): Robots, Spiders, Crawlers and Wanderers - Highway Robbers on your Web Site. The following resources help you identify robots and tell you what they do. You'll also find information about how to exclude them from your website.
Karnak. Karnak helps you find and accumulate a personal library of knowledge. It compiles information from multiple internet sources and providing you a summarized, condensed, and highlighted report. Karnak regularly updates you on the status of your research by email. Guest access is completely free!
The Search Engine Watch: SpiderSpotting Chart. An index of search engine spiders including their host and agent name.
Protect Your Webserver From Spam Harvesters. Browser IDs of programs that spammers use to harvest email addresses from your Web pages along with advice how to block these spammers from Apache Web servers.
Submit Corner. This website compiled guides on indexing techniques (Site Improvement Guide), what search engines look for (META Tags Guide), and how to optimize your site to get the highest results in each search engine (Search Engine Guide). Go to Using Robots, and Robots Generator. This web tool will help webmasters create a set of robots used to control which files and directories to allow search engines to index.
The Web Robots Pages. An Index of Web Robots (or Web Wanderers, Crawlers, Spiders), programs that traverse the Web automatically.