 |
/ Home
/ Web Directory / Computers / Internet / Searching / Search Engines / Robots (8)
Web Sites in Category Robots
All About Search Indexing Robots and Spiders - Search Tools Consulting explains how the search engine programs called "robots" or "spiders" work, and reviews related sites.
HTTP User Agent Index - An alphabetical list of user agents and the deployer behind them, compiled by Christoph Rüegg.
List of Robot Agent Strings - A list from PGTS of Web robots with the identifying data they leave in Web site logs.
Psychedelix List of User-Agents - Andreas Staeding's large list of search engine spiders, similar Web robots, and Web browsers: their web-log identification and links to their originators.
Robots, Spiders and Other User Agents: a Resource for WebMasters - José Luis Pellicer's searchable database of robots, spiders and other user agents for programs that surf the web.
Robots.txt Syntax Checker - Online tool for syntax verification to robots.txt files, provided by Simon Wilkinson.
Search Engine IP Addresses - Lists IP addresses of search engine spiders. Can be searched by IP address. Also links to resources on spiders.
Search Engine Robots and Other User Agents - John A. Fotheringham presents data in tabular form on the robots sent by search engines and other sites to read and index Web pages: their origins, names and IP addresses.
Related Categories:
Computers > Internet > Abuse > Spam > Preventing
Computers > Internet > Web Design and Development > Authoring > Online Tools > robots.txt
Computers > Software > Internet > Site Management > Log Analysis
|
|
|
|
