Definition: Spiders are programs (or automated scripts) that "crawl" through the web looking for data. They go through sites listing URLs and can pull data from web pages like email addresses. This is one tool spammers use to find email addresses on the Internet to create spam lists. It is also a tool used by search engines to find out more information about your website.
Webmasters can view logs to see what spiders and other robots have visited their sites. This information helps webmasters know who is indexing their site, and how often.
This information is useful because it allows webmasters to fine tune their SEO and update robot.txt files to prohibit certain robots from crawling their site in the future.
Also Known As: Robots, webcrawlers, webcrawling
