Definition: Robots is a catch-all, or generic term for programs and automated scripts that “crawl” through the web (the Internet) and collect data from websites and anything else on the Internet that they can find.
- Spiders: Are the same thing as robots, and the terms can be used interchangeably.
- Worms: Are robots, but are distinguished because worms are replicating programs (unlike other robots).
- Web crawlers: Are also the same as robots. Note: WebCrawler is a specific robot.
- WebAnts: Cooperating robots that share and distribute information. Ants that find information share their data so other ants do not have to explore the same file.
Who Uses Robots?
- Hackers: To find ways in attack or take over your website. A great way to have your website hacked is to use out-of-date software and web applications that may have security breaches. Hackers often use worm robots.
- Spammers: Mass marketers and spammers look for email addresses on websites, forums, blogs, social networks, etc. using robots.
- Search Engines: To crawl your website to collect data and index it. Search engines look at meta data and content as well as other information about webpages using robot technology.
- Webmasters: To study competing sites and their own site. Webmasters also study logs of who is crawling their site and how often to fine tune SEO and robot.tx files (a file that allows, restricts, or forbids robots from crawling your site). Every website should have a robot.txt file!
