Question: WIB Answers: What is a robots text (robots.txt) file?
A reader from Connecticut asks: "An SEO company told me I should add a robot file to my website. I don't know what that is. Do I need one or is this a scam?"
Answer: All websites should have a robots.txt file. They are generally simple to create on your own. However, if you have parts of your site you need to be protected (inaccessible to robots) and you are not familiar with directory structures, you should let your SEO company create the file and upload it for you.
A robots.txt file is a plain text file that instructs search engine robots and web crawlers on how to crawl your website. It tells them where you want them to go and where you do not want them to go. Key points:
- Must be created in a plain text editor (do not create them in Word or HTML.)
- Should always have a .txt file extension and be named plural, all lowercase like this: "robots.txt."
- Are always uploaded to the main root directory of your website.
- Contains simple commands that allow or disallow (forbid) robots access to your entire website, or to selected directories, and/or files.
- Command most reputable robots and crawlers for major search engines.
- Are not fool-proof and some sophisticated malicious robots may ignore robots.txt file commands.
- Are an important part of your sites' overall search engine optimization strategy.
- All websites should have a robots.txt file.
