Not having a robots.txt file is just like leaving your front door wide open for any passers-by to walk in at will and look through your entire home – including your drawers and closets!
A robots.txt file can help restrict access to robots and keep them out of areas in your site that may be private, or that you do not want to be indexed.
You can think of a robots.txt file as a lock box in real estate. Only those you authorize can get inside.
Even a very basic robot file can help protect your site. There are other ways (i.e., labeling directories and files as private) but the fastest way is often to simply set up a robots.txt file and list specific directories and files you do not want robots to see. You can also ban certain robots from crawling your website.
