The robots.txt is a text file placed at the root of a website allowing you to give instructions to search engines crawlers indicating the pages that can be indexed. It is also used for SEO purposes.
To add a robots.txt file, access the Toolbar and click on Settings.
And then on ‘SEO’ on the menu bar on the left. The ‘Yes’ field is checked by default.
Note:
Changing the content of the robots.txt file can harm your SEO. Do not modify the content of robots.txt unless you are an experienced Webmaster.