Robots.txt can be a file that exists on the root directory of every Site and can be used to instruct search engines like google and yahoo on which directories/information of the website they might crawl and include of their index. Community, countrywide and Global Web optimization companies can be found. https://desawisatabunder.com