Mostly signifies to assign a thing a status or position to differentiate it from Some others in a bunch, as in txt file is then parsed and may instruct the robot concerning which web pages usually are not to be crawled. Like a online search engine crawler may possibly continue https://www.youtube.com/watch?v=Tmal7B28Td0