Definition
Robots.txt is a text file at the root of a website that tells search engine crawlers which paths they should not crawl.
Robots.txt explained
Blocking a page in robots.txt stops crawling but does not guarantee it stays out of search results. To keep a page out of the index, use a noindex tag and allow crawling so the tag can be seen.