- To stop content from being crawled and indexed
- To tell search engines about what pages and files you think are important on your site
- To help search engines find and understand media content and news articles
- To control what pages appear in search results and the order in which they appear