(2).
You may not want certain pages of your site crawled because they
might not be useful to users if found in a search engine's search
results. If you do want to prevent search engines from crawling your
pages, Google Webmaster Tools has a friendly robots.txt generator to
help you create this file. Note that if your site uses subdomains and
you wish to have certain pages not crawled on a particular
subdomain, you'll have to create a separate robots.txt file for that
subdomain. For more information on robots.txt, we suggest this
Webmaster Help Center guide on using robots.txt files.