You need a robots.txt file in your root directory to let search engine spiders know to index all your site files. You can also specify which bots, web directories or web pages to disallow search engine to spider (go to www.robotstxt.org for more information.) Now you can also tell the search engine spiders the location of your site map and what file types to not index within the robots.txt file.
Note: Do not have empty lines in-between text in the robots.txt file.
example codes:
Disallow: /*.swf$ (stops indexing of swf files)
Disallow: /folder/ (stops indexing of a particular folder)
Disallow: /folder/page.html (stops indexing of a particular webpage)
Sitemap: http://www.yoursite.com/sitemap.xml (mark location of your sitemap)
Analyze your robots.txt file This tool identifies many common errors in the robots.txt file.
Here is the file used on ogdenian.com: ogdenian.com/robots.txt