I do not edit the robot txt file myself as I use Thesis to noindex pages and nofollow.. It is all in the settings and makes it very easy. I do not index tags and sometimes categories (as Rancid said also) because there are just too many tags and there is nothing there anyway. I like to index all pages and posts. Sometimes I nofollow links to about me pages or privacy policies as they are not content I want showing in the search engines.