Google's John Mueller said he would advice that if "that's something where if you see that these pages are important enough that people are linking to them then I would try to avoid blocking them by robot's txt." In short, if you have important or popular pages with a lot of links to them, make sure Google can access the page....