Matt Cutts just put up a post reminding us about the Google webmaster console. In particular, he highlights the robots.txt analysis tool and gives an example showing how it can find errors.

Just for fun, I used this tool on the firsthit website. Here’s a screen grab (click the thumbnail and a full size image will open in a new window).

What this tells me is that my robots.txt file is fine. It does two things. First, it lets every search engine spider find whatever it wants on my site. No limitations. Secondly, it tells the search engines where my sitemap file is. Because I use the sitemap generator plugin (details in this post) it automaticallypings Google whenever there’s a change, so the last line in my robots.txt file is really only there for Yahoo!, MSN Live and ASK.