Today I learned about Fetch as Google. Haven’t used it yet, though.
Fetch as Google
Reply
Tag Archives: googlebot
Chillin’ with the bots
It’s just me, the Googlebot and the Baidu spider chillin’ at ProgClub tonight.
I wish more people were involved with ProgClub. We’re ready for so much more. Patience isn’t really a virtue of mine. :P
Help, I’m being raped by Googlebot!
Friggin’ Googlebot is crawling all over pcrepo, even though I told it NOT TO DO THAT. My robots.txt file has been updated to disallow svnro for most of the day. Google really should have a facility for web-masters to be able to force a reload of robots.txt in Googlebot. They’re going to hit me thousands of times before this is over. :(