Some stuff about Google’s Crawler

  • SubscribeToday!

12 users commented

MajesticSEO also just posted some of its own reasons why you might have crawler issues there.

said admin on March 5th, 2012 at 9:07 am

I’ll be a little cheeky here.

If things like Google only hitting robots.txt approx 1:24Hr, or +1 buttons overring robots.txt disallow directives are a surprise/new to you,
then I strongly suggest you go and get yourself in the Google Webmaster Central Forums, and reading Googles Official Webmaster Blog.

Spending a few Hours each week reading through that lot (the blog has low activity, and the forums tend to repeat a lot – so it’s not too much time/efforT),
will definitely help keep you ontop of such things.

said Autocrat (Lyndon NA) on March 5th, 2012 at 4:22 pm

Very interesting, especially the part about a +1 being a stronger signal than Robots.txt…

said Keith on March 5th, 2012 at 4:53 pm

You are welcome to be cheeky, Lyndon – but I put that into the last line of the post. :)

said admin on March 5th, 2012 at 7:54 pm

Thank you for this summary – I had overlooked Pierre’s appearance at ThinkVisibility, glad I stumbled upon your post!

said JS on March 5th, 2012 at 8:12 pm

Thanks for the summary, couldn’t make it to ThinkVisibility but this is a great summary of Pierre’s presentation, its always good to hear from real googlers. Especially since my site has had issues with 500 errors in the past.

said carla on March 6th, 2012 at 12:13 pm

Great write up, but your “simple maths” in Pierre’s answer is a little off. 1 URL/sec is approximately 84,000 (not 8,400) or exactly 86,400 per day (if Googlebot is hitting your site 24 hours/day).

Otherwise, great info and thanks to John Mueller for informing us of this post.

said Brett on March 7th, 2012 at 10:26 pm

Thanks. I suspect Pierre was not responsible for the 8400 maths error as much as my note taking skills.

said admin on March 7th, 2012 at 11:46 pm

google is playing with us…. that’s all I know…

said alt-med10 on March 15th, 2012 at 2:00 pm

Like the anchor text you were trying to get away with? :)

said admin on March 15th, 2012 at 8:03 pm
said Trackbacks on December 5th, 2016 at 6:38 pm

Leave a reply

Categories Archives Who’s saying what

Recent Posts Recent CommentsBadges
View additional information about through Majestic SEO Site Explorer.