Google Webmaster Tools report a significant increase of 404 errors

This is the first post after a while, and we’re trying to have more than one post every two years. But let’s be honest, you’re not interested in this yadda yadda anyway and just came here because Google Webmaster Tools sent you a kind message like

Increase in not found errors

Then you went to GWT and saw an unexpected spike in 404 (or 410 where Google doesn’t make a difference) like this one:

Screen Shot 2013-07-04 at 11.23.59

The total amount of course varies from case to case as Google sends the warning if an unusual increase happens. However the numbers can be as high as hundreds of thousands not found pages.

And then you immediately thought “What’s wrong? Will this harm my rankings?”

You surely stumbled across the official post that confirmed that it won’t harm your rankings in most cases.

So you’re kind of safe now, but you want to know what caused the spike and how to get rid of the errors? Well I can’t tell you what your problem was, but I’ll show you mine and maybe, just maybe, it’s the same 😉

The reason for the spike was that Google crawled our Javascript and discovered something that looked like an URL but in fact was a cometd channel id. This can of course also happen to generated links in javascript with snippets (or the infamous /a generated by jquery). Most times you don’t want Google to crawl those links – but how can you avoid Google to crawl Javascript?

To put it simple: you shouldn’t.

Here’s a short explanation why: By all means returning a 404 in this case it the right thing to do, but a rising amount of 404 in your webmaster tools console and a weekly warning message just don’t feel right. The immediate thought usually is to block the URLs via robots.txt. But that’s the wrong thing to do for two reasons:

  1. You tell Google that there is a page existing (which isn’t) and it’s just not allowed to crawl
  2. You move the 404 errors to the “blocked URLs” section of your webmaster tools

So if you want to get rid of you 404 error in Google Webmaster Tools that were caused by Google crawling your javascript, perform a 301 redirect to the best matching page, or if there isn’t one to the homepage. This way you’ll get the errors out of GWT without moving the errors to the blocked section.

But why believe me? I could just fool around with you, couldn’t I? Well no, I spent some time researching on this topic and came across an official answer by a Google employee: see the initial thread on Google Groups and the follow up question on stackoverflow

Now have fun redirecting!

tl;dr Google reports 404 errors in GWT due to crawling your Javascript? 301 the URLs to the next category page (or homepage)