Should You Trust Google Data?

I'm a pretty big fan of Google. They have their shortcomings, like any other big company, but overall I've had a pretty favorable impression of them. After a recent finding, however, I must say that my level of trust in the organization has decreased significantly.

I've been working with a client to optimize their large pay-per-click campaign. They spend a few thousand dollars each day on PPC programs. I was conducting some analysis as a part of a PPC consulting project and found something that was quite disturbing in the Google data.

I was using the external Google keyword tool to generate search volume trends by historical month for a particular keyword phrase. I ran the query for a general phrase and asked for synonyms. The results included the specific keyword phrase I was targeting. I then entered the specific keyword phrase and asked again for the same information - search volume by trend. This data shows you the relative search volume by month for your keyword.

I exported the data to an excel spreadsheet for analysis and noticed something very peculiar. The data for the exact same keyword was very different in the two searches I conducted. In other words, the data that should have been exactly the same, was not. I've removed the specific search term for confidentiality reasons, but here is the actual resulting monthly data for both searches:

Notice that the trend data for each month is entirely different for the same exact search phrase. Even the data for the most traffic volume in a given month doesn't correlate. In the first line, it says December has the most search volume. In the 2nd search, it says October has the most search volume. That is not data you can trust.

I contacted Google. The customer service rep from the Google Adwords Team was able to duplicate the error and pass along to their engineering team. Here's a snippet of the service reps message from their finding:

I wanted to update you that our specialist team found the same behavior you described. Currently, our engineers are investigating the situation and are working to resolve it as soon as possible.

Now, I will give Google a ton of credit in that they admitted their tool is not working correctly. The issue I have, though, is the tool is still operational and does not include any message notifying users that the data may not be valid. That's a little scary in my opinion.

Here's the deal. If the data is completely wrong for some search trend data using their tool, what else could be wrong in the data Google provides? Should you believe the click-thru rates they provide? Should you believe what they say as far as click fraud is concerned? Should you believe what they tell you about impressions, maximum CPC, conversions and a myriad of critical data that you rely solely on them to provide?

I'm not saying anyone at Google is doing anything intentional to mislead customers. It is disconcerting, though, when they continue to knowingly provide misleading data to users of their service.

What was Ronald Reagan's axiom? Trust, but verify. Sounds like something we need to do with Google.

Rob Reed
Terrakon - A sales consulting and internet marketing firm.