There has been a major attack on big data over the last month with all sorts of sensational titles from leading publications. Some examples:
• Science Magazine: The Parable of Google Flu: Traps in Big Data Analysis
• Financial Times: Big data: are we making a big mistake?
• Harvard Business Review: Google Flu Trends’ Failure Shows Good Data > Big Data
• The New York Times: Eight (No, Nine!) Problems With Big Data
The trigger for this firestorm was the fact that Google Flu Trends proved to be not as good as people had originally projected. This should really be no surprise. It was an original and innovative solution with no real precedent. Those accustomed to innovative approaches understand that iteration is necessary and that you will most likely not get it 100% correct on your first try.
Finance-types might have some deeper insights into this topic as well, and (hopefully) were not as taken-in as many others on the apparent perfection of flu forecasts. It appears that Google Flu Trends relied heavily on back-testing. That is, they took historical results and the data they had on hand and basically reverse engineered a solution that would have worked extremely well. Algorithmic traders do this all of the time.
Back-tests are both required and seen as highly questionable within finance. They are required as they show that you did your homework and, at least superficially, show that your system appears to work quite well. They are questionable in that the general, but not specific, algorithm is revealed, it is unclear just how much optimization occurred to get to those sparkling results, and for the most part assumes that future circumstances will generally replicate those of the test period (which is in and of itself a problem as circumstances rarely repeat exactly).
It appears what happened was that non-financial types (in other words those with little knowledge of the weaknesses surrounding back-testing) saw the initial results associated with Google Flu Trends, were enamored by the sexiness of the new data (search), and thought that a brand new element was discovered. They then were ‘shocked’ upon discovering that forecasting the future is in fact difficult and that improvements are necessary in any innovative approach.
The most unfortunate thing is that the means of analysis (big data) has been viscously attacked due to the poor performance of Google Flu Trends. In the financial markets, a popular expression is “Throwing the baby out with the bath water”, meaning not discriminating between good and bad elements and just trying to be done with the whole of the situation. This saying appears to apply well here.
By far the best critique, in my opinion, comes from The Economist: Is big data bullshit?
This is actually a video interview, but the tone and insights are very good. They question Google Flu Trends, shed more light on big data in general, but do not throw the baby out with the bath water. This is the type of analysis that should be highlighted.
My favorite quote is:
“if you trust any one source of data you are probably going to get it wrong but if you ensemble it together (with other data) you are probably going to get closer to the truth.”
Unfortunately, Google relied too heavily on one source of data. Forecasting using one single variable is always going to be difficult. It really does not matter how great or innovative your data source is, or if it is ‘big data’ or ‘small data’ you are dealing with.
For me, this is extremely important in the development of big data analysis in general. An implied conclusion is that multiple datasets should be conferred in analysis. This is a point that I have been hammering on for years. Social media, search, sentiment, quantified news, etc. are all great sources of information for financial market analysis. However, taken in isolation, none of these data sources will be full-proof. Taken together, such datasets excel in providing improved insights and analysis.
The recent Google Flu Trends firestorm is really just a flash in the pan. It will not slow down the march towards the collection and analysis of increasingly larger datasets. And the main observations highlighted by critics really do not appear new in the sense that for many of the observations you could simply take out “big data” and replace it with “data”. In other words, many of the critiques apply equally as well to any data analysis. Some of the other critiques such the ability to “game” big data sources or the inherent noise in big data are really controllable using the right techniques.
So, Google Flu Trends did not perform as well as expected. Other than churning up some sensational news articles, the longer term impact appears frankly quite minor.