Big Data Applications in the Economics/Financial World Part II: Econometric Modeling in the 21st Century

"One of the first things taught in introductory statistics books is that correlation is not causation. It is also one of the first things forgotten." – Thomas Sowell

Executive Summary

Data analysis and econometric modeling are major tools that help decision makers design effective polices. This ever evolving world is driven by continuous developments in econometric tools and data sources. One such development in recent years is the use of big data—large and complex datasets that present both opportunities and challenges for analysts and their statistical toolkits. To inform our readers about the potential benefits and limitations of big data applications, we started a two-part series, and the first report looked at the potential benefits from big data for analysts and decision makers.1

This report focuses on issues and solutions related to econometric modeling and forecasting using big data. It is worth mentioning that we recognize that there are other issues, such as privacy and cyber security concerns, that represent huge hurdles for the advancement of utilizing big data on a mass scale. We are not lawyers, information technology or privacy experts, however, and as such, we will solely focus on using big data for modeling and analysis from an economics perspective.

Data as Insight

Typically, analysts utilize a data series to represent or proxy a sector's (or the overall economy's) activities, and by analyzing that series an analyst gains insight about the sector's state. For example, the Bureau of Economic Analysis releases personal spending data every month, and analysts use that series to learn about consumers' behavior and its impact on the overall economy, Figures 1 & 2. This data lags by about one month, however, and is only available in the aggregate. More granular data that include other useful information, such as demographic data, are often unavailable in real-time. Big data, with its hundreds of millions of observations, high frequency and detailed richness, can help to fill this information void. These very traits, however, can create both known and unknown challenges for analysts using traditional statistical techniques and applications. The relative newness of big data analysis likely means that as this phenomenon matures over time, problems with current methods will arise and new statistical tools and techniques will be needed to address these currently unknown problems lurking in the vast sea of big data. For now, however, we cannot know these unknown unknowns, and we must rely on the tools at hand. This report outlines some known potential problems and suggests a few key statistical principles, methods and best practices to bear in mind when conducting big data analysis.

Download The Full Special Reports