The predictive power of Big Data is being explored — and shows promise — in fields like
public health, economic development and economic forecasting. Researchers have found
a spike in Google search requests for terms like “flu symptoms” and “flu treatments” a
couple of weeks before there is an increase in flu patients coming to hospital emergency
rooms in a region (and emergency room reports usually lag behind visits by two weeks or
so).
Global Pulse, a new initiative by the United Nations, wants to leverage Big Data for global
development. The group will conduct so-called sentiment analysis of messages in social
networks and text messages — using natural-language deciphering software — to help
predict job losses, spending reductions or disease outbreaks in a given region. The goal is
to use digital early-warning signals to guide assistance programs in advance to, for
example, prevent a region from slipping back into poverty.
In economic forecasting, research has shown that trends in increasing or decreasing
volumes of housing-related search queries in Google are a more accurate predictor of
house sales in the next quarter than the forecasts of real estate economists. The Federal
Reserve, among others, has taken notice. In July, the National Bureau of Economic
Research is holding a workshop on “Opportunities in Big Data” and its implications for
the economics profession.
Big Data is already transforming the study of how social networks function. In the 1960s,
Stanley Milgram of Harvard used packages as his research medium in a famous
experiment in social connections. He sent packages to volunteers in the Midwest,
instructing them to get the packages to strangers in Boston, but not directly; participants
could mail a package only to someone they knew. The average number of times a package
changed hands was remarkably few, about six. It was a classic demonstration of the
“small-world phenomenon,” captured in the popular phrase “six degrees of separation.”
Today, social-network research involves mining huge digital data sets of collective
behavior online. Among the findings: people whom you know but don’t communicate
with often — “weak ties,” in sociology — are the best sources of tips about job openings.
They travel in slightly different social worlds than close friends, so they see opportunities
you and your best friends do not.
Researchers can see patterns of influence and peaks in communication on a subject — by
following trending hashtags on Twitter, for example. The online fishbowl is a window
into the real-time behavior of huge numbers of people. “I look for hot spots in the data,
an outbreak of activity that I need to understand,” says Jon Kleinberg, a professor at
Cornell. “It’s something you can only do with Big Data.”