4. Solving
One of the most powerful aspects of the big data revolution is the unification
of large datasets with advanced analytics for problem solving. With advances in
algorithms and machine learning, data-driven insights can be extracted from
information sources that are too expansive or complex to tackle using traditional
desktop analysis. This ability to spot patterns and solve problems beyond human
mental capabilities has led to two main sources of insight derived from big data.
First, very large and/or multidimensional datasets can be examined to look
for previously hidden patterns and correlations. Sometimes this can validate
positions that were previously supported by common sense, practical experience
or received wisdom. On other occasions, this sort of analysis can deliver entirely
new insights into the underlying dynamics of a population, market or business.
The Livehoods project, for example, looks for patterns in location-aware social
network activity to identify neighbourhoods in urban areas based on overlapping
social patterns rather than traditional geography.10
Second, big data opens up the realm of reliable predictive analytics. By
examining the relationships embedded in large datasets it is possible to build a
new generation of models describing how things are likely to evolve in future.
This approach can be combined with scenario planning to develop a series of
predictions for how a system will respond to different policy choices. The state
of the art in predictive analytics can deliver forecasts for some domains with a
very high degree of precision, providing an auditable, scientific basis for making
decisions in complex systems.
Using big data analytics for problem solving has other advantages beyond
seeing deep and far. Increased use of computational techniques can free up
an organisation’s staff to focus on tasks where human beings continue to
outperform computers, increasing overall productivity. And working rigorously to
quantitatively link insights to supporting evidence can provide a check against any
bias inadvertently introduced by the individuals involved the process. “Evidencebased
policymaking” is an oft-repeated Whitehall mantra; big data analytics can
help embed this in the culture where previous efforts have failed.