Urban Big Data: Where is the Signal in the Noise?

The era of Big Data—of increasingly vast streams of information generated by people and their devices—is raising thorny questions for city planners.  

To hear some boosters describe the benefits, you might suppose Big Data is about to unleash an unqualified urban renaissance. We've all heard the claims: Sensors will eliminate traffic jams and optimize energy demand. New mapping technologies will identify, and help to correct, urban problems no one could see before. Social media and crowdsourcing will target big and little problems, down to which potholes to fix—maybe even make governments work again. 

As these claims have inevitably failed to live up to their hype, a pronounced backlash has set in.  Big Data, and its cousin the Smart Cities movement, are both increasingly dismissed as fads and technology marketing schemes—and worse for some, examples of a too-reductionist science gone amok. The charges are worth considering.

But the vast streams of data are already with us. (For example, humanity has now generated and stored almost 8 sextillion bytes of data—many times the number of grains of sand on all the beaches of the world.)  The more pointed question is, therefore, how we will manage this already existing technology and what it might offer us as a resource, beyond the hype, to deal with our most pressing human challenges. 

Unquestionably there are dangers in Big Data, in threats to privacy and in disorganized, overwhelming data—too much "noise" and not enough "signal."  Alarmingly, there is also a runaway proliferation of trivial data and near-meaningless junk. 

Star Trek actor George Takei recently held a contest on Facebook, asking entrants to describe the hardest thing to explain about life today to a time traveler from the 1950s. The winner: "I possess a device in my pocket that is capable of accessing the entirety of information known to man. I use it to look at pictures of cats and get in arguments with strangers."

The anecdote is funny, but it belies a serious threat to our species intelligence—to our ability to identify and solve our larger collective problems. Yet we have made some notable progress in curating useful knowledge among all the vast junkish information, on the Web and elsewhere.  Wikipedia, for example, is a very intriguing example of useful knowledge acquired through big-data, open-source collaboration, offering an instructive contrast with the increasing trend toward junky data. The challenge is to build on this and other successes with a new generation of data tools that help us to identify and collaborate effectively, notably on urban challenges.

"It is so important be able to have data on real processes involving real people and communities, and the emergence of social organizations to solve problems," says Luis Bettencourt of the Santa Fe Institute. Bettencourt and his colleagues Jose Lobo and Joe Hand recently invited international NGO leaders, designers, researchers, and policy experts to a conference at the institute titled "Acting Locally, Understanding Globally: Scaling Up Community-Collected Data in Developing Cities."

The question conference-goers asked: How can Big Data be used in practical ways, as a tool to meet humanity's most pressing problems—particularly those posed by rapidly urbanizing cities? The issue is especially pressing because the current urbanization includes large areas of sprawl, carrying potentially catastrophic impacts for resource use, greenhouse gas emissions, and a range of other ills. Yet still, urbanization can bring significant benefits, including expanded opportunities for human development and even simultaneous reductions in resource use and emissions per person—if we can understand and exploit the remarkable capacities of cities.   

The Santa Fe Institute is an interdisciplinary gathering place for the study of complex adaptive systems, a broad term that includes hard-to-understand phenomena like biological systems, large-scale economies, and cities. Collaborators include physicists, mathematicians, economists, biologists, and others, who ask big questions about the structures of such systems and the challenges and opportunities they pose to humanity.

Bettencourt and his colleagues are asking some of the biggest questions now being asked of cities: What kinds of systems are they? What do they offer to help us to meet the looming challenges of the future—especially as we enter an unprecedented era of rapid urbanization?

"I think we have made some substantial progress in the theory of cities as integrated, dynamical networks," Bettencourt says. We build cities, he points out, because they are "social reactors"—they provide the dense structured spaces for our social evolution. "Cities are the niches we construct to magnify our prolific sociality."

Big Data, he says, can help to provide a clearer picture of how cities work, and could work better, or “how people can or cannot connect socially in cities, how this has costs in terms of transportation of people, goods and ideas, how such connections allow people to differentiate, learn, and create new economic value," he says. This is a theory that is more about process than form per se: "It is a theory of social connectivity, how it gets articulated in urban space, and what learning, development, and growth processes drive it."

Bettencourt points to Jane Jacobs' early and remarkable insights into these urban processes. But he thinks it's time to develop the theory more generally, beyond Jacobs' limited subject of New York: "A better urban theory should be able to say how we may measure these effects and how they may also occur in different urban forms, facilitated by different technologies, under possibly different social and economic systems – say, China instead of the US."

For many participants, the question of Big Data comes down to whether it can be used to empower residents as agents of their own change. The science by Bettencourt and others also points to the limits of top-down approaches by expert specialists, and the value of distributed approaches driven by agents, i.e., by the residents themselves. After all, that's how urban economies—urban systems of many kinds—actually function. 

At the end of the conference, participants drafted a short declaration affirming the point. "The process of synthesizing local knowledge also provides individuals an opportunity for self-organization, empowerment, and engagement with the decision making process," it reads.

More to the point, expert specialists ought not be in the business of determining what is best for residents: if self-organization is going to be meaningful, they themselves must be able to determine it. "Recognizing that a plurality of conceptions of well-being exists, we must ensure that the capacity for individuals and local governments to actively engage and improve their own well-being is built into the synthesis process."

In other words, Big Data has to be made useful at the level of individuals and groups—by those individuals and groups. That's the signal to be found in what is otherwise noise—creating the capacity for effective responses at the local level. "Mechanisms of change," asserts the declaration, "are ultimately local."

===============

Declaration

Acting Locally, Understanding Globally: Scaling Up Community-Collected Data in Developing Cities

Cities are the essential means by which human development happens.

The science of cities will provide an integrated understanding of the process of human social change in shared built spaces.  

Historically, scientific models of cities have not been able to systematically tap into local knowledge, preventing them from iteratively improving.

Local knowledge provides a true view of the immense diversity and heterogeneity in cites, how individuals behave, and of processes of change in peoples lives and communities.  Mechanisms of change are ultimately local. 

The process of synthesizing local knowledge also provides individuals an opportunity for self-organization, empowerment, and engagement with the decision making process. Recognizing that a plurality of conceptions of well-being exists, we must ensure that the capacity for individuals and local governments to actively engage and improve their own wellbeing is built into this synthesis process.

We cannot understand how cities function as a whole without an understanding of the individual, particular and local, and how that integrates back up to the global level.

Thus, urban development policy and practice must be based on observing and understanding how people actually live in communities. 

Michael Mehaffy is a planner, urban designer, and executive director of Portland, Oregon-based Sustasis Foundation, where he works with colleagues on new neighborhood-scale and "peer to peer" planning tools. He is indebted to Luis Bettencourt for assistance with this essay.