Archive

Tag Archives: prediction

The right kind of variance

By
October 6, 2014
prediction
9 Comments

or, How I learned to stop worrying and love event data. 

Nobody in their right mind would think that the chances of civil war in Denmark and Mauritania are the same. One is a well-established democracy with a GDP of $38,000 per person and which ranks in the top 10 by Human Development Index (HDI), while the other is a fledgling republic in which the current President gained power through a military coup, with a GDP of $2,000 per person and near the bottom of the HDI rankings. A lot of existing models of civil war do a good job at separating such countries on the basis of structural factors like those in this example: regime type, wealth, ethnic diversity, military spending. Ditto for similar structural models of other expressions of political conflict, like coups and insurgencies. What they fail to do well is to predict the timing of civil wars, insurgencies, etc. in places like Mauritania that we know are at risk because of their structural characteristics. And this gets worse as you leave the conventional country-year paradigm and try to predict over shorter time periods.

The reason for this is obvious when you consider the underlying variance structure. First, to predict something that changes, say dissident-government conflict, the nature of relationships between political parties, or political conflict, you need predictors that change.

thailand

Predictions for regime change in Thailand from a model based on reports of government-dissident interactions (top). White noise, with intrinsically high variance, is not helpful (middle), but neither is GDP per capita (bottom).

Read More

Modeling and predicting IEDs

By
August 8, 2014
Lab projects
Leave a comment

Improvised explosive devices, or IEDs, were extensively used during the US wars in Iraq and Afghanistan, causing half of all US and coalition casualties despite increasingly sophisticated countermeasures. Although both of these wars have come to a close, it is unlikely that the threat of IEDs will disappear. If anything, their success implies that US and European forces are more likely to face them in similar future conflicts. As a result there is value in understanding the process by which they are employed, and being able to predict where and when they will be used. This is a goal we have been working on for some time now as part of a project funded by the Office of Naval Research, using SIGACT event data on IEDs and other forms of violence in Afghanistan.

expl-haz

Explosive hazards, which include IEDs, for our SIGACT data.

Read More

Prediction and Good Judgment: Can ICEWS Inform Forecasts?

By
November 12, 2013
Lab projects
5 Comments

The prediction community owes a great deal to Phil Tetlock, who has been involved in some of the largest and longest evaluations of expert forecasts to date. Tetlock is perhaps most widely known for his two-decade long study of political forecasters, which found that “foxes” (who know a little about a lot of different topics) typically outperform “hedgehogs” (who know a lot about one specific domain) in near-term forecasting. Over the last three years, Tetlock, Barbara Mellers, and Don Moore have led the Good Judgment Project, a large-scale forecasting tournament.

The Good Judgment Project began in mid-2011 as a forecasting tournament between five teams, sponsored by the US Government. (Read early coverage of the project from The Economist here.) Each of these teams had its own methods for leveraging the knowledge of its members to generate accurate forecasts about political and economic developments around the world. For example, the Good Judgment Team now assigns its forecasters to smaller teams of about a dozen members. This allows for collaboration in sharing information, discussing questions, and keeping each member motivated. Example questions include “What will the highest price of one ounce of gold be between January 1, 2014 and May 1, 2014?” or “Who will be the King of Saudi Arabia on March 15, 2014?” Predictions are scored both individually and as a team using Brier scores.

Season 3 of the tournament began this summer, and for the first time forecasters now have access to information from ICEWS, provided directly by the ICEWS project. ICEWS covers five events of interest (insurgency, rebellion, ethnic or religious violence, domestic political crisis, and international crisis) around the world on a monthly basis, and makes forecasts six months into the future. Two current Good Judgment questions related to ICEWS are:

  • Will Chad experience an onset of insurgency between October 2013 and March 2014?
  • Will Mozambique experience an onset of insurgency between October 2013 and March 2014?

Read More

Prediction versus Explanation?

By
October 31, 2013
Academic papers, prediction, Uncategorized
4 Comments

ICEWS is an early warning system designed to help US policy analysts predict a variety of international crises. This project was created at the Defense Advanced Research Projects Agency in 2007, but has since been funded (through 2013) by the Office of Naval Research. ICEWS has not been widely written about, in part because of its operational nature, and in part because articles about prediction in politics face special hurdles in the publication process. An academic article (gated) described the early phase of the project in 2010, including assessments of its accuracy, and a WIRED article in 2011 criticized ICEWS for missing the Arab Spring–at a time when the project was only focused on Asia.

In an article (here for now) forthcoming in the International Studies Review, as one of the original teams on the ICEWS project, we highlight the basic framework used in the more recent, worldwide version of ICEWS. Specifically, we discuss our model that is focused on forecasting, which is our main contribution to the larger, overall project. We call this CRISP. We argue that forecasting not only increases the dialogue between academia and the policy community, but that it also provides a gold standard for evaluating the empirical content of models. Thus, this gold standard improves not only the dialogue, but actually augments the science itself. In an earlier article in Foreign Policy, with Nils Metternich, we compared Billy Beane and Lewis Frye Richardson (sort of).

wardlab

Read More