Gilbert F. White was a giant in the field of natural hazards, and a former colleague in Boulder at the University of Colorado, where he was an early director (beginning in 1970) of the Institute of Behavioral Science. Decades before that he had written his dissertation about how humans dealt with floods and his work led to the establishment in the early 1950s of a Federal framework that graded the probability of floods. Now it is easy to ascertain the 100-year flood plain for any locale in the United States, since by law this is required of city and state planners. The city to which he moved, and in which we were colleagues, has it’s own connection to the subject of his research, as Boulder experienced a massive flood that devastated the city about a century ago.

A Century Flood?

The 1894 Boulder Flood

The Boulder flood plain for 100 and 500-year floods developed in part as a result of White’s activism in planning for floods. Gilbert White’s office was just outside of the flood plain, up on a hill, overlooking it–near where I am temporarily sitting at this instance. But his last house in Boulder was not. And, anyone who followed the news this fall of the floods in Boulder–which were considered by many to be of the 100-year variety, may not know that Gilbert White’s advice probably saved many lives, as he argued for structures to be built that could interact with floods in a way to diminish risk (i.e., breakaway bridges, et cetera). Gil was famous for many things, including the quote “Floods are `acts of God’ but flood losses are largely acts of man,” which was taken from his dissertation. In the 1980s he convinced the Boulder City Council that Boulder had previously experienced a flood even larger than the huge flood of 1894. As a result building in the flood plain was restricted (a bit) and knockout bridges were built. I remember reading an article when I arrived in Boulder in the early 1980s about Gilbert’s warnings about a 100-year flood, which pictured Gilbert then in his 70s standing in the rushing Boulder Creek. You can listen to Gilbert discussing this issue as well as see a version of the Boulder floodplain.

Read More

Reviewer 1
This turkey is a bit over done. I think the problem is that the authors need a better theory of turkey before they try to stick one in an oven for four hours and then serve it. A recent example is recently published in the Journal of Poultry and many earlier contributions in Giblets and Drumsticks have been overlooked. Many earlier scholars have actually caught their own turkeys and fed them assumptions and corn to produce a really substantial turkey, that not only reflects the theory of turkey, but also glistens with the implications of a well thought out turkey. Until a better theory of turkey is employed to motivate this particular baked turkey, it is hard to reach a satisfactory conclusion with this effort. While I appreciate the efforts, I don’t support revising this particular turkey for resubmission, though I am tempted to suggest that a soup be created with the remains.
Reviewer 2
Have the authors never tasted chicken? Neither duck? Medieval scholars knew that a combination of these fowl with turkey was necessary to provide a substantial empirical test of the “Thanksgiving Hypothesis.” Curiously, the authors have ignored this long standing research tradition, even though there is a Stata recipe that will undertake this effortlessly for them. Surely this could easily be done in revisions.
Reviewer 3
I appreciate the authors efforts to examine the “Thanksgiving Hypothesis,” but it would appear there is a serious flaw in their analysis. The turkey has been cooked, and we see the standard inclusions: sweet potatoes, mashed potatoes, gravy, freshly cooked rolls, and even cranberry sauce. I even appreciate the introduction of oysters as an instrument into the stuffing to rule out the endogeneity that the turkey was actually fed ground fishmeal. But there is no adequate control–such as a tofurkey–introduced to examine the possiblity that a general triptophane coma is responsible for outcomes in the “Thanksgiving Hypothesis.” That and the absence of soup leads me to conclude that this project is not ready. But I am encouraged enough to recommend revisions.

Editor: The reviewers see much merit in your work, but point to serious missteps as well. I have personally tasted a Turkey dinner, and would like to suggest that after considering the comments above, you revise your procedures and resubmit the results. If you choose to do so, I will send the effort to a new round of reviewers, including one of the original critics. If you decide to accept this invitation, I will need to have your submission by November 27th, 2014.

The prediction community owes a great deal to Phil Tetlock, who has been involved in some of the largest and longest evaluations of expert forecasts to date. Tetlock is perhaps most widely known for his two-decade long study of political forecasters, which found that “foxes” (who know a little about a lot of different topics) typically outperform “hedgehogs” (who know a lot about one specific domain) in near-term forecasting. Over the last three years, Tetlock, Barbara Mellers, and Don Moore have led the Good Judgment Project, a large-scale forecasting tournament.

The Good Judgment Project began in mid-2011 as a forecasting tournament between five teams, sponsored by the US Government. (Read early coverage of the project from The Economist here.) Each of these teams had its own methods for leveraging the knowledge of its members to generate accurate forecasts about political and economic developments around the world. For example, the Good Judgment Team now assigns its forecasters to smaller teams of about a dozen members. This allows for collaboration in sharing information, discussing questions, and keeping each member motivated. Example questions include “What will the highest price of one ounce of gold be between January 1, 2014 and May 1, 2014?” or “Who will be the King of Saudi Arabia on March 15, 2014?” Predictions are scored both individually and as a team using Brier scores.

Season 3 of the tournament began this summer, and for the first time forecasters now have access to information from ICEWS, provided directly by the ICEWS project. ICEWS covers five events of interest (insurgency, rebellion, ethnic or religious violence, domestic political crisis, and international crisis) around the world on a monthly basis, and makes forecasts six months into the future. Two current Good Judgment questions related to ICEWS are:

  • Will Chad experience an onset of insurgency between October 2013 and March 2014?
  • Will Mozambique experience an onset of insurgency between October 2013 and March 2014?

Read More

ICEWS is an early warning system designed to help US policy analysts predict a variety of international crises. This project was created at the Defense Advanced Research Projects Agency in 2007, but has since been funded (through 2013) by the Office of Naval Research. ICEWS has not been widely written about, in part because of its operational nature, and in part because articles about prediction in politics face special hurdles in the publication process. An academic article (gated) described the early phase of the project in 2010, including assessments of its accuracy, and a WIRED article in 2011 criticized ICEWS for missing the Arab Spring–at a time when the project was only focused on Asia.

In an article (here for now) forthcoming in the International Studies Review, as one of the original teams on the ICEWS project, we highlight the basic framework used in the more recent, worldwide version of ICEWS. Specifically, we discuss our model that is focused on forecasting, which is our main contribution to the larger, overall project. We call this CRISP. We argue that forecasting not only increases the dialogue between academia and the policy community, but that it also provides a gold standard for evaluating the empirical content of models. Thus, this gold standard improves not only the dialogue, but actually augments the science itself. In an earlier article in Foreign Policy, with Nils Metternich, we compared Billy Beane and Lewis Frye Richardson (sort of).


Read More

Political conflicts are rarely between two parties.  In Iraq, for example, there were as many as 19 different groups engaged, including the Islamic Army in Iraq, Al-Qaeda in Iraq, the Jihadist Leagues, and the Just Punishment Brigades. In Syria, we see a similar picture, including the Free Syrian Army, the Syrian Liberation Front, the Syrian Islamic Front, and Jabhat al-Nusra. Many attempts to understand these kinds of situations group all the rebel forces together against a government.  But neither are the rebels unified, and monolithic. Nor, necessarily, is the government. We explore a theory of the interactions among these various kinds of factions in order to better understand what kinds of actions are most likely to be undertaken. To do so, we combine elements of strategic calculation and the analysis of networks. The basic insight is the old saw, often attributed to the 6th century (BCE) Chinese general, Sun Tzu: hold your friends close, and your enemies closer.

Thailand data

The top rug shows the different parties that are in power in Thailand during the observation period, with markers for changes in power. The bottom plot shows conflictual events in Thailand from 1998 on.

Read More

GDELT ( is a global database of events which have been coded from vast quantities of publicly available text that is produced by the world’s new media. It has created a great deal of excitement in the social science community, especially within the field of international relations. But it has had wider visibility as well: in August 2013, there were 150,000 views of a map of protest activity around the world, based on the GDELT database.  Event data have been around for several decades, but the GDELT project has generated new interest.

ICEWS is an early warning system designed to help US policy analysts predict a variety of international crises to which the US might have to respond. These include international and domestic crises, ethnic and religious violence, as well as rebellion and insurgency. This project was created at the Defense Advanced Research Projects  Agency, but has since been funded (through 2013) by the Office of Naval Research. ICEWS also produces  a  rich corpus of text which is analyzed with powerful techniques  of automated event-data production.  Since GDELT and ICEWS are based on similar, though not identical methods and sources, it is interesting to compare them.

ICEWS data

ICEWS event data, gray line for stories and black line for events, 2001-2013

One area in which they are most conceptually different is that ICEWS follows a more traditional approach to event data in seeking to encode a chronology of events that reflects in some sense  the putative ground truth of what occurred. The figure on the right shows the corpus of stories in ICEWS (gray) and the resulting events (black): total events are fairly stable over time event though the number of media stories increases. GDELT is more concerned with getting a comprehensive catalogue of all media stories (and other text) on reported events, and the corpus of those media stories is increasing exponentially, as the figure below shows. As a result, the number of events in GDELT is also increasing over time, much more so than ICEWS.

Read More

Welcome to Predictive Heuristics, the Ward Lab‘s blog. The Ward Lab consists of a group of researchers at Duke University, led by Michael Ward. Its current instantiation at Duke University since 2009 is considerably enhanced by a terrific group of colleagues and students (and lab alumni). All of our work aims in some sense at prediction, either for theory building or decision making. The lab has expanded in size and scope over time, and we now have a diverse set of members working on a broad array of projects:

  1. Modeling of protests, insurgencies, rebellions, ethnic and religious violence, as well as international and domestic crises in 167 countries and predicting same for six months–yes, in the future (DOD funded);
  2. The prediction of IEDs (explosions, locations, discoveries) in Afghanistan and other countries (ONR funded);
  3. The study of how network dynamics affect the evolution of conflict and cooperation (ONR);
  4. How ensembles can be used to combine predictions (ONR);
  5. The use of textual analysis protocols to encode large scale events, such as militarized interstate disputes (NSF);
  6. The use of machine learning techniques to characterize the regime characteristics of contemporary polities (NSF);
  7. Use of advanced techniques for predicting missing data in “big data”;
  8. Prediction of Coups and other rapid regime changes;
  9. Examination of the latent networks among contemporary political parties, especially how they respond to global crises;
  10. Creation of techniques for simultaneously studying the onset, duration, and cessation of events;
  11. and many others found at

These efforts are united by two common threads: the use of advanced (social science) methods, and a focus on predictions to aid decision-making and policy. Social science data and modern computational techniques have both exploded to the point that it is now possible to use them in principled ways to gain leverage on predictive tasks. Indeed, prediction is the gold standard for understanding, not something that has to stand in contradistinction to it. Predictive heuristics is a way of combining these new data and new techniques with the goal of helping decision making and improving what many social scientists call “theory”– a term that gives certain lab directors hives.

Many lab members have their own blogs, and others (Ward) have been thoughtfully placed in the control group: the group without blogs.  Apparently, Ward has been reassigned to the treatment group.  This will not be a blog about squirrels gnawing though his cable lines, but will focus on the substance and methods employed by lab members to do social science research.  The goal of this blog is to provide an outlet for updates on lab-wide projects, such as W-ICEWS, as well as projects by individual lab members. There is no formal structure, although we aim for posts with a bi-weekly, not annual nor hourly, frequency

For those specifically interested in our global forecasts of conflict, check out our website.

This is an experiment for us, and we’ll be supplying it for the foreseeable future.  There is apparently no price equilibration in the blogsphere but we hope that you will find it worth the price.

Finally, we invite anyone interested in contributing a guest post on a topic related to predictive heuristics to contact us at