The prediction community owes a great deal to Phil Tetlock, who has been involved in some of the largest and longest evaluations of expert forecasts to date. Tetlock is perhaps most widely known for his two-decade long study of political forecasters, which found that “foxes” (who know a little about a lot of different topics) typically outperform “hedgehogs” (who know a lot about one specific domain) in near-term forecasting. Over the last three years, Tetlock, Barbara Mellers, and Don Moore have led the Good Judgment Project, a large-scale forecasting tournament.
The Good Judgment Project began in mid-2011 as a forecasting tournament between five teams, sponsored by the US Government. (Read early coverage of the project from The Economist here.) Each of these teams had its own methods for leveraging the knowledge of its members to generate accurate forecasts about political and economic developments around the world. For example, the Good Judgment Team now assigns its forecasters to smaller teams of about a dozen members. This allows for collaboration in sharing information, discussing questions, and keeping each member motivated. Example questions include “What will the highest price of one ounce of gold be between January 1, 2014 and May 1, 2014?” or “Who will be the King of Saudi Arabia on March 15, 2014?” Predictions are scored both individually and as a team using Brier scores.
Season 3 of the tournament began this summer, and for the first time forecasters now have access to information from ICEWS, provided directly by the ICEWS project. ICEWS covers five events of interest (insurgency, rebellion, ethnic or religious violence, domestic political crisis, and international crisis) around the world on a monthly basis, and makes forecasts six months into the future. Two current Good Judgment questions related to ICEWS are:
- Will Chad experience an onset of insurgency between October 2013 and March 2014?
- Will Mozambique experience an onset of insurgency between October 2013 and March 2014?