Dollars are U.S. unless stated otherwise.



Time for the U.N. to find its superforecasters

By Peter Hulm

The new Secretary-General can do something immediately to improve the U.N.’s dusty reputation: assemble an expert panel on the risks of political, economic and social disaster around the world and the possibility of progress in its development efforts. But first the panel will need to prove its credibility.

Many of you know this story, I’m sure. The U.S. intelligence services who investigated George W. Bush’s justification for invading Iraq backed him with a public declaration in October 2002: "We judge that Iraq has continued its weapons of mass destruction in defiance of UN resolutions and restrictions."

Next year the U.S. and its allies invaded and discovered they were wrong. It’s been described as arguably the worst intelligence failure in modern history. But that’s not the story.

Political analyst Robert Jervis — who investigated why the US failed to predict the 1979 revolution in Iran — conducted a postmortem on the WMD story for the CIA. He found the assessment reasonable though completely wrong. After all, why was Saddam Hussein playing hide-and-seek with the U.N. weapons inspectors if there was nothing to hide?

The benefits of being less certain

But Jervis also concluded: "There were not only errors but correctable ones. Analysis could and should have been better." The intelligence services might have reached the same faulty conclusion (the U.N. inspectors didn’t), but in any case their assessments would have been "less certain". This might have made a big difference in Congress.

The story is told by political scientist Philip Tetlock in Superforecasting. He writes: "If some in Congress had set the bar at ‘beyond a reasonable doubt’ for supporting the invasion, then a 60% or 70% estimate that Saddam was churning out weapons of mass destruction would not have satisfied them."

The point for me is that the intelligence services "never seriously explored the idea that it could be wrong": no ‘red teams’ to attack prevailing views, no papers suggesting competing possibilities. "Most strikingly," declared Jervis, "no one proposed a view close to the one we now believe to be true."

Dart-throwing chimpanzees

Philip Tetlock is himself famous for demonstrating that many so-called political experts can do no better than a "dart-throwing chimpanzee" at hitting the bullseye with their predictions. His 2015 book recounts assembling a group of lay people who did (reportedly) 30% better in one year than professional intelligence analysts at predicting future events, though all these superforecasters had to go on was the information from the Internet.

What has this to do with the U.N.? The U.N. has made a number of blunders over the years — from allowing the Rwanda massacres to take place, to screwing up in Haiti, missing the boat on Ebola, standing by helpless in Southern Sudan, letting Europe as a community turn its back on refugees, covering up rape and violence by peacekeeping troops in West Africa and failing to control North Korea’s belligerence and oppression, to mention only the most recent.

Second opinions

No-one can claim these disasters and mistakes were not foreseen. But whistleblowers have been pursued if not punished.

Looking further back, structural adjustment, as promoted by the World Bank and IMF, was clearly from the beginning a horrendous non-Keynesian approach to economic development. But who was around to give non-partisan analysis of the options?

With a superforecasting team on board, perhaps Kofi Annan could have given more weight to the U.N. peacekeepers who were predicting massacres in Rwanda unless the U.N. stepped up its presence. Would President Bill Clinton have approved NATO cluster-bomb attacks on civilians in Serbia if someone with more clout than U.N. Human Rights officials had publicly pointed out the violation of ordinary humanitarian standards?

Once the International Red Cross had that moral weight, but it all vanished when the Khmer Rouge took over in Cambodia, refusing to talk to the ICRC. Claude Lanzmann in the 1990s interviewed a Red Cross delegate who visited Theresienstadt during the Nazi extermination of Jews and found the camp inmates strangely listless but conditions acceptable.

Superforecasters can be trained

Tetlock’s argument is that superforecasters are made not born. He sets out 10 simple rules for developing these skills. They are the same I have found in the people I consider superforecasters in the U.N. in Geneva. Among these principles:

Remember the Bay of Pigs

This last rule is perhaps the most important difference between experts and superforecasters. Tetlock reminds us that the military told President Kennedy there was a "fair chance" of the anti-Castro invasion of Cuba succeeding when they meant one chance in three. And the CIA escape plan never changed even when the invasion forces switched to the Bay of Pigs, "across a tangle of swamps and jungle" from the mountains that were supposed to be a refuge.

Tetlock now runs a new forecasting project at, where anyone can exercise their own powers of prediction on a series of questions.

Many of the questions of are U.S.-centred but others get into key international issues.

For example, 55% of 88 forecasters believe there will be a mass killing in Afghanistan before the end of the year. Some 95% of 106 forecasters believe Hungary’s October referendum will reject the European Union’s right to set quotas for migrant relocation there.

The problem with committees

The UN already has a number of ways to gather and air differing opinions but rarely does so, except at the political level. Committees are a perfect place to hear dissenters but their reports aim more for consensus or the least common denominator of agreement than quantifying the uncertainties.

When the Intergovernmental Panel on Climate Change published its first assessment 26 years ago, I criticised it for taking a minimalist view rather than recognizing that science is not about consensus, and allowing governments like the U.K. to defer action citing the unquantified "uncertainties".

I give myself a 60% success score on that one — just better than coin-tossing, and higher in the 1990s than later.

The problem with journalists

But my rating goes down to 20% with regard to the Paris Climate Change conference. I didn’t reckon with the determination of Laurent Fabius to close his career with an achievement, and wrote the prospects down just before the talks started.

I’m keeping the 20% until I am sure that the U.S., the U.K., India and China will live up to their promises. But I don’t expect to be among the superforecasters. Journalists are better at predicting the present than the future. We’re too excitable.

Altering the parameters of discussions

Tetlock’s argument for specifying the scale of your uncertainties (in the Iran section of his website) is simple: "Asking falsifiable questions and forecasting on them has the potential to moderate polarizing policy debates because accountability fundamentally alters the parameters of the discussion."

The same would be true for the U.N. I’d like to know whether North Korea might spark an international military incident this year. But if the U.N.’s superforecasters publicly declared they thought it 80% likely, surely the Security Council would have to take notice.

What’s the chance of halting the massacres in Southern Sudan? If the superforecasters were pessimistic, could the U.N. fail to put the issue higher on its agenda?

World Bank President Jim Yong Kim, with his earlier experiences as a medical anthropologist with an NGO, moved faster than the sclerotic WHO (within days) to give assistance to West African countries during the Ebola epidemic.

This innovative action from the former head of WHO’s HIV/aids unit proved controversial in the World Bank’s financial-oriented bureaucracy but won praise from outside. But we shouldn’t count on bulldozer bosses to achieve results in the UN.

The organization desperately needs some counterweight to the diplomatic immobilism of decisions based on political expediency.

Finding our superforecasters

Where to find the U.N.'s superforecasters? Tetlock’s website is open to organizations as well as individuals. The new Secretary-General could sign up the UN staff and run the same kind of tournament that U.S. intelligence asked Tetlock to arrange.

You might not find the superforecasters where the organization teaches you to expect them. Tetlock’s star among 2800 superforecasters was a retired IBM programmer in Santa Barbara with no international expertise. In Geneva it might be a Filipino mail clerk (used to handling lots of information quickly) or a cafeteria cleaner (used to discovering superefficient patterns of work).

I’d put my money on a security guard (unemotional, slow to jump to conclusions and careful to check out assumptions). In the days when I tramped the Palais corridors on a regular basis, I knew an Irish officer with all those qualities. He was the least perturbable man I ever met and able to deal with any tricky situation, including stupid instructions from his superiors.

Wikipedia has a full article on the Good Judgment Project.

Superforecasting: The Art and Science of Prediction was written with journalist Dan Gardner.

You can see the ideas applied in a Reuter story on oil prices here.

GJP has a Global Political Challenge for honing your skills here.

Superforecasting runs a blog here. It also has a Twitter feed.

For discussion of the U.N. and Rwanda, see this 1998 article.

The Haiti tragedy is summarized here. The scientific analysis by Renaud Piarroux and others was published in 2011.

The U.N. whisteblower who was suspended after he exposed the rape of children by peacekeepers has resigned in protest over the "complete impunity" given to the perpetrators and leaves at the end of August, AFP reported in June 2016.


Superforecasters reveal their sources for Trump's first 100 days

Back to top up to top Previous Page