EVEN WHEN WRONG, POLITICAL EXPERTS SAY THEY WERE ALMOST RIGHT
COLUMBUS, Ohio -- How do political experts react when their predictions -- about election results or the fate of countries or other important issues -- turn out to be completely wrong?
They dont worry about it.
A new study found that most experts shrug off their errors, claiming that they were almost right and that their understanding of the situation was basically sound.
The study involved asking media, government and academic experts to predict political situations -- such as the fate of the Soviet Union in 1988 -- and then re-interviewing the experts after the situation was resolved.
The results revealed that experts were often overconfident in their initial predictions, sometimes massively so, said Philip Tetlock, author of the study and the Harold Burtt Professor of Psychology and Political Science at Ohio State University. Experts who said they were 80 percent confident in their predictions ended up being right only about half the time. And even when they were wrong, experts minimized their errors.
The results suggest that political leaders and others may have a hard time learning from history -- or at least learning lessons that dont fit their existing beliefs and ideologies, Tetlock said. When experts were wrong, they interpreted events to fit their preconceived notions, rather than change their notions to fit reality. Some of the popular defenses used by incorrect experts included I was almost right (The predicted events didnt happen, but almost did) and I was just off on timing (The predicted events have not occurred yet, but they eventually will).
The findings show that even thoughtful observers are often unwilling to change their minds very much, even when they are shown to be wrong, Tetlock said.
These results, published in the April 1999 issue of the American Journal of Political Science, are part of a long-running study in which Tetlock has been collecting experts predictions on a wide range of political, economic and military events. All of the experts had received some graduate training in social science or history, specialized in the region under examination, and earned their livelihoods as advanced graduate students and professors at universities, policy analysts at think tanks, intelligence analysts for the government, or journalists.
In one part of this study, Tetlock asked experts years ago to predict outcomes on seven different issues. In 1988, for example, he asked 38 Soviet experts whether the Communist Party would still be in power in 1993; and he asked 34 American political experts in 1992 whether President Bush would be re-elected later that year.
After the events occurred, Tetlock then re-contacted the experts to ask them about their predictions. In all seven scenarios, only slightly more than half of the experts correctly predicted the events that occurred. Still, even those who were wrong had been quite confident in their predictions. Experts who said they were 80 percent or more confident in their predictions were correct only 45 percent of the time.
Not surprisingly, experts who were correct credited their accuracy to their sound reading of the basic forces at play in the situation, Tetlock said.
More surprisingly, experts who were wrong were almost as likely as those who were right to believe their reading of the political situation was basically sound, he said.
When asked to rate how confident they were in their analysis of the issues involved, experts who were correct gave average ratings of 6.6 to 7.3 on a nine-point scale. However, even experts who were wrong continued to give average confidence ratings of 6.3 to 7.1. It is striking that forecasters who had been incorrect managed to retain nearly as much confidence in the fundamental soundness of their judgments as the forecasters who had been correct, Tetlock said.
One of the most popular defenses of those experts who were wrong was that their prediction almost occurred. These experts in essence said that if a few minor details of history had been different, their predicted outcome would have occurred. For example, consider the Soviet experts who in 1988 predicted the Communist Party would be firmly in control in 1993. (The Communist Party lost control of the Soviet Union in 1991 and the country disintegrated). Many of the experts argued that Communist hard-liners almost succeeded in a August 1991 coup attempt that would have solidified Communist power, and that these hard-liners would have succeeded if it were not for their own ineptitude.
Not all the experts who were wrong were loathe to admit that fact. About 20 to 25 percent of the experts, depending on the issue, showed a substantial willingness to admit they were wrong when their prediction was not correct, Tetlock said.
The point of the study, Tetlock said, is not to show that experts are often wrong in their predictions. All of the issues involved in the study were complex and would be difficult to predict. In addition, some of the experts who are wrong now may later prove to be right as history unfolds. However, he said the results do show how difficult it is for people to let go of their beliefs, even in the face of contrary evidence.
I think weve found that learning from history is a slow, painfully slow process, Tetlock said.