Superforecasting Book Summary
The Art and Science of Prediction
Book by Philip Tetlock
Summary
In Superforecasting, Philip Tetlock and Dan Gardner reveal the techniques used by elite forecasters to predict future events with remarkable accuracy, and show how these skills can be cultivated by anyone to make better decisions in an uncertain world.
The Skeptic And The Optimist
Philip Tetlock considers himself an "optimistic skeptic" when it comes to forecasting. The skeptical side recognizes the huge challenges of predicting the future in a complex, nonlinear world. Even small unpredictable events, like the self-immolation of a Tunisian fruit vendor, can have cascading consequences no one foresaw, like the Arab Spring uprisings.
However, the optimistic side believes foresight is possible, to some degree, in some circumstances. We make mundane forecasts constantly in everyday life. Sophisticated forecasts underpin things like insurance and inventory management. The key is to figure out what makes forecasts more or less accurate, by gathering many forecasts, measuring accuracy, and rigorously analyzing results. This is rarely done today - but it can be.
Section: 1, Chapter: 1
"We Are All Forecasters"
"We are all forecasters. When we think about changing jobs, getting married, buying a home, making an investment, launching a product, or retiring, we decide based on how we expect the future will unfold. These expectations are forecasts."
Section: 1, Chapter: 1
Even Smart, Accomplished People Make Simple Forecasting Errors
In 1956, the respected physician Archie Cochrane was diagnosed with terminal cancer. An eminent specialist said Cochrane's axilla was "full of cancerous tissue" and he likely didn't have long to live. Cochrane immediately accepted this and started planning for death.
However, a pathologist later found no cancer in the tissue that was removed. The specialist was completely wrong. Being intelligent and accomplished was no protection against overconfidence.
Even more striking, Cochrane himself made this mistake, despite being a pioneer of evidence-based medicine. He railed against the "God complex" of physicians who relied on intuition rather than rigorous testing. Yet he blindly accepted the specialist's judgment.
Section: 1, Chapter: 2
WYSIATI Explains Why We Jump To Conclusions
WYSIATI (What You See Is All There Is) is a key mental trap that leads to flawed predictions. It refers to our mind's tendency to draw firm conclusions from whatever limited information is available, rather than recognizing the information we don't have.
For example, after the 2011 Norway terrorist attacks, many people immediately assumed Islamist terrorists were responsible, based on recent events like 9/11 and the bits of evidence available, like the scale of the attacks. However, the perpetrator turned out to be a right-wing anti-Muslim extremist, Anders Breivik.
WYSIATI explains why we jump to conclusions rather than saying "I don't know" or "I need more information." Our minds abhor uncertainty. We impose coherent narratives on events, even when key facts are missing. Breaking this habit is crucial to forecasting better.
Section: 1, Chapter: 2
Beliefs Are Hypotheses To Be Tested, Not Treasures To Be Guarded
Superforecasters treat their beliefs as tentative hypotheses to be tested, rather than sacred possessions to be guarded. This is encapsulated in the idea of "actively open-minded thinking."
Some key tenets of actively open-minded thinking:
- Be willing to change your mind when presented with new evidence
- Actively seek out information that challenges your views
- Embrace uncertainty and complexity; don't be afraid to say "maybe"
- View problems from multiple perspectives; don't get wedded to one narrative
- Resist the urge to simplify and impose falsely tidy stories on reality
- Expect your beliefs to shift over time as you learn and discover your mistakes
By holding beliefs lightly, and being eager to stress-test and refine them, we can gradually move closer to the truth. Superforecasters show that this approach produces vastly better predictions compared to stubborn, overconfident ideologues.
Section: 1, Chapter: 2
The Value Of Precise Forecasts
Vague language like "a serious possibility" or "a non-negligible chance" makes it impossible to assess whether a forecast was accurate or not. In contrast, precise probabilities, like "a 62% chance", allow predictions to be unambiguously judged. Precision is necessary for forecasts to be properly tested, tracked and improved. Some key principles:
- Replace vague language with numerical odds as much as possible
- Use finely grained percentage scales (30%, 31%, 32%) rather than coarse buckets (certain, likely, toss-up, etc.)
- Specify clear time horizons and definitions for all forecast questions
- Track predictions and grade them against what actually happened
- Calculate forecasters' accuracy using quantitative measures like Brier scores
Precision takes more mental effort. But embracing it is necessary to separate lucky guesses from true skill - and to refine that skill with practice and feedback.
Section: 1, Chapter: 3
This concept is also discussed in:
The Signal and the Noise
Why It's Hard To Assess Forecast Accuracy
Steve Ballmer's infamous 2007 forecast that "There's no chance that the iPhone is going to get any significant market share" looks hugely wrong in hindsight. But Ballmer never specified what "significant" market share meant, or what time period he was referring to. His forecast was too vague to definitely judge as right or wrong.
This is extremely common - and makes it effectively impossible to assess forecast accuracy. To be testable, forecasts need:
- Specific definitions. What counts as a "default" or a "bubble" or a "coup"?
- Precise time horizons. By what date will the event happen or not?
- Numerical probabilities that can be scored. "60% chance" can be graded later as right or wrong; "pretty likely" cannot.
- Repeated forecasts over time. One forecast is not enough - we need a track record.
Most real-world forecasts fail these criteria. As a result, we have little idea how accurate experts actually are, despite how much influence their predictions have.
Section: 1, Chapter: 3
Foxy Forecasters Beat Hedgehog Historians
In his famous essay "The Hedgehog and the Fox," Isaiah Berlin argued that thinkers can be classified into two categories: Hedgehogs, who view the world through the lens of a single defining idea, and Foxes, who draw on a wide variety of experiences and perspectives.
Forecasters who were Hedgehogs - with one big theoretical view of how the world works - tended to perform quite poorly. They were overconfident and reluctant to change their minds. Foxy forecasters were much more accurate. Rather than trying to cram complex reality into a single framework, they were comfortable with cognitive dissonance and pragmatically adapted their views based on new information. Some key Fox behaviors:
- Pursuing breadth rather than depth, gathering information from diverse sources
- Aggregating many micro-theories rather than trying to build one grand theory
- Frequently using qualifying words like "however" and "on the other hand"
- Readily admitting mistakes and changing their minds
- Expressing degrees of uncertainty, rather than certainty
The Hedgehog/Fox distinction points to a crucial insight: In a complex, rapidly changing world, cognitive flexibility is more valuable than theoretical elegance. The nimble fox prevails over the stubborn hedgehog.
Section: 1, Chapter: 3
Superforecasters Beat The Wisdom Of The Crowd By 60%
The Good Judgment Project (GJP), led by Philip Tetlock and Barbara Mellers, recruited thousands of volunteer forecasters to predict global events as part of a tournament sponsored by the research agency IARPA. Questions covered politics, economics, national security and other topics relevant to intelligence analysts.
The GJP used multiple methods to boost forecast accuracy, including training, teaming, and statistical aggregation. But its most striking finding was that a small group of forecasters, the "superforecasters", consistently outperformed others by huge margins.
Across the first 2 years of the tournament, superforecasters beat the "wisdom of the crowd" (the average forecast of all participants) by 60% - a stunning margin. They even outperformed professional intelligence analysts with access to classified data. This suggests that generating excellent prediction accuracy doesn't require subject matter expertise or insider information - just the right cognitive skills and habits.
Section: 1, Chapter: 4
Related Content
The Signal and the Noise Book Summary
Nate Silver
In The Signal and the Noise, Nate Silver explores the art and science of prediction, explaining what separates good forecasters from bad ones and how we can all improve our understanding of an uncertain world.
In The Signal and the Noise, Nate Silver explores the art and science of prediction, explaining what separates good forecasters from bad ones and how we can all improve our understanding of an uncertain world.
Prediction
Decision Making
Economics
Range Book Summary
David Epstein
"Range" challenges the conventional wisdom that early specialization is the key to success. Instead, Epstein argues that in our increasingly complex and unpredictable world, it is those with broad experience and diverse skills who are best equipped to thrive.
"Range" challenges the conventional wisdom that early specialization is the key to success. Instead, Epstein argues that in our increasingly complex and unpredictable world, it is those with broad experience and diverse skills who are best equipped to thrive.
Personal Development
Learning
Psychology