The Peculiar Blindness of Experts

The power of curious generalists. (See also https://en.m.wikipedia.org/wiki/The_Hedgehog_and_the_Fox.)

Kind of makes one wonder if there’s something behind Michael Gove’s statement that “we’ve had enough of experts.” Besides Russia, that is.

Or this could just be my happy confirmation bias.

«Tetlock decided to put expert political and economic predictions to the test. With the Cold War in full swing, he collected forecasts from 284 highly educated experts who averaged more than 12 years of experience in their specialties. To ensure that the predictions were concrete, experts had to give specific probabilities of future events. Tetlock had to collect enough predictions that he could separate lucky and unlucky streaks from true skill. The project lasted 20 years, and comprised 82,361 probability estimates about the future.

The result: The experts were, by and large, horrific forecasters. Their areas of specialty, years of experience, and (for some) access to classified information made no difference. They were bad at short-term forecasting and bad at long-term forecasting. They were bad at forecasting in every domain. When experts declared that future events were impossible or nearly impossible, 15 percent of them occurred nonetheless. When they declared events to be a sure thing, more than one-quarter of them failed to transpire. As the Danish proverb warns, “It is difficult to make predictions, especially about the future.”

….

Tetlock, along with his wife and collaborator, the psychologist Barbara Mellers, ran a team named the Good Judgment Project. Rather than recruit decorated experts, they issued an open call for volunteers. After a simple screening, they invited 3,200 people to start forecasting. Among those, they identified a small group of the foxiest forecasters—bright people with extremely wide-ranging interests and unusually expansive reading habits, but no particular relevant background—and weighted team forecasts toward their predictions. They destroyed the competition.

Tetlock and Mellers found that not only were the best forecasters foxy as individuals, but they tended to have qualities that made them particularly effective collaborators. They were “curious about, well, really everything,” as one of the top forecasters told me. They crossed disciplines, and viewed their teammates as sources for learning, rather than peers to be convinced. When those foxes were later grouped into much smaller teams—12 members each—they became even more accurate. They outperformed—by a lot—a group of experienced intelligence analysts with access to classified data.»

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.