This weekend I finished reading David Orrell’s book “The Future of Everything: The Science of Prediction“. As an applied scientist, the public perception of scientific modeling has been a side interest of mine. In particular, as science is pressed more and more into the service of politics and ideology, the general lack of understanding about what scientists know and how they know it should be a deep concern to us all. In The Future of Everything, Orrell attempts to give an overview of how scientific modeling has developed, what its shortcomings are, and how far we can really expect mathematical models to predict the future.
Effectively, Orrell starts with the following observation: despite an exponential increase in funding and computing power over the last 100 years, predictive models (particularly in the fields of weather and economic forecasting) have made surprisingly little progress in producing accurate predictions about the future. In fact, modern weather forecasts for beyond a few days are only marginally more accurate, on the whole, than a forecast based on the climatological average for a particular day, in spite of their increasing complexity. Orrell spends much of the book exploring why models fail to give accurate predictions, with climate, the economy, and genetics as his three case studies.
Over the course of the book, Orrell explores a variety of shortcomings in modern mathematical models which aren’t necessarily solved by better computers or more complicated models. Some of the most important ones are (in no particular order):
- Attempting to model complex non-linear systems is mathematically problematic: In the 18th century, mathematical modeling seemed to offer limitless progress. Newton’s laws had transformed a seemingly complicated universe into a few lines of mathematics. If we could predict the course of the stars and planets, surely the world was at our command. Well, not exactly. As it turns out, Newton’s laws of motion turn out to be one of the easiest physical things to model. As Orrell says, part of Newton’s genius was picking a system that was possible to model – the same being true of Gregor Mendel’s study of genetic traits in peas. There may be simple equations for how a planet moves around the sun, but trying to predict how the wind blows (or how a plane flies) is a lot more complicated.
- Chaos: Jeff Goldblum made chaos a trendy term in Jurassic Park, but it remains fairly misunderstood. In modeling, a chaotic system is one where small changes in the initial conditions can dramatically alter the trajectory of the system. Because we can never know the precise initial conditions of a system like the atmosphere or the economy, small perturbations in the initial conditions (or parameters) used in models can have a large effect on the resulting predictions. The fact that model parameterization is often at least somewhat subjective compounds this issue.
- Computational irreducibility: Systems exist which are fairly simple, non-chaotic, produce clear patterns, behave according to only a few rules, and yet are computationally impossible to predict. The best example of this is Conway’s Game of Life. The Game of Life functions according to only four rules, yet it is impossible to write equations which will predict the state of a cell at any arbitrary time. The only way to find out is to run the system.
- Emergent properties: Emergent properties refer to the unpredictable ways which simple entities interact to form complex results. Think “the whole is greater than the sum of its parts.” These emergent properties cannot be simplified to simple physical laws.
- Feedback loops: Most systems have competing positive and negative feedback loops which control the system. One example is blood clotting. Positive feedback is necessary to quickly stop bleeding. If unchecked, all your blood would clot and you would die, so negative feedback slows the process when it reaches an appropriate level. The way feedback loops interact with each other complicates model parameterization.
- Matching the model to past observed data does not ensure accurate predictions: Just because a model matches past observed data does not mean it is correct, nor that it offers any predictive power about the future. A chicken might build a model that predicts a long and happy life based on observations of the farmer coming to feed him every morning. That model holds well, until the day he becomes the farmer’s dinner.
Orrell summarizes as follows:
- Prediction is a holistic business. Our future weather, health, and wealth depend on interrelated effects and must be treated in an integrated fashion.
- Long-term prediction is no easier than short-term prediction. The comparison with reality is just farther away.
- We cannot accurately predict systems such as the climate for two reasons: (1) We don’t have the equations. In an uncomputable system, they don’t exist; and (2) The ones we have are sensitive to errors in parameterization. Small changes to existing models often result in a wide spread of different predictions.
- We cannot accurately state the uncertainty in predictions. For the same two reasons.
- The effects of climate change on health and the economy (and their effects on the climate) are even harder to forecast. When different models are combined, the uncertainties multiply.
- The emergence of new diseases is inherently random and unpredictable. Avian flu may be the next big killer – but a bigger worry is the one that no one has heard about yet.
- Simple predictions are still possible. These usually take the form of general warnings rather than precise statements.
- Models can help us understand system fragilities. A warmer climate may cause tundra to melt and rainforests to burn, thus releasing their massive stores of carbon. However, the models cannot predict the exact probability of such events, or their exact consequences.
So where does that leave us? Orrell again:
Einstein’s theory of relativity was accepted not because a committee agreed that it was a very sensible model, but because its predictions, most of which were highly counterintuitive, could be experimentally verified. Modern GCMs (Global Climate Models) have no such objective claim to validity, because they cannot predict the weather over any relevant time scale. Many of their parameters are invented and adjusted to approximate past climate patterns. Even if this is done using mathematical procedures, the process is no less subjective because the goals and assumptions are those of the model builders. Their projections into the future – especially when combined with the output of economic models – are therefore a kind of fiction. The fact that climate change is an important and contentious issue makes it all the more important that we acknowledge this. The problem with the models is not that they are subjective or objective – there is nothing wrong with a good story, or an informed and honestly argued opinion. It is that they are couched in the language of mathematics and probabilities: subjectivity masquerading as objectivity. Like the Wizard of Oz, they are a bit of a sham.
…
[A]s I argued in this book, we cannot obtain accurate equations for atmospheric, biological, or social systems, and those we have are typically sensitive to errors in parameterization. By varying a handful of parameters within apparently reasonable bounds, we can get a single climate model to give radically different answers. These problems do not go away with more research or a faster computer; the number of unknown parameters explodes, and the crystal ball grows murkier still. … We can’t mathematically calculate the odds, even if it looks serious, scientific, and somehow reassuring to do so.
Orrell is clear to point out, however, that the fact we cannot guarantee the accuracy of our predictions does not mean they are necessarily wrong, or shouldn’t be heeded. Varying parameters in climate models may in fact produce a wide range of results, but that doesn’t mean we should take a wait and see approach. Economic models failed spectacularly to predict the current economic crisis – but it still happened.
Orrell’s argument, then, is for a kind of literacy when using scientific models to inform decisions. Scientific predictions can be helpful, and often are. But they are limited in their ability to predict future events with certainty, and these problems aren’t necessarily going to be solved with better data and models, or with more powerful computers. They shouldn’t be ignored, but rather viewed for what they are: a tool for helping us understand the present, and hopefully make the best decisions we can about the future.