The best books on Physics and Financial Markets

I wonder if you could give us a brief overview of the history of financial thinking and its cross pollination with physics?

The start of mathematical financial modelling is also the start of the cross pollination with physics. The first person who developed a serious mathematical model of a financial product was the French mathematician and mathematical physicist Louis Bachelier in 1900. He approached the problem of how much you should be willing to pay for an option – an option being a contract to give you the right but not the obligation to buy or sell some security at a future time and at a price you determine in advance when you purchase the option. He had the idea that some basic statistical reasoning about probabilities should be used to work out how much that option should be worth. In particular, how its value should change over time. He developed this very striking connection between option pricing and certain ideas in statistical physics, and in thermodynamics in particular.

Bachelier was way before his time. No-one in mathematics and finance appreciated his work and it took half a century before he was recognised. In the 1950s, the great economist Paul Samuelson, the author of one my book choices, worked on the same sort of problem that Bachelier was interested in and, by chance, had  his attention drawn to Bachelier’s work. He managed to track down Bachelier’s dissertation and discovered that, although he did find one major error in its reasoning, the basic outline was identical to what he was working on. So it was at that moment that the kind of methods that had been pioneered by Bachelier half a century earlier became widely recognised as relevant to financial economists.

“Many traders have a deep understanding about how markets work but may not understand the tools they are using, in particular the models their computer systems are using.”

By the 1960s Samuelson and others had developed a real appreciation of the importance of a certain type of mathematical reasoning in finance. Other mathematicians and physicists, notably MFM Osborne and Ed Thorp, also rediscovered some of the Bachelier’s ideas and extended them. Thorp was the first to figure out how to take these abstract ideas and use them to make money. He started what is widely understood as being the first modern quantitative hedge fund in the late 1960s. However, much like Bachelier, he was a bit before his time. He wrote a book Beat the Market, another of my books choices, in 1967 but nobody paid any attention to it. It was only when the ideas of Bachelier, Samuelson and Thorp were repackaged by economist Myron Scholes and applied mathematician-turned-economist Fischer Black that their importance for practice was widely recognised by the big banks. The Black-Scholes paper was published in 1973, which, coincidentally, was the same year that the first US options exchange opened in Chicago. When the options market opened, banks found themselves drawn into a market they hadn’t really participated in before. The Black-Scholes paper showed how ideas developed in maths and physics could be applied to understand option pricing. Things really accelerated from that point on.

Was the 2007/2008 financial crisis caused by a failure of these ideas?

There is a sense in which you can say that, but I don’t like to as I think it runs together two separate issues. On the one hand is the reliability and value of mathematical models in general and the methodology that is used to construct mathematical models in finance. On the other hand, you have an entirely separate issue of how those models are used. I think a big part of what happened in 2007/2008, particularly in regard to mortgage backed securities, was far more to do with the misapplication of models than with the failure of models themselves.

In some sense the models did not give accurate predictions of the value of certain products, but they shouldn’t have been expected to. Any mathematical model represents a simplification. One begins by making a number of strong assumptions about the market conditions one is operating under and the nature of the product one is using, and in the presence of those assumptions you come up with some set of equations that will provide useful information to you. But when those assumptions fail, you cannot expect those models to be reliable any longer. So what happened in 2007/2008 is that we continued using models that were designed for market conditions that disappeared in 2005. The problem had to do with the way people who were using these models failed to pay attention to the reliability of their assumptions.

So the problem lies with the people who use these models rather than the models themselves?

Many traders can have deep understanding and intuitions about how markets work but may not fully understand all the tools they are using, in particular some of the models their computer systems are using. As a result they are unable to recognise situations where the assumptions underlying the models begin to make them less reliable. But in lots of fields, not just finance, there are lots of people in jobs who do not understand the broader ramifications of what they are doing. Presumably an inability to see the long-term ramifications of your specific acts shouldn’t single out finance over say politics or other forms of business.

But, putting politics and finance to one side, you could argue that the consequences of poor assumptions and modelling in other fields are generally not so catastrophic. In 2007/2008 most western economies were plunged into an unprecedented crisis. 

I want to turn that on its head. Yes, it’s absolutely right that problems in financial markets have severe consequences and one has to be careful to avoid introducing new types of risks into markets. But one thing we have learnt in the twentieth century is that the general methodology underlying mathematical models in finance is really just the best collection of tools we have for learning about almost any subject. What’s the alternative? If we don’t use mathematical models we’re stuck using gut instinct and intuition, which are certainly less sophisticated methods. It’s not as if those don’t have a role to play, but to say that using mathematical models is what led to the 2007/2008 crisis is short sighted. The benefits we have enjoyed from using these financial instruments over the last 40 years are such that their use is well justified.

Your first book choice tells us that financial markets are much riskier than we have wanted to believe. Please tell us more.

Let me begin by touching on Mandelbrot’s role in the history of financial thinking. As I have said, it was in the 1950s and 1960s that Bachelier’s work was rediscovered, that Osborne began introducing new ideas from physics and that Thorp began to get interested in the whole field. There was a flurry of interest in the use of statistics, particularly the idea of randomness in stock markets. The important thing to note here is that you might think that randomness in markets makes markets unpredictable. In fact, it’s the opposite. If markets are random that means that certain kinds of statistical tools are going to be extraordinarily useful. There isn’t extra information on top of the statistics that one could use to gain advantage, if in fact markets are random. So, if you want to price something like an option, where what you care about is the probability of certain events happening, random markets actually make things much more convenient for making the sorts of predictions that you’d like.

What Mandelbrot realised early on, at the start of the 1960s, was that the kind of assumptions about statistics that everyone was making were wrong. They just weren’t borne out by the data. He argued there was a much broader class of ways in which a system could be random, including ways of being random in which extreme events, such as market crashes, are much more common. Mandelbrot developed a theory of these sorts of extreme events, and how to do statistics in the presence of the dominance of these events. He showed that the kind of models being constructed were systematically missing the fact that market crashes and huge market gains were going to happen much more often.

No-one paid attention to him at the time. His papers were published and anthologised, but the field progressed without him. The Misbehaviour of Markets, which was published in 2004, is, in some sense, his vindication. After being ignored, he left the field and developed lots of fabulous new ideas related to fractal geometry. It was only in the early 1990s, after the 1987 financial crash, that people in mainstream finance began to think there was something wrong with the Black-Scholes model and related ideas.  Many saw that the problem was precisely the one Mandelbrot had first identified 30 years earlier.

This really points to the way in which the problem isn’t with mathematical modelling and the use of mathematical methodology to understand finance, but with the inattentiveness to certain kinds of assumptions. Mandelbrot was able to figure out the way in which these assumptions were faulty. In this book he very concretely lays out proposals for how we ought to think about the mathematics in light of these sorts of issues and questions.

It is said that Mandelbrot was able to simplify the complex. How readable is this book?

It’s not a very technical book. It’s written for a general audience.

Samuelson is often referred to as the father of modern economics. Why is this book such an important work?

Samuelson is rightly called the father of modern mathematical economics and this book is its foundations. It was Samuelson’s doctoral dissertation that was turned into a monograph.

Samuelson was a student of Edwin Bidwell Wilson, who, when Samuelson met him, was a polymath at the School of Public Health at Harvard, but had previously been a very successful mathematician and engineer. Wilson himself had been a student of J Willard Gibbs, probably the first great American mathematical physicist. He developed much of the modern theory of thermodynamics and was one of the first Americans able to genuinely compete with the greatest European scientists in the nineteenth century. So Samuelson had this direct academic lineage back to Gibbs and what is striking about this book is the extent to which this new economic science that Samuelson was introducing was so strongly in the Gibbsian tradition – it was doing economics the way that Gibbs did thermodynamics and statistical physics.

The basic idea here is that one can relate certain postulates about how individual atoms – or in Samuelson’s case individual investors and consumers – behave with higher level statistical treatments and even general macroscopic observables. The idea is that one can treat a box of gas in terms of things like its temperature, pressure and volume and derive formulas relating them to one another while ignoring the details of what each individual particle is doing. That one can develop a whole theory of gases, heat and of engines, using those kinds of macroscopic variables was part of Gibbs’ big contribution.

These are the ideas that you’ll find in Samuelson’s book – that we can use mathematics in the same way to develop statistical accounts of individual level phenomenon and also develop macroscopic, large scale treatments of things like inflation, GDP and economy wide variables. He saw very clearly that you could treat economies as a whole mathematically and that there was a statistical relationship between economy wide variables and the behaviours of people who make up an economy.

It was certainly a ground breaking work at the time, but more than 60 years on, is it still relevant?

What makes it relevant today is that basically it’s where everything in mathematical economics comes from. What’s striking from the point of view of science and markets is that we tend to think that sometime in the 1980s and 1990s, physicists started coming to Wall Street and the City of London introducing new ideas that changed economics and finance. Samuelson’s book makes clear how important statistical physics was at the very beginning of mathematical economics.

Didier Sornette examines how and why stock markets crash and suggests there are parallels with extreme natural events such as earthquakes. Why did you pick this book?

Sornette is another hugely impressive and wide ranging intellect who has contributed to many different areas of physics and economics. The ideas in this book can be traced back to his work on Ariane 4 rockets for the European Space Agency. There was a problem with these rockets when certain pressure tanks would randomly explode and no one could predict when this was going to happen. He developed a theory for how to predict when the explosions would happen. His theory relied on this basic insight: that the explosions were occurring not because of some particularly large effect – it wasn’t that the tanks were being hit by something or gas inside them was applying a large force – rather they were exploding because the tanks themselves were evolving into a state that would make them particularly susceptible to small kinds of effects.

This was an idea that Sornette developed in many different contexts. One of the areas it was most productive was, as you say, in trying to predict earthquakes. He suggested that the largest earthquakes weren’t caused by a single large outside effect, such as a large change in the centre of the earth; it was that in certain circumstances the rocks at the surface of the earth were much like the rocket pressure tanks, and evolved into a state where relatively small fractures would get amplified.

Sornette then looked to see if the same ideas could be applied to stock markets. Under what circumstances, he asked, do the small ups and downs in stock markets get amplified into major crashes? He argued that the same sorts of considerations were at play in markets as in rocket pressure tanks and earthquakes. The basic idea is a kind of co-ordination – that small effects are going to be amplified when there are correlations between different kinds of investor behaviour. One thing we hear about in connection with finance is ‘group think’ – that it seems at times that entire markets will take certain sorts of assumptions, behave as if they are true and stop questioning them even though the evidence is questionable; there is this co-ordination of investor behaviour based on a belief that something is true. Sornette argues that it’s the analogue of that sort of co-ordination in pressure tanks, earthquakes and financial markets that leads to this amplification process. He develops a whole mathematical theory of how to identify when that sort of condition develops in a market and to using that to predict when an ordinary intraday change is going to potentially get amplified into a market crash or bubble.

Sornette has plotted a set of early warning signals tracking potential bubbles before they burst. How accurate has he been? 

I think of it as the gold standard of what ideas from mathematics and physics can in principle do. He takes an incredibly difficult problem and applies an entirely novel way of thinking about it and offers concrete mathematical tools coming out of physics to bring that problem under control. I think what Sornette has shown is just how wide ranging methods can be and just how hard the problems that can be tackled are.

Your fourth book, first published in 1967, offers both amateur and private investors instructions on how to outperform the market. Why did you pick it?

Ed Thorp is a really fascinating guy. He started as a physics graduate student before switching to maths at UCLA, where he developed an interest in gambling and in particular roulette. He thought that basic physics was pretty good at telling you how balls behave on rotating wheels and tried to use it to predict where the balls were going to land. That sounds a bit outlandish, but he believed that you could build a computer to do this and sure enough he did ultimately manage to do that.  His interest in roulette took him to Las Vegas, where he also became interested in Blackjack. He realised that there was a source of information that was not being taken into account in the standard Blackjack strategy books, namely card counting. If you pay attention to the cards that have already appeared, he concluded, you can develop a strategy that allows you to reliably beat casinos at Blackjack. He wrote a book called Beat the Dealer, and in some ways Beat the Market is its sequel.

After writing about Blackjack, Thorp began looking for other betting scenarios in which the methods he had developed to beat casinos could be used. Financial markets were natural candidates. So, he read as much as he could about statistics and stock markets – including Mandelbrot, Osborne Samuelson and Bachelier – and developed a version of Bachelier’s option pricing formula that he thought would give reliable predictions about the value of an option. Beat the Market really does draw on many of the ideas in his Blackjack book. One of the keys in Beat the Dealer is figuring out how to manage your money. In Blackjack you have to sit there playing at the table, watching how the cards come out; you know when conditions are bad for you and you know when the conditions are good. The question is how much you should bet under those different circumstances, because even if the conditions are good for you there’s a chance you could lose, so you don’t want to bet everything. At the same time, you also want to maximise your profits when things are in your favour. So, he developed a set of tools for thinking about how much to bet under certain circumstances. These tools also play a central role in his stock market strategy.

“In 2007/2008 we continued using models that were designed for market conditions that disappeared in 2005.”

What’s shocking about this book is how little attention it got when it was first published. Subsequently it has become something all quantitative hedge fund managers have read and Thorp has become something of an idol. These days people are so secretive about their trading strategies and the models they develop. In 1967, Thorp wrote book on how to do it, and almost no-one did anything with it. There was, however, a New York broker dealer called Jay Regan who recognised its importance and suggested to Thorp that they start a hedge fund together, which they did. They consistently made almost 20% a year over a 20-year period which is pretty outstanding. The methods they started with had been in the public domain through Thorp’s book, but no one else recognised their value.

The author of your final book seems to be arguing that theoretical physics has lost its way over the last couple of decades. Is that about right?

That is about right. Smolin is a leader in an area of physics called loop quantum gravity, which is an attempt to put together Einstein’s theory of general relativity and quantum mechanics to develop a quantum theory of gravity. Physicists have been working on this for more than 100 years, and they haven’t yet done it.

Smolin has his approach to this problem but his is essentially the runner up when it comes to funding and position in the physics community.  The more dominant approach is string theory, which has had a lot of attention over the last 30 years. Smolin’s argument in The Trouble with Physics is that string theory is both puzzling and hugely problematic.

String theory, if I understand correctly, is a self-contained mathematical model that describes all fundamental forces and forms of matter. It’s basically a theory of everything. Is that correct?

It’s supposed to be a theory of everything – that’s its aspiration. Smolin’s contention, which is widely agreed upon, is that it hasn’t yet accomplished that goal. There is significant disagreement about how far along that path it has gotten and how successful it’s likely to be, but people have been working on it for 30 years and it hasn’t had the kind of experimental and theoretical successes that one might have hoped for and were promised by the theory’s advocates. Smolin’s point is that given that it hasn’t lived up to its promise, we should be very cautious about it and should be keen to look for alternatives. He argues that the physics community hasn’t done this and that this one approach has continued to dominate despite criticisms.

The reason I put this book on the list is that Smolin’s basic point is that sociological factors can be an impediment to success in science. His diagnosis about what’s happened in physics is that the string theory community became powerful when there was a lot of initial excitement about the programme, but then became entrenched and self-reinforcing. He argues that people who work on string theory are extraordinarily influential and hold powerful positions in the best universities and are able to dictate what kind of work is most important and interesting, and, in some sense, discourage criticism of that work.

I think the same kind of sociological factors that Smolin says have derailed physics, play an even stronger role in economics. Economics is in many ways a political discipline. There are strong vested interests in business and politics that are concerned about how certain problems in economics are solved, and what views get developed. For individual economists, there’s often a lot of money on the line in consulting contracts and this means there’s often more resistance in economics to outside ideas from fields like physics or even heterodox ideas coming from economics itself. If we learnt anything since the 2007/2008 crisis, it’s that the old ways of doing things have gaping holes. But we still aren’t looking to new ideas, or looking beyond the well-worn contours of classical debates in economics, and I think that we should.