The best books on Watson

You’ve been following Watson behind the scenes as he was created at IBM. We’ve now all seen him thrash two humans, Ken Jennings and Brad Rutter, both former champions, on Jeopardy!. Here’s what I’ve been dying to ask someone in the know: how was it possible, on the second night of the tournament, that Watson thought Toronto was a U.S. city?

Let’s review this. The Jeopardy! category was U.S. Cities, and the clue was, ‘This city’s largest airport is named after a World War II hero, and its second largest airport after a World War II battle.’ Watson starts hunting around, looking for these two connections to airports throughout the U.S. and, more broadly, North America and the world. Why would it look beyond the U.S.? Because Watson is never completely sure that it understands the clue. It has to hedge a bit, and allow for the fact that it might not understand.

Watson has also learned, through statistical analysis of the Jeopardy! categories, that they don’t always coincide with the question. For example, a clue on American novelists might say, ‘This masterpiece features a young man named Holden Caulfield.’ The answer to the clue is not J D Salinger, it’s Catcher in the Rye. Watson is aware – statistically at least — that categories can’t always be trusted.

So in the U.S. cities/airport question, Watson goes on a hunt and never really finds an answer it has high confidence in. It has abysmal confidence in both Toronto, which has a couple of airports named after World War I heroes, and Chicago. It probably doesn’t understand the Battle of Midway, so it doesn’t make that connection. Because it has very low confidence, it doesn’t rule out Canada. A lot of people would say, ‘Well, that’s a sign of idiocy,’ and you could argue that, in this case, it was. But Watson has to allow for exceptions.

What exactly do you mean? 

Say there was a clue about a 1970s rock star, a male, famous for a song called ‘School’s Out’. If Watson, like traditional computers, had a list of female names to use to figure out if someone was a male or female, it would never consider Alice Cooper. The list would tell it that Alice Cooper was a woman’s name. (The same applies to Evelyn Waugh.) Watson is built so that if there’s enough evidence that something is right, even if it appears wrong, it can go with it.

But the IBM people, at one point, had a team devoted to trying to steer Watson away from this type of embarrassment. It was called the ‘dumb team’. They knew how much it would hurt IBM if Watson, despite an overwhelming performance, committed a brain-dead mistake like that. They actually considered having Watson say ‘I don’t know’ instead of hazarding a guess! Because we have common sense when we guess, and Watson utterly lacks it…

We should be getting on to your books, but I can’t resist asking: What about the timing issue? A lot of these questions are quite easy, perhaps not for me, but for the likes of Ken Jennings and Brad Rutter. So it’s not about whether they know the answer, but how quickly they can get to the buzzer. Now I know Watson can’t hear, so he was receiving his question via an electronic signal, which presumably arrives instantly. Doesn’t that give him an unfair advantage over the humans, who have to wait for Alex Trebek [the show’s host] to read out the question?

Watson ‘sees’ the electronic version of the message at the same moment that that message is exposed to the humans on the screen. Humans do not listen to Alex Trebek. No one listens to Alex Trebek.

Oh. What do they do?

What the humans do is they speed read the answer on the screen. They immediately look for keywords to see if they have enough confidence to bet on it, so that if they win the buzz, they can come out with the answer in the 4-5 seconds they have to come up with it. You can see Ken Jennings, especially, doing that: winning the buzz and then struggling to come up with the answer. That’s what humans do.

But doesn’t Watson have an advantage because he can read it instantly?

I would argue the opposite. I would say that that is one of Watson’s great disadvantages — seeing words and struggling to understand them. Language is extremely foreign and difficult for Watson. Where Watson has a real advantage is in its speed to the buzzer. Once it has calculated that it has enough confidence in the clue to buzz, then, as soon as the light goes on, opening the floor to buzzing, Watson requires only 1/100th of a second to buzz. The humans are not that fast. And the way humans compensate for this —  and Jennings and Rutter were themselves famous for this, frustrating other Jeopardy! players — is that they anticipate the end of Alex Trebek’s sentence. Almost the way a jazz musician can sense the coming downbeat, they establish a rhythm with the guy sitting on the desk who opens the buzzer. So when Alex Trebek is finishing his sentence, they anticipate it and beat normal humans to the buzz. Of course, the danger is that in Jeopardy!, if you buzz too early, you’re locked out for a crucial quarter of a second. So Watson beat them on the buzz in most categories, and some people  — Alex Trebek for one — didn’t think this was fair.

I felt the same.

But think of it from this point of view. In 2006, the head of research at IBM was going around the various research teams saying, ‘We need a new challenge. We did chess. Now we need to do something that puts computing into the worlds of language and knowledge. Let’s do a Jeopardy! thing.’ Everyone turned him down, because IBM had a system that answered questions — it was a slow-responding programme that got two out of three questions wrong. And these were questions that were much more simply posed than the Jeopardy! questions. So it seemed like an extremely difficult goal. At that time, you would not think that this machine had any advantages at all. Watson had to overcome enormous obstacles, and now, when you get that machine to the point where it gets 80% of the questions right, and reaches an answer in 2 to 3 seconds, all of a sudden you start saying, ‘Hey, that’s not fair!’

It’s a human instinct.

The other thing you hear is, ‘Yeah, well humans are still smarter.’ And that is undoubtedly true. This machine is a contrivance. In almost every area, other than answering certain types of questions, Watson is a blithering idiot.

Let’s discuss your books. Your first choice is After Thought, about ‘the computer challenge to human intelligence’.

This book is by James Bailey. He wrote it in 1996, at the dawn of the age of the Internet. It’s for generalists, and it’s a very interesting look at the history of maths and science. Bailey’s thesis is that we create mathematical systems that fit with our needs and the tools that we have available. The Greeks wrote on papyrus, and the question they wanted to address was, ‘Where are we in the universe?’ In order to answer that, they came up with geometry, which also fit their medium. You can draw right angles and circles on papyrus. A little bit to the east, in Persia, people used reed styluses to poke in clay tablets. It was harder to do geometric forms that way, so they worked in algebra. With the Scientific Revolution, all of a sudden you had mechanical clocks. The question was no longer ‘Where are we?’, but ‘How do we measure or predict the pace of change from one place to another?’  So Newton and Leibniz came up with calculus. Lines and circles do not work well in movable type, which is what they were using, and so they described their findings with algebraic formulae of letters and numbers.

If you move ahead to the 1920s, the job of the first electronic computers was to replace people doing computations, and speed that process up. But now, Bailey says, we’re entering a new age, where what we’re looking for is patterns in enormous sets of data. He argues that we need new types of maths — and new types of machine configurations — to address that. It’s an extremely interesting and well-written book.

From the reviews I read, it sounded like Bailey was being quite critical — that he thinks we’re creating computers too much in our own intellectual image.

Bailey worked for one of the early super-computing companies, founded by Danny Hillis. They were doing sophisticated parallel processing. They divided up the work and distributed it to a lot of different computers, analysing enormous amounts of data and looking for patterns. Watson does this too. It’s a different kind of thinking. Bailey claims that we’re not educating ourselves for this; we’re educating ourselves and our children for the kind of maths Galileo needed, not the kind of maths the founders of Google or the Watson people are using. I found it a very engaging book, and I have trouble with maths and science books – I don’t usually finish them.

Let’s move on to Darwin Among the Machines, by George Dyson.

Dyson is looking at the genesis of thinking machines. Well, machines that do something like thinking. He is really good on the history of computing. One of his early chapters covers Samuel Butler, who in the mid-19th century took a ship from London to New Zealand and set up a sheep farm there. He was on the other side of the world, both literally and figuratively, and yet, around 1860, a boat came into the harbour carrying a copy of Charles Darwin’s Origin of Species. Butler read it, and it provoked him into some very interesting thoughts. Humans aren’t really evolving much physically: we have pretty much the same brains and bodies that people had in ancient times. Machines, on the other hand, are moving ahead very, very quickly. In other words, the evolution of humans is actually taking place through our tools. Our intellectual advance is going to be tied to our tools. He was thinking this when the telegraph was still very new, and the machines he was thinking about were steam engines. So for him to come to this conclusion was brilliant. He foresaw the age when we would create smart machines that would take cognitive leadership of the planet. Hopefully, the machines will be kind to us, the way we are to animals we care about…

Until recently, I was worried that technology also weakens us: Soon no one will be able to read a map, let alone navigate by the position of the stars, because we’re entirely reliant on GPS. But then I interviewed Daniel Wilson about Robotics. He pointed out that humans have always been completely useless anyway. Out in the wild, we’re fangless and clawless — easy prey for any wild animal. We’ve always just been a function of our tools.

Our tools, and the know-how of our fellow humans. I couldn’t farm; I couldn’t build a house or a car. If I were reduced to my own abilities, I would really be living a Stone Age type of existence.

I don’t think I’d be alive at all!

We should all take a moment to appreciate engineers, because they take this stuff and they make it happen for us.

Your next book is Glut: Mastering Information Through the Ages.

This is a fascinating book. It’s by a young guy called Alex Wright. He looks at the beginning of information, starting from single- and multi-cell organisms and how they transmit information, and goes right up to the Ice Age information explosion. From there he goes on to the Ancient Library of Alexandria, and then Gutenberg. It’s a whole history of information and what we, as humans, decide to keep in our heads. This is something I wanted to deal with in my own book, Final Jeopardy, about Watson. With machines like this, we have to say, ‘OK, so these things are becoming an external lobe or a brain that we all share. And if it’s an external lobe, what do we keep in the part of the brain that is between our ears?’

You mentioned GPS. Alex Wright deals with it very intelligently, and he has great stories about information economies that I never knew about. For example, he talks about Ancient Peru, where messengers used to travel across the Andes carrying woven threads known as ‘quipus’, or talking strings. When a messenger arrived at his destination, he would deliver his news while reeling off knots in the string like a rosary. For the Incas, people of no written language, the quipu served as their core information technology — it was a newspaper, a calculator, even a repository of laws. There’s a lot of interesting stuff in the book. And he finishes up with the Internet. From the time of Gutenberg, we’ve settled on the written language as the basis of learning. Now, with the Internet, we are reverting to a more oral, conversational tradition.

What was your conclusion about what we should be keeping in our brains?

Each one of us will have a different answer to that. But when you learn about Watson, and see Watson getting built, you see not only what Watson is strong at, but also what it struggles mightily to do, and what it is utterly incapable of. That’s a road map of where the greatest potential is for us. Watson has no capacity to come up with original ideas. It cannot carry on a conversation; it cannot create theory; it has no understanding of causal relationships. All of these are areas where we can, for many decades to come, remain masters in the world of information.

But if we have Watson answering questions for us, how much information do we ourselves need to know?

A lot of us like having information; it’s nice to have information up in your head. If you look at the knowledge Ken and Brad have, you can say, ‘Well, I can find out those things by asking Watson, or by looking it up on Google.’ But if you have knowledge in your head, ideas come from that. It’s much harder to have that magic occur when you have to look up 20 things on Google first, and then figure out the connection between them and come up with a brilliant idea. It’s more likely to happen if it’s already up there.

On to The User Illusion: Cutting Consciousness Down to Size.

This is by Tor Nørretranders, who is Danish. It’s about how our minds and our memories work. What he focuses on is the tiny bit of bandwidth through which information can go into our brains. It can handle very, very little information at any one time. Much of what we experience in the world is furnished by our memory and what we expect to see. For example, I’m looking outside. I’ve looked out this window many times, and I see the tiles with the last bit of snow melting on them. My eyes are focused on one particular tile with a little bit of snow on it. Around it, the shed across the street, the trees, I’m not really seeing them. I have an image of them, but it’s furnished by my memory. That’s important in Jeopardy! and all kinds of areas of our life. We, as humans, fill in the blanks with what we expect to see, or what we expect to think or hear or know. Very little is actually coming from our senses.

I’m going to experiment on you, based on something I read in this book. Here’s my question: How many of each type of animal did Moses bring onto the ark?

Two.

You fell for it! It should be zero. Because Moses did not have an ark. But your mind says two because you’re focusing on the important part of the question, which is how many animals went onto the ark. That’s what we do. Jeopardy! creates clues to stump players sometimes, and the funny thing is that Watson actually falls for those tricks too. Watson too will think, ‘Animals, ark, Biblical figure, I’ll answer two, even though it shouldn’t be Moses but Noah.’

The other fascinating thing in the Nørretranders book relates to information theory. He spends a lot of time on the second law of thermodynamics, and extends that into the world of information. You’ve got all this information and it’s subject to entropy. It’s all over the place; there’s no intelligence to it; it’s just a flood of data. You’ve got to use a whole lot of energy to distill that information into intelligence. And that is what Watson — and computer science generally — is all about.

Lastly, you chose The Soul of A New Machine.

This is a highly acclaimed work by Tracy Kidder. He was embedded in a team at a computer company called Data General (which no longer exists). It was 1979, and they were building a mini-computer. He was introducing to the world this group of people that later became known as ‘nerds’ —  though I call them the numerati. These were computer scientists and electrical engineers, and they were putting together this machine that for most of us was utterly foreign back then. We knew NASA used computers; we knew they were important; but they weren’t part of our lives. Kidder was introducing that engineering culture to us — the way they thought, the way they analysed things. It’s a marvelously detailed book, and I found it very inspiring, as I tried to write a book that, in a way, had a similar goal, as I followed a team at IBM that was building a machine.

It’s also an interesting way of looking at how computing has utterly changed in the last 30 years. They were building a machine, but all it could do, from our perspective, was count things and put things into columns and calculate stuff. Now these machines are dealing with music and images and ideas and facts. They’ve invaded our world.

Post-Watson, there’s been a lot of speculation about where it’s all heading. As someone close to this particular machine, what’s your prediction?

If you look at Watson, it made some foolish mistakes and it struggles to understand English. They’ve made remarkable progress in that area, but there is a whole lot of refining to do.

So it’s the language that’s the key to the whole thing?

Yes, the language in understanding the clue, and also in the information that it is searching through. So finding meaning and then figuring out not only that it has an answer, but also that it has enough confidence in that answer to bet on it. That’s the remarkable thing. There are all sorts of critics out there who pooh-pooh Watson and say, ‘Oh, it’s not that smart — it’s just looking up facts.’ But this in itself is a very hard thing to do, and Watson only does it well enough to win a game of Jeopardy!, which means it gets about 85% of questions right. So Watson has a lot of learning to do. It has to get a lot smarter and also a lot cheaper, so that we can all have access to this kind of information technology.

And we all will, because we need it. We are all swimming in vast oceans of data that we ourselves are creating. All of that data has little bits of intelligence in it, but we can’t find them. Let’s say you’re a neuroscientist. Every day, neuroscientists are doing brain scans and creating massive amounts of data and studies about how the brain and the nervous system work. And they’re publishing their findings. Last year alone, there were some 50,000 academic studies published on neuroscience. If you’re a neuroscientist, how could you possibly be on top of your field? You need machines to help you with that. Sometimes a machine like this, if you ask it a question, will come back with 10 possible answers, and confidence in each one. A number of those might be stupid. One or two might even be outlandish. But if it comes back with some good ideas, then the humans, with our superior brains, can pick out the ones that are worth pursuing.