Best of — Is A.I. The Problem? Or Are We?

Highlights

The Dopamine Dilemma
Key takeaways:
(* Dopamine is related to reward, surprise, and motivation, but scientists are still unsure what the dopamine signal actually corresponds to., * In parallel, scientists were also learning how to monitor dopamine in real time and this was leading to a better understanding of the dopamine system.)
Transcript:
Speaker 2
So we've been talking here about what happens once a machine has learned something. But a lot of your book is about how we are learning to help machines learn and the places we're taking inspiration from on that. And a lot of where we're finding some inspiration is actually us. So can you tell the story just of dopamine and what we have learned dopamine is for in the human mind? Yeah.
Speaker 1
So in the 70s and 80s, we were learning a lot about the dopamine system and we were developing the actual technology to monitor dopamine, individual dopamine neurons in real time and watch them spike. And it was producing a pretty mysterious story. We could see a monkey reach into a little compartment and find a piece of fruit and boom, there'd be this dopamine spike. But the fifth or sixth time the spike would go away. And so what was going on here and to make a long story short, you know, there was this question of is dopamine encoding our sense of reward? No, not exactly. Is it encoding our sense of surprise? No, not exactly. So what's going on? Because we know it's related to those things, but we can't really pin down what this signal actually corresponds to. In parallel, there had been kind of this movement within the computer science AI community called reinforcement learning. And the basic idea was let's build systems that can learn to take actions within an environment to get as many points as possible.

The Fruit is Always in the Box: A Story about Computer Science and Cognitive Neuroscience
Key takeaways:
(* Computer science and cognitive neuroscience are engaging in a dialogue that is informing each other., * This tells us that AI is not just working out engineering hacks, but is discovering fundamental mechanisms of intelligence and learning., * There are many different evolutionarily independent temporal difference mechanisms.)
Transcript:
Speaker 1
Suddenly, when you find a piece of food that you weren't expecting, you know, life just got a lot better than you thought it was going to be a moment ago. But if you come to realize that fruit's always in the box, then life is always about as good as you currently expect it to be. So there's nothing more to learn. I just think this is a remarkable story for a couple of reasons. I mean, primarily it is this idea that computer science and cognitive neuroscience are engaging in this dialogue, this kind of feedback loop where each is now informing the other. And it also, I think, tells us a story about AI that we are, in my view, this gives us some evidence that we're not just merely working out some engineering hacks that happen to solve video games or whatever, but that we're actually discovering some of the fundamental mechanisms of intelligence and learning. We've kind of independently found the same mechanism that evolution found. And in fact, there are many different kind of evolutionarily independent temporal difference mechanisms. If you look at bees, if you look at different species, we really are on to, in my view, the philosophical pay dirt of artificial intelligence, which is to figure out how our minds work to begin with.
Speaker 2
So I love this. So basically what is being said here is that dopamine is a way of updating our expectations about the world.

The Hedonic Treadmill: How Our Beliefs Affect Our Happiness
Key takeaways:
(* The hedonic treadmill is a general purpose learning mechanism that helps us adapt to our environment., * We have alignment problems with ourselves, meaning that our beliefs about the things that make us happy don't always end up making us happy.)
Transcript:
Speaker 1
And so for me, it tells a story about not just the hedonic treadmill as adults, but the way that evolution has given us this really general purpose learning mechanism that when you're one year old, you know, or six months old or whatever, waving a hand in front of your face, at first it's really delightful because you don't expect what's going to happen or you push something off a table and it falls on the ground and you're delighted because you had no idea what was going to happen. And eventually, right, you need to get your kicks by playing sports or by writing academic papers or writing books or whatever it is. And it's I think pretty remarkable that there is this general purpose, take delight in the ways that your predictions are wrong, but also improve your predictions, right? And this kind of sets up this whole trajectory of our life in a way.
Speaker 2
But one thing that made me reflect on is the way that we have alignment problems with ourselves. Oh, yeah. So take, you know, the dopamine function you're talking about there. One way of describing the hedonic treadmill is that the things we believe in advance will make us happy don't end up making us happy. So here we are like wandering through our lives, telling ourselves that if we just like work so hard and get to this point, we're going to be happy and then we're not. And now dopamine is not doing something wrong. Like it's, you know, optimized for fitness and all the other things evolution wanted for us, which is not always happiness.

AI and Baby's Learning Process
Key takeaways:
(* Montezuma's Revenge is a difficult game for an AI system to beat because it has sparse rewards., * Most Atari games are easy for the AI to learn from., * Babies learn how to play the game by mashing buttons and figuring out how to get more points.)
Transcript:
Speaker 1
But there was one game at which this model scored a total of zero points. And that game is called Montezuma's Revenge. And so there is this question of why was this one Atari game so difficult for this AI system to beat? And the basic answer is that it has what's known as sparse rewards. So most Atari games, you can essentially just mash buttons until you get some kind of points. And for an AI system, that's enough to bootstrap the learning process. And you can say, OK, how did I get those points? Let me do a little bit more of that in the future and so on. But in Montezuma's Revenge, you have to execute this huge sequence of really precise movements. Any mistake basically kills you. And only if you do this huge long chain of things correctly, do you get even the first points of the game? And so the whole premise of this learning algorithm was that you would mash buttons until you got points and then figure out how to get more. But how do you learn if you can never get the first points to begin with? And this is a riddle that is, I think, wonderfully solved by babies. So the computer science community starts looking over the fence at ideas from developmental psychology, because of course, human beings play these games with no problem. So there's something going on that enables us to understand how to play these games. We've known since the 60s that infants have this really strong novelty drive. And in psychology, it's sometimes known as preferential looking. So if you show an infant a toy and then an hour later, you give it a choice between that toy and a new toy, they almost always pick the new toy. And this is such a bedrock result that it's used as a way to study memory and perception and things like this in basically newborns.

The Importance of Novelty Rewards in AI
Key takeaways:
(* Novelty rewards can help AI software have a humanlike drive, which can help it achieve goals it couldn't before., * There is still some uncertainty about how powerful AI will become, but it is likely that we will see superintelligent general AI in the future.)
Transcript:
Speaker 1
And this is such a bedrock result that it's used as a way to study memory and perception and things like this in basically newborns. So there's a very fundamental reward essentially that people get from seeing new things. The idea was, what if we just plug this novelty reward into our video game system such that we treat the encountering of kind of new images on the screen as literally tantamount to getting in-game points, just as good as getting in-game points. Suddenly when you do this, the program has this kind of human-like drive. It wants to explore the limits of the game. It wants to go through the locked door just to see what's on the other side. It wants to kind of climb the ladder and jump the fence. And that's what it takes to beat this particular game. So that to me is just another one of these wonderful convergences where some of the insights that we're getting into these very fundamental drives in human intelligence end up being imported in a very literal and direct way into AI software and then suddenly it can do things it could never do before.
Speaker 2
Do you believe we're going to get super-intelligent general AI or do you believe what we're going to get is sort of like kids with sub-aut-like capabilities and certain things that we need?

The Subjectivity of AI
Key takeaways:
(* AI is becoming more complex and subjective,., * There is a potential for AI to be on a par with animal welfare by the end of the century.)
Transcript:
Speaker 1
But going back to some of the things we've talked about, you know, the dopamine system, some of these drives that are, you know, the fact that we are building artificial neural networks that at least, you know, to some degree of approximation are modeled explicitly on the brain. We're using TD learning, which is modeled explicitly on the dopamine system. We are building these things in our own image. And so the odds of them having some kind of subjective experience, I think, are higher than if we were just writing, you know, generic software. This is the huge question of philosophy of mind is like, are we going to know if we manage to create something with a subjectivity or not? I'm not sure. But these questions, I think, are going to go from seemingly crazy now to, you know, maybe on a par with something like animal welfare by the end of the century. I think that's not crazy prediction to make. Yeah.
Speaker 2
And then you add in the fact that you can create, it's not literally an unlimited number because there's a computing power associated with this. But at some point, the idea is this will be simple enough and computing power cheap enough that you can create marginal AI agents at very, very low cost, right? That is how you all of a sudden get all these, you know, super low cost human level labors. And it does seem a little scary. I've given a lot of attention to this question of what would AI do to us, you know, including well before super intelligent, right?

The Post-Scarcity Future
Key takeaways:
(* Society has given status to ridiculous things, maybe because we need to keep people busy, maybe because capitalism rewards somewhat absurd things sometimes., * However, in the postscarcity future, automated agents will do a lot of the fundamental work of the economy, freeing people to concentrate on things closer to the good life., * This will be looked down upon, but in the end, it will be a more enjoyable way of living.)
Transcript:
Speaker 2
One thing you might say about our society as it currently exists is we've given status to ridiculous things, maybe because we need to keep people busy, maybe because capitalism rewards somewhat absurd things sometimes. But however it is, like the way we actually attach dignity to roles and then train people from the time they're born to be like achievement monsters trying to get through those roles is also not a great way of doing society. And so maybe it'll be the case that at some point in the kind of post-scarcity future, you have automated agents doing a lot of the fundamental work of the economy and people can concentrate on things that say the classical philosophers would have thought are closer to the good life. Things that John Maynard Keynes thought would be doing when he imagined how rich we'd be, which is like we'll be painting and thinking about philosophy or even just more, I think, prosaically spending time with our families and going to the park and playing sports with our friends and having a drink with our buddies. And there will just be more time to enjoy being human and that won't be looked down upon because the reason it is looked down upon is we have needed to make that a low status, low class activity in order to keep everybody very engaged in this huge economic machine we want to feed.
Speaker 1
Yeah, I mean, I'm very sympathetic to Keynes' vision of like, you know, in hindsight, you're like, well, what went wrong, right? Because we really were on track.

The Problem with Technology
Key takeaways:
(* There is something pleasurable about visual unpredictability., * The natural world offers an antidote to the world of tech., * There is a problem with no one making money when people are in nature.)
Transcript:
Speaker 1
I want to offer a complimentary story to that, which is I think as some of this work on the dopamine system and intrinsic novelty seeking behavior and inference shows us not everything is about scarcity. There's something pleasurable about just visual unpredictability. That's why we like looking at screen savers, is why we like looking at campfires or moving water. And I think there's something valuable there too. And for me, the natural world offers something that's really an antidote to the world of tech. There's something that I have come to through the classic stuff of mindfulness meditation and just hiking and being in nature, looking at trees and finding that beautiful. You realize that there's a lot more psychological sufficiency in the act of just existing in the world, controlling your own attention, letting the world just as it is, entrust you and surprise you. And there's a problem, which is that no one's making money when you do that. And so this is kind of a macro version of what we were talking about earlier in our conversation about all models are wrong.

Analog: What Matters in a Digital World
Key takeaways:
(* Everything we do is about status competition., * Walking in a park is not as enjoyable when other people are not around., * Books can be helpful in overcoming status competition.)
Transcript:
Speaker 1
And so in order to get that very primitive level of just visual surprise, you have to check your Twitter feed or whatever it might be. And that puts you into this world of status competition because that's how they get you to engage and create the content that other people consume. But if you just walked to your neighborhood park, the park doesn't require anything from you. And so I think it is worth bearing in mind, at some level, skeptical to the idea that everything we do is about this kind of positional good, this kind of status competition. When I'm walking through a park, I don't think this is really cool because other people aren't in this park. I mean, you appreciate the absence of other people for not getting in the way of your experience of nature, but you don't think about it as like, this is cool because it is scarce or I am the victor of this competition to be in this park or whatever it is. But I get as much from that as I get from Instagram, you know, like that's funny thing.
Speaker 2
I guess actually a good place to bring it to an end here. So let's go back to analog for a minute. What are three books you'd recommend to the audience?

The Future of Religion
Key takeaways:
(* Julie Shaw and Laura Major's book, "What to Expect When You're Expecting Robots," is a persuasive look at the next decade in terms of human robot interaction., * James Kars' book, "Finite and Infinite Games," is a unique book that looks at what people are really trying to do and how they achieve certain ends.)
Transcript:
Speaker 1
Yeah. So thinking about both what's coming down the highway and also kind of how do we think about human motivation and human desire? So the first book that comes to my mind is by Julie Shaw and Laura Major. Julie and I were high school classmates and she's now an MIT roboticist who works on aerospace manufacturing and all sorts of things. Their book is called What to Expect When You're Expecting Robots. I think it's a really interesting and persuasive look at kind of the next decade-ish in terms of human robot interaction. I'm also thinking about a book by James Kars who was a professor of religion at NYU called Finite and Infinite Games. And there's this wonderful backstory to this book where he is a professor of religion who attended a game theory conference in the 80s and then writes this book which is religion meets game theory meets like Wittgenstein's Tractatus. It's this very weird, very unique book that is all about what are people really trying to do and there are certain things that we do to achieve like a specific end that we can envision in advance, other things that we do in this more sort of horizonal open-ended way to kind of surprise ourselves or prolong an experience. And I think it's a useful way of actually thinking about living one's life. It also maps to problems in AI in a very interesting way. The third one kind of coming back to what we were saying about the pleasure of being in the neighborhood park. I'm thinking about a book by my friend Jenny Odell called How to Do Nothing, Resisting the Attention Economy.

How to Do Nothing: Resisting the Attention Economy
Key takeaways:
(* Jenny Odell's book, How to Do Nothing, is all about how to live a life that is directed towards objectives rather than goals, and how this can be a useful way of thinking about problems with AI., * There are surprisingly relevant resonances between Odell's book and AI, as both rely on an explicit objective function to achieve their goals.)
Transcript:
Speaker 1
It's this very weird, very unique book that is all about what are people really trying to do and there are certain things that we do to achieve like a specific end that we can envision in advance, other things that we do in this more sort of horizonal open-ended way to kind of surprise ourselves or prolong an experience. And I think it's a useful way of actually thinking about living one's life. It also maps to problems in AI in a very interesting way. The third one kind of coming back to what we were saying about the pleasure of being in the neighborhood park. I'm thinking about a book by my friend Jenny Odell called How to Do Nothing, Resisting the Attention Economy. And it's in one level, you know, a love letter to hear neighborhood park and at another level, you know, an invitation to think about a world in which most of our activity is directed at some kind of objective. And again, I think there are surprisingly relevant resonances here with AI. You know, you can't make an AI system without an explicit objective function that it's trying to maximize. What does it mean to quote unquote do nothing? And there's something I think powerful about that.

up:: 📥 Sources