Terry Sejnowski on NeurIPS and the Future of AI

Speaker 2

Yeah, that space of algorithms out there being infinite is something I've often wondered how you think about it or people in the field think about it particularly with regards to neuroscience and this impulse to build a model of the brain or figure out how the brain works or build an artificial analog to the brain. Do you think these algorithms as in mathematics, they exist in nature and we're discovering them or do you think that we're creating them? A little of both, like the Boltzmann machine was a version of the Hopfield net.

Speaker 1

We basically heated it up so that it was fluctuating and from that we got a learning algorithm with Jeff Hinton and we had the wake and sleep phases during which you gave it data that you're trying to train on then you let it free run and you subtract the two and Jeff Hinton is coming back to that. So it's still that concept is still permitting.

Speaker 1

We basically heated it up so that it was fluctuating and from that we got a learning algorithm with Jeff Hinton and we had the wake and sleep phases during which you gave it data that you're trying to train on then you let it free run and you subtract the two and Jeff Hinton is coming back to that. So it's still that concept is still permitting. And by the way, you know, humans also have wake and sleep cycles too, right? But what we're looking for are principles that we can extract from nature and one of the principles that is probably the most important principle is this principle of scaling. So all algorithms, as you increase size of the problem, they scale with some exponent. For example, if you're doing some kind of a search and you have to compare all pairs that goes as n squared within objects that you're searching trying to find the optimal one. But the problem is that as the end gets bigger and bigger, a million, a billion, which is where we are today, then it just runs out of steam. You run out of memory. It's not practical. And most of the algorithms in traditional AI have that problem. And when we started in the 80s, we had tiny little networks.

Speaker 1

And that could be applied to any problem. It's not that each domain has a separate set of rules. It's like we could learn them. And now here's what we didn't know back then. The 80s, we had these small networks and we had the learning algorithms. The same ones we have today, by the way, but we didn't know how well it scaled. What would happen if you scaled it up by a factor of a thousand or a million? Well, now we know. It turns out they scale beautifully. And this is a principle that we actually had observed in nature. If you look at the size of the cortex of different species, what you discover is that, especially in primates, more and more, the volume of the brain is devoted to the cortex, cerebral cortex that is on the outside. And in humans, it's so abundant that it gets convoluted, it gets all these folds. And so it looks like a walnut from the outside, right? That's because you need more surface area. You've got to get the same surface area and the same volume. And so how do you do that? And it's very, very rare, even in biology, because most of the other parts of the brain don't scale that way. So there's something special about cortex more is better. And that's what we've discovered about some of these neural network models that it scales linearly. That is to say, as you add more units or parameters, it scales with that number. It's not N squared, it's not N cubed, it's N.