99 - CARLA CREMER IGOR KRAWCZUK - X-Risk Governance Effective Altruism

Speaker 3
The collapse of FTX is a vindication of the view that institutions, not individuals, must shoulder the job of keeping excessive risk-taking at bay, according to Carla. She closed the introduction by saying institutional designs must shepherd safe collective risk-taking and help navigate decision-making under uncertainty. So to finish off summarizing Carla's article on effective altruism, I mean, first of all, I should clarify these are Carla's words, not mine. I don't really have any skin in the game of effective altruism. I mean, if anything, my intuition tells me to be skeptical of it. It seems to have an air of elitism and grandiosity. It also has cult-like qualities that make me uneasy. Nevertheless, I think there are some very interesting topics to explore here in terms of governance, the effect of AI and other technologies, and of course, the power of information in society. Carla said, I'm quoting, she's sympathetic to the type of greed that drives us beyond wanting to be good, to instead be certain that we are good.

Speaker 3
Carla concluded that in order to make EA members take responsibility for their decisions, open philanthropy should fund a new organization to research on epistemic mechanism design. Despite the fears of bureaucratization, failures and wasted resources are inevitable. Epistemic collaboration is not a nice to have. According to Carla Cremma, it's essential. This is a clip from Professor Luciano Fluridi from Oxford University. I spoke with him last week, by the way.

Speaker 4
I published that episode. Similarly to Carla, he thinks that the real problem here isn't superintelligence or even AI alignment per se.

Speaker 3
It's more of a classical problem of governance and having the right institutions, the right policies, having more friction between the legal landscape and the technology landscape.

Speaker 4
And it should not be left to individuals to make the right decisions.

Speaker 5
You can devalue human skills. You can remove human responsibility. You can reduce human control. They rule the human self-determination. The rock who thought that their little drop of water was nothing. 18 years later has a big hole in it because drop after drop after drop, the drop will shape the stone or shape the rock. And I'm a bit worried that here, human nature, no, we philosophers, we know something about human nature. We'll kick in.

Speaker 3
Also joining us this evening is Igor Kraftchuk. He is a researcher and PhD student at the Lions Laboratory at EPFL. His current research focuses on generative models and reinforcement learning for combinatorial domains, in particular, their application to integrated circuit design. On the humanities side, he's particularly interested in the interaction of specific algorithm systems and their biases with humans and society at large. In particular, he wants to contribute to the mechanisms of accountability that ensure benefits of AI systems are not monopolized or abused and improve our understanding of governance structures in general. Somewhere in between is the subject of algorithmic mechanism design and game theoretical analysis of existing political economic systems. Now, I have had privileged access to a position paper that Igor is in the business of writing. As far as I can tell, he's almost finished writing it, but it's called Why I Do Not Worry About AGI Risk. He said that he's unhappy with the current state of the discussion about AGI risk that he encounters in the effective altruism community and its adjacent circles. On the first page of his paper, he gives an outline. He says that he thinks alignment is either a non-problem or identical with the principal agent, social choice, and good corporate governance problems in politics.

Speaker 4
He said that he doesn't agree that the risks posed by losing control of human level or beyond, AGI are plausible, or indeed large enough to pose an existential risk.

Speaker 3
I think it's interesting to compare Ethan's opinion. He is an AGI scaling maximalist with Igor's opinion.

Speaker 8
But do you think there's something like, because we're talking about scaling bottlenecks, is that an intrinsic thing in systems?

Speaker 7
I mean, there's a notion of like, there's an exponential and an exponential goes, it's an exponential forever until it hits the physical bottleneck and then turns into a sigmoid. So there's that notion of bottleneck. If you scale so far, you're even scaling further than the fastest future supercomputers, then you hit the bottleneck of computer on the x-axis.

Speaker 8
So what do you think the bottlenecks will be in the world that we live in?

Speaker 7
We're talking about extrapolating timelines at Valk AI, JACO, Trist stuff. I mean, basically the current narrative is like for the next 10 to 20, okay, for text, it's like, it's basically, computer's going to be the bottleneck until mid 2020s or mid 30 30s. For video, computer's going to be the bottleneck at least until the 2040s. And then at that point, it's going to switch to data being the bottleneck.

Speaker 3
Okay. Okay.

Speaker 7
And that's just for the capability. For the alignment part, it gets more complicated because they're like your bottleneck by how fast can humans send their preference data to the machine, do like RO from human feedback or whatever.

Speaker 1
Like it used to be that you were fine if you couldn't read, and then that became mandatory. And then it was fine to not really know anything about computers and now that is mandatory. Like if you want to participate and be like a fully kind of emancipated human in these societies. And in the same way, it's gonna probably be a requirement to enable people to own their own data, own their own, you call it like a digital footprint and have a type of sovereignty. And the things lost like the GDPR are like a response that is coming like just in time. But again, these are the same types of processes that have been happening in similar ways throughout the history. And we're acting as if it is big momentous thing, because it's always nice to live at the end of history. Like this is the final thing we need to solve. And then either utopia or disaster. But actually, usually things are very boring. And we will like have lots of lawsuits and lots of regulation, lots of political debates. But like, there's no magic. And if you go into this with his mindset of like, Oh, this is this is different. You kind of implicitly hyping up the technology. So like the companies selling the technology to deal with this data, they love it. Like it's a really good marketing thing.

Speaker 2
Bring it back full circle because the this idea which often crops up in the rationality crowd as well right the less one community and so on that right that the voter the average person is full of biases and it's all about trying to avoid those biases on browser and others have done fantastic work to show that a lot of the biases that we see in human psychology obviously have very good reasons for being there because they're very adapted to particular environments and even the current environments they make a lot of sense right and you can read those papers they're great but when we're thinking about systems and environments that aggregate those biases together allow them to exchange information through these deliberation tools we can also think about the complementary capacity that AI can bring to this right surely there will be spaces that no aggregation And deliberation procedure can properly cover and that is where we can complement now with algorithmic tools that yeah we can call them AI or AGI they will be generally intelligent in different ways and I think Chris is making this argument right that there's no reason to replicate humans we have a lot of humans around let's try and find a space where actually we are benefiting from having a different kind of bias which we have in these algorithms to complement the kind of

Speaker 1
So one thing that I would really like to hear almost like my personal take on this like AI risk alignment risk thing is that the framing of this is a technical problem is extremely like overblown and that like the really careful really competent researchers that like I meet and like I have in my lab like the most competent in my eyes they don't worry about the technical say they see technical problems to solve and like my my lab because I'm biased like they have like work on robust AI safe AI like they fairness and and so on but these are like these small incremental improvements that like we have tools to solve this when you blow it up to a degree that is being discussed in the less wrong alignment and so on crowd which all like trying to do the best and I think I'm actually like happy that they exist they have like a different opinion than me and they're taking this this more seriously but my critique is that they really seem to buy into the hive that this is something different and I remember joining my lab my PI actually kind of being a bit aggressive in my enthusiasm or this is something