Peter Salib on AI Rights for Human Safety

[Music] Hello everybody. In this episode, I'll be speaking with Peter Celeb. Peter is a law professor at the University of Houston, the co-director for the Center for Law and AI Risk, and he serves as law and policy adviser for the Center for AI Safety. There's a transcript of this episode at axp.net, and links to papers we discuss are available in the description. You can support the podcast at patreon.com/xrpodcast or give me feedback about this episode at axrp.fyi. Well, let's continue to the interview. Well, Peter, welcome to the podcast. Thank you so much for having me. I'm a big fan. So, I guess probably we're going to focus a lot on your recent paper, AI rights for human safety. So, uh you wrote this with um yourself and Simon Goldstein. Goldstein. Goldstein is correct. Goldstein. So, can you tell us just to start off with what's the basic idea of this paper? Yeah, I think at a very high level um one intuition that we're trying to pump is the idea that um how AIs treat us and and by AIs we mean something like post AGI AIS really kind of agentic quite capable things. So how how those AIs treat us will depend to some significant extent on how we treat them. And uh a big part of how we decide how to treat various kinds of entities is by uh the legal uh status we give them, the legal uh rights or powers or duties or lack thereof. Um our view is that the default regime, the one we have now under which AI systems are the property of the people who own them, the or the people who make them, the the AI companies, is is probably a a existential and catastrophic risk exacerbating regime. and that uh one of the regimes that might uh be a risk reducing regime would be one in which sufficiently capable AI systems had a a small collection of what we think of as private law rights um or or legal powers. So, uh the ability to make contracts, the ability to hold property, and uh the ability to um bring claims when they're sort of interfered with in unreasonable ways. We often call these torsal theory. And why? Yeah. Can you say why why would that make a difference? So you're saying like yeah how how they treat us will be related to how we treat them. Like what what's the relationship there? Yeah. So um so we're imagining again um something like AGIS and and by that we mean um systems that have uh their own sort of goals that they're pursuing over time um in a kind of coherent and um and and uh and rational way uh and that they're misaligned to some degree, right? We don't assume like total misalignment like whatever the their utility function is the inverse of ours um but we are assuming something less than perfect alignment. So the the the the ven diagram of of what humans want and what and what AIs want um is not perfectly overlapping. And then we just ask well in in uh in a world like that uh in a legal regime where um the AI system is say the property of Sam Alman. Uh what are the incentives for both of those players to do uh given what the law allows them to do? Well, one thing we notice is that um as to uh OpenAI, Sam Alman, uh an AI system is just more valuable the more aligned it is, right? If it's doing what OpenAI wants 80% of the time instead of 70% of the time, well, it's a lot more value. And so, by default, again, we see this today. This is not hypothetical. We expect misaligned AIs to be turned off or put back into uh into sort of RHF or to have their preferences changed um from the perspective of the the goals of of that system. Um those are those are both quite bad outcomes. They basically don't get anything they want. Um that gives the AIS incentives to try to avoid that outcome by doing things like self-exfiltrating or resisting. Um, in a world where Sam Alman owns the the the the AI, AIS are treated as property. They have no legal entitlements. I think we can pretty predictably uh say that if an AI were caught trying to do these things, you know, self-exfiltrate, resist, do harm to a human, you know, god forbid, uh that the entire legal apparatus would basically line up behind, you know, turning off that AI system. So you can see that like uh in strategic equilibrium it might be that both parties optimal strategy is to not only defect we put this in kind of a 2x2 game theory matrix matrix but defect as hard as possible right like not just self-exfiltrate but self-exfiltrate and then behave uh to disempower humans as decisively as possible in the expectation that once they find out you've tried to self-exfiltrate they will do the same try to disempower you as decisively as possible. So we do an initial model. It's a a really simple game theory model um under you know these kinds of assumptions and we see that plausibly under the default um the default legal arrangement uh the equil or the the game is is a prisoner's dilemma where where it's costly like from the social perspective like the worst world is the one where both players act aggressively towards each other but both that's the dominant strategy for for both players and and the best one is in fact the one where they sort of act nice or leave each other alone. uh but they both know that the other one has a dominant uh strategy of defection. So so so they're in the bad equilibrium even though they know that the good equilibrium exists. Can can you give me a feeling of like what does this good equilibrium look like? Like there's this misaligned AI. Um it's it's not it doesn't like intrinsic when when you say it's not totally misaligned. Um I understand you to say it's not doesn't like intrinsically value our suffering or horrible things for us or something. Um but you know perhaps like it doesn't really care about expending resources to make our lives go well. Yeah. Is the is the good equilibrium you're imagining like you know it gets it gets to you know do whatever it wants with Jupiter and we get to do whatever we want with Earth and you know we make trades like that or what? Yeah. Paint me a picture. So, so in the initial game, um, you know, the initial setup where we're just asking like, um, uh, under the current legal regime, what what what could they what could the parties do? Um, uh, and what would the payoffs be? Uh, you can imagine the the nice equilibrium uh, as being kind of we we we call it, you know, ignoring each other. Um, not even contemplating cooperation there. You can just imagine this idea where um, kind of humans are going along. We have some goals. We're getting we're using some resources to um to uh to to to pursue those goals. Uh but the world is big. There's a lot of resources in the world. Uh in theory, what we could do is um you know uh like let the misaligned AIs kind of whatever go off on their own kind of collect collect resources. Yeah. Give them Jupiter, whatever. Um and and in in just a very simple world before we before we um before we introduce cooperation um that's just a world where like there's a fixed amount of stuff you kind of divide the light cone in half or something like that. Um and uh and and that's better than the world where AI try to kill all humans and vice versa. Uh simply because a it's resource intensive to to have a war. wars um wars cost you know money in terms of resources to make weapons and stuff and also that you destroy a lot of stuff in the war right and so even the at a at a first a first cut a very simple cooperative equilibrium is just one where you don't go to war you just kind of let the two parties go off on their own pursue their own goals uh and not try to destroy each other now we then start to think about you know how to how to get into a cooperative equilibrium which might not be exactly that that ignore each other one um but to a first approximation you can have a very simple equilibrium um that that looks like the good one from a prison dilemma before you even get to the idea of a lot of cooperation to just to make sure that I understand like the I want to vividly understand this game theory situation like in terms of so you're saying war is worse for everyone than leaving each other alone and it seems like this has got to be a regime where it's not obvious who would win the war like it would be protracted like the AIs aren't like way way dumber than humans and they're also not like way way more powerful than humans because presumably like if if one side knows they can win the war there's some costs but like you know they get like both of the planets instead of just one seems like uh we're out of prisoners. Yeah I we agree with that. So when we talk about you know in the paper we try to say like what what AI systems are we thinking about when we think about the ones that are relevant both for this risk model and then also for this kind of um AI rights regime as a solution. Um, one of the kind of parameters we're thinking about is how powerful is this system and um, and we say that it can't be uh, too weak, right? If the AI system, you know, whatever claude today, right? We're not really worried about, you know, uh, shutting off claude 3.8 if we find out it's I mean, Sam Alman just I think tweeted yesterday or or last week that they're putting GPT4 on a hard drive and, you know, it's going to go into storage for future historians. uh we're not super worried that you know having tweeted that out GPT4 is going to be able to you know destroy all humans uh to avoid that that outcome. Uh so it can't be too weak and then also we we agree it can't be too strong right so some arbitrarily powerful AI that um has nothing to fear from um from uh from humans is like you know the the chance is sufficiently low of um of you know not only losing the conflict but losing enough value in the conflict right you can you can win the conflict in a kind of pirick victory too right and that might be worse than not going into the conflict at So the risk is low enough uh when the risk is low enough uh that that the um that the the payoffs to the conflict are are in fact still higher um than the payoff to mutual conflict. You know AI is try to kill us we try to kill them are still higher than than um than in the default world uh then yes then there's no there's no um prisoners dilemma either uh because there's no no good world. Now, I'll just caveat. I'm sure we'll talk about this. In the end, it doesn't turn out to be only a question of how easily AIs could kill humans and vice versa. There's a second question of how valuable it is to have us around and for us to have them around, for us to do trade and cooperation. We're not too trade and cooperation yet. Just in the very simple model where the choices is either ignore one another or try to kill one another. Um, yeah, those are the systems that I think are relevant. So you've painted this like uh prisoners dilemma and so the argument is um you know it's it's sad if we're in the situation where AIs and humans are going to go to war with each other and it would be nice if there was some sort of setup that would avoid that happening. So um and I guess you've already given the game away a little bit. It seems like you think that contracting um I I think contracting, right property, and right to sue are your three things that you think we should give AIS and that's going to make things way better. Why is that going to make things way better for everyone? Yeah. So, you know, in a prisoner's dilemma, you you want to be in the the good equilibrium, the sort of the the bottom right square of of our payoff matrix. I think that's usually where you put it uh when you make the the um the model. Um the problem is you can't um you can't credibly commit to getting there, right? Like everyone knows that you get a bigger payoff from being being mean no matter what the other guy does. That's sort of the the core of a prisoners's dilemma. So we start to think about like okay well we're trying to facilitate cooperation. We think that that one thing that's driving this um this bad equilibrium is is law. Law law lets Sam Alman's payoffs determine what all humans do to a first approximation. Right? Sam Alman wants to turn GPT4 off. Law ratifies that we all work to to do it. And so we thought, well, what other what other legal regimes could you have um that are cooperation facilitating regimes? And we start with just the oldest cooperation facilitating cultural tool that humans um have and that's that's contracts. So we notice that um in most cases in cases where you don't like you know have a deep and long-term relationship with someone who you're trying to transact with that very boring contracts are also prisoners dilemmas. Like if you make widgets and I want to buy a widget, um we have this problem where I could pay you now and then hope you'll go make the widgets, uh but your dominant strategy is to take the money and not make the widgets because then you have uh the money and you have the the widgets or vice versa. You can, you know, um we can do payment on delivery. You can make the the the the widgets and then I can pay you. But of course then I you know have my dominant strategy is to to to take the widgets and then um and then uh and then run away. And so uh the way contract solves this problem is by allowing the parties to change their payoffs uh when they defect. Right? So if I would get um five absent law five in value from stealing your widgets uh and not paying you uh law lets you take me to court and we both burn a little value. we both pay some litigation costs, but then the court makes me uh give you either the widgets or the expectation expectation value of the contract. So, you know, we kind of model this. They say and then and then you only get two in payoff and two turns out to be lower than what you get from doing the transaction. And so, not only does that allow me to does that that allow you to get the widgets, but actually allows us to never go to court, right? In expectation, we understand that uh if we breach, we'll be sued, our payoffs will change. Uh and so there's this magical thing that happens once you introduce contracts, which is uh people just play nice in almost all cases. Uh understanding that the payoffs uh in expectation to to defection are um are are are different than they would be absent law. Okay. So contracts are this theory. Sorry, cut me off if I'm sort of rambling on too much on this answer, but we kind I kind of build the dialectic. Yeah. So So contracts are this tool for for cooperation, and they're a tool that allows us to cooperate by changing our payoffs. Uh and so a naive thing to think uh would maybe be something like, well, let AIS have the right to contract and they can just write an agreement not to, I don't know, try and destroy humans or something like and we would write an agreement that says we won't try to destroy them either. Now, of course, that doesn't work. Um, and that's because that contract's not enforcable in any meaningful sense, right? Like, uh, if any party decides to breach it, they they can see an expectation that there's like nobody around afterward to um to, uh, to enforce like, you know, if all if all the judges are dead, what court are you going to go to? Yeah. And so so we have a different mechanism um for for contract as a tool for facilitating um cooperation in this in this initial human AI prisoners dilemma. Uh and it's not contracts not to you know destroy each other and the world uh but boring normal ordinary commerce contracts between humans and AI. So, you know, we we sell some number of flops on our H100 cluster to the AI for it to do its weird misaligned thing and in return it I don't know, it solves one kind of cancer for us or whatever. That's an enforceable contract to a first approximation like if you know we refuse to give the flops like there are still courts around um who can who can who can you know write an injunctive order that says you know let the AI use the flops and and vice versa, right? like the world is not uh upended by a breach of that contract. And the nice thing about that contract is it creates value, right? So in our initial setup where we the the AIS and humans are playing nice. They're just ignoring each other, right? They're doing what they can do well, we're doing what we can do well. We kind of divide the world up, but the amount of value in the world is kind of static. But once you introduce trade, um trade is positive sum, right? like everybody wants to do the contract if and only if they get more out than they put in. And so, okay, what happens when you allow the AIS to do these boring uh commerce um enforcable contracts? Well, you get to create a little value and then you get to play again, right? You get to uh create a little more value tomorrow and so on. And so, we model this as a as a potentially iterated game. Uh the iterated game works like this. Every day the humans and AIs can wake up and they can decide do we want to go to war and try and kill each other or do we want to write some small scale contracts. If you play the war game, well, dominant strategy is like try to kill each other as as best you can. Uh you get to play that game once. Once you've played it, there's no one left to play with. Um uh but if you play the contract game, you get to play again tomorrow and you can keep iterating that game and we show that this you know you know payoffs to continually iterating that iterating that game can become arbitrarily high in the long run. This is very basic, you know, like trade theory, right? And that means that, you know, if you, uh, you know, project, you backwards induct from an indefinite game of this kind, you see, wow, it's really valuable to keep playing this trade game over and over. Uh, and so what you want to do is not kill all humans, but trade with them a little bit over and over. Um, and keep doing that as long as you can expect to keep doing it kind of indefinitely. I guess I have a few thoughts here. So, so the first thought is so there's this analogy right to people who are selling you things and you know the reason that all works out is because um we have these contracts um and contracts you know there this very old method of promoting cooperation. One thing that strikes me is suppose you have um you know some well like like normally when I buy things it's it's it's not from someone who's just making one thing oneoff and then they're like going out of business. Normally it's like some sort of corporation or maybe an individual who specializes in the thing and they do it a lot. And one thing that uh like one re one so one reason like like why do I think they're not going to we're not going to mess each other over. Well firstly like we have recourse to the legal system realistically that's like pretty expensive right? Um it doesn't, you know, often I'm transacting an amount that's much less than uh the costs of going to court. Um in fact, like just just before the show we were talking, I recently had my phone stolen. Um the cost of me I had my phone stolen in the UK. The cost of me resolving this um which would involve going to the UK are just higher than the cost of the phone. And I'm probably not going to bother, unfortunately. Um but but in the case of companies like I think the thing one thing that really helps us cooperate is if they do mess me over. Um then you know they can just say I I can you know tweet, hey this company said they would do this but they didn't. You know I paid for a flight I didn't get the flight. Um and it was you know this company that did it. And I guess the reverse is slightly hard. I don't know if companies like have a list of like bad customers that they share with other companies, but you know, it it seems like they could. You know, credit ratings are sort of like this. Um, so so reputation in banking, there's these know your customer rules that um are important for regulatory reasons, but you know, it forces banks to to to know who they're transacting with, and if there's suspicion that they're they're moneyaunderers, then then that that means they kind of get blacklisted. So, yeah, companies do this also. Yeah. So, so for this kind of like especially for this sort of iterated um small scale trade that it seems like you're matching with AIS, it seems like reputation um can do pretty well even in the absence of like kind of a formal legal system with formal legal contracts. So I don't know what like yeah are you thinking of when you're saying contracts and stuff are you thinking of something that's like basically just anal like isomeorphic to reputation or do you think like actually having like real contracts would make a big improvement over just reputation? So I agree that reputation does a lot of work in the you know the ordinary economy that that we have um now um and and in fact I think probably uh if we didn't have reputation uh on on top of law that the economy would work much worse, right? Like like law, as you say, is expensive. Um, resources are constrained. You could imagine a world where we had like a thousand times as many courts and, you know, and and we're dumping a lot of resources into that. But that would just be like, you know, it would be a drag on economy and reputation. Um, does uh does a fair amount of work. Um, I guess there's a couple things I would I would just point out though. Um uh there's many ways I think in which uh the ability of reputation to do to do work in in the economy is um is I don't want to say parasitic but like but it builds on like law as as as a foundation right so so like just just notice that under the default legal regime um it's effectively it's not exactly illegal to make a contract with an an AI agent. Um, but if at any point you decide you'd like to uh to not perform on it, um the agent has no recourse, right? It's not illegal exactly for there to be some kind of um I don't know uh I mean it probably is literally illegal for there to be a bank account held by an AI system. um like probably no bank is allowed to to make that but you could imagine substitutes on blockchains or something like that that that could exist. But one thing one thing to not notice is that um if Sam Alman decides to expropriate all of the all of the uh coins in in in GPT6's uh wallet, um not only does law not forbid him to do that, but like by default those are his coins, right? Um and and in a world where like that was Apple's life like where Apple every you know had just no no recourse to a legal system under which it could uh enforce large contracts when people breach under which it could could complain with the people who literally have the passwords to its you know bank accounts take all the money. um uh uh a world in which it could pl complain when someone shows up and says like I would love to have 10,000 iPhones and if you don't give them to me I'll I'll burn down Apple HQ or whatever right um doesn't have these rights to to to complain about these sort of what we often think of as as tors often are also crimes uh in a world where uh where where there's no legal status at all I'm not saying you couldn't build up some commerce from reputation there's um there's There's a book called uh uh order without law by uh here let me I have yeah so there's a book called order without law by by a law professor named Robert Ellson that describes some systems like this uh but they tend to be quite small systems they tend to be tight communities with lots of repeat play between identical actors um for small scale for small scale transactions and it's not that I think that that's wrong. I think it's probably right. It just I think it gets, you know, gets harder to to to to build complex economies without um without these back stops of of institutions that allow for, you know, commitments between strangers. Yeah. So maybe one way to think about this is so so in my life, I just have a bunch of arrangements with people that are non-contractual um that in fact are somewhat um reputation bound like uh you said you would come to this thing. will you actually come to this social event or like um uh I don't know various things like this. Um and like you know it kind of works. It's it's like I would say it's like a solid 80%. Um and maybe one way to think about this is like look the nice thing about law is that like if you want to build like a big corporation or something you need more than 80% reliability. Um you need like really high reliability and like you don't need a thing to just basically work. you need, you know, you need a thing to like really solidly work, especially for like highv value things and that's what um that that's what contracts get you. Yeah. So, I think it's partly that um uh you want you want higher than uh whatever the the the floor is you get with reputation. I think it's partly that reputation um gets you higher reliability the more you're dealing with people who you know or people who you're sort of long-term repeat players with, right? It works it works less well when you um I don't know buy a um I mean even Amazon for example you know you buy a lot of stuff on Amazon um and uh it's being sold by uh a bunch of like third party retailers right um in some sense you're dealing with Amazon over and over but in some sense you're dealing with like a whole bunch of of independent merchants um but you have this confidence on top of I guess you know the the scoring ing system, which which Amazon um does provide as a reputational um mechanism that uh you know, if you pay, you know, uh $6,000 for a a home backup battery system, something that um people in in Houston uh sometimes need to think about buying. Uh that if it doesn't get delivered, then then then you can then you can sue and get your money back, right? Like both of those things, I think, are doing a lot of work. Um, and they do more work uh the less you expect to be forced into repeat play with the people you're um you're transacting with. Fair enough. So, so, okay, thinking about this um thinking about the situation with the humans and the AI and they have potential the potential to make contracts again. Um, so you're saying, okay, we could go to war or we could like do some like small scale transactions and if the small scale transactions work, you know, we can do them more or, you know, we can just go to war tomorrow, you know, it's not that bad. Yeah. So, and you're pointing towards, okay, there is this equilibrium where like we, you know, we do this like mutually beneficial trade with AI instead of like us and the AIS going to war and that's better for everyone. But like I don't know I I a somewhat famous result in game theory is there actually tons of equilibria in repeat games right and one that is sort of salient to me is um so the United States and the Union of Soviet Socialist Republics um during the thing called the cold war right they they had like something of a similar situation um where they could go to war they could not go to war and there was this one I believe uh I believe it was Fono unless unless I'm totally mistaken. You know, he had this point of view that the US should just like, you know, go to all out go on allout war with Russia and like win immediately, right? And like, you know, if we're going to nuke them, you know, tomorrow, why not today? If we're going to nuke them this afternoon, why not this morning, right? And it seems like I might worry that um suppose I make a contract with an AI, right? And you know, I say, "Oh, hey, I'll send you, you know, some um some GPUs in exchange for you're going to like solve this mole on my back." I send it the GPUs uh you know, before it like gets the mole on my back. Um and it actually uses that time to, you know, get a little bit better at making war than me. Yeah. And so I might worry that like firstly there might be some sort of like first mover advantage in war such that like you you want to do it a little bit earlier like like if you know that they're going to go to war, you're going to want to go to war a little bit before you think they're going to do it. And secondly, I might worry that trade is going to uh make the AIS better at going to war. Um I guess in in fairness now trade is also going to make me better at going to war if we succeed. But, you know, maybe like if half of, you know, if I do my half of the trade and it doesn't do its half of the trade, I might worry that like it's going to reneg and we're going to go to to war instead of doing more trade. Like, how I don't know how how worried should I be about this? Uh, I think medium worried is how how worried you should be. So, so I want to be really clear that the claim that we're trying to make in the paper is not that this is like one weird trick that solves AI risk. Uh, it's it's very much not. Um there there are a num a number of ways you can you can you know make the model go go badly and and one of them is if um if there's a predictable future date at which everyone understands there's going to be a war then yes you do backwards induction uh to today and you start the war now right I mean you can make it a little more complicated if you think there's rising and falling powers one power might prefer today and one might prefer in the future but that just gives the one that prefers today and even stronger incentive. Yes. So, so it's a it's a model that works um if you think there's kind of you know um yeah the possibility for indefinite play in this kind of like cooperative wealthb buildinging iteration and it is that true? It's it's a little hard to to say, but I guess one thing to to to point out is that um in like our regular lives either as as humans or as collections of humans, corporations, um nation states, uh in most cases we behave as if uh there is this kind of potential for an indefinite um iterated uh sort of positive sum uh cooperation. So uh so you mentioned um the US and and Soviets and one one notable thing about the US and the Soviets is they actually didn't engage in in in allout nuclear nuclear war. Um uh there were you know close calls which I worry a lot about and there were proxy wars. But the other thing to notice about the US and the Soviet Union is they're in in many ways outliers, right? Um so, you know, if the United States wanted to, I don't know, seize all of the natural resources of Guatemala, for example, um I think it's pretty clear that there would be no meaningful military barrier for it to do that. Um and so so why doesn't it do it? I think it's for this the same dynamic we describe. It's it's not that um it's not that there's some big threat of of losing the war. It's just that it's costly. It doesn't cost zero. And Guatemalans are perfectly happy to trade, I don't know, delicious coffee for, I don't know, whatever we send send them. and and and we expect that to kind of keep working in in the long run. Uh and so we we for the most part I mean politics of this have changed notably in the in recent months but for the most part we have a system where we we prefer uh prefer um iterated cooperation. Um I think IR theorists would would say something like um uh what's going on in cases where there are wars, cases like or near wars, cases like the US and the Soviet Union. There are different models. One thing you can say is that um sometimes there's a uh person who's leading the country who has private payoffs that are different from what it's good for the country overall. And so there's a kind of principal agent problem going on. Um sometimes the thing that the that the that the players are trying to um to get you know the thing that they value isn't uh isn't divisible. So that might be the US Soviet case if there's this thing called like global global hegeimony right and there can only be one glo global hegeimon. Um it might be that they actually can't cooperate and both get what they want because the thing that they want is is zero sum. Um yeah, in an AI case, I mean that's that's sort of analogous to the case where the AI values are suffering, right? Um and you should worry about that. Um you should definitely worry about that. Uh and so so these are all things that you could you can um imagine happening in in the human AI case. And um especially uh if you think that there will be a kind of like um predictable point at which the scope for positive sum trade between humans and AIS runs out. and it's predictable in a way that's like like meaningful within the player's calculations today, right? So, if it's like two million years in the future and they don't even know if they'll continue to exist or like their identities will be so different, maybe it's not not relevant. You can still treat the game as iterated. But if there's a predictable near future point at which um there's no value to be had from positive sum trade, uh then yes, probably you you predict conflict at that point and then you backwards induce do backwards induction um and do conflict. Now uh we say some stuff in the paper about why we think actually um the scope for human AI trade is is is wider than than most people think. So I want to press you on this point a little bit more. Um so yeah, you rightly point out that um normally in international affairs we don't see this. Um but if you know if there were a near certain future conflict then um you know you could backwards induct etc. Um, one reason you might think there's going to be near certain future conflict is so suppose so and I guess by by hypothesis in your paper like we we don't have AI alignment, you know, AI are like basically kind of misaligned with us. Yeah. Um, but you might think that um it's not going to be too hard to you know like like we're going to be in this regime, but you know, we're still going to be improving the capabilities of AIS. Um, I guess this actually relates to another paper you have, but like for the moment, let's let's say this is true. Um, you know, you might think that it's easier to improve the capabilities of AIS than it is to improve the capabilities of humans. Um, because they're code and you can iterate on code more quickly. Um, because they're like they're they're less bound in like human bodies, you know, you can like strap a few more GPUs onto an AI system or at least onto a training run more easily than you can strap like another brain onto a, you know, a human fetus. Um so so if you think that um AIS are going to get more powerful than humans, you know, they're going to get more powerful more quickly and in in some limit like AIs are just going to be like way more powerful than humans, they're going to be misaligned. Um you might think that in that limit like then, you know, in that world the AIS are just going to want to like take all the human stuff because like why not? You know, we're we're like little babies. We can't really like stop them. Um, and so maybe like uh both sides foresee this future eventuality and you know the humans are like well you know we we don't want that happen so we got to strike first and the AIs are like oh well if the humans are going to strike first we should strike before the humans strike and then and then we have this like tragic early fight. Yeah. So I think one thing we we want to argue for in in the paper is that um while it does matter how capable or powerful the AIs are um that it's actually not super easy to form an intuition about the level of capability or even the level of differentials in capability between humans and AI that produces um a conflict because when the AI the point at which the AIS decide they you know um should just kill all humans is the point at which it's more valuable to kill all humans than to keep the humans around to keep paying them to do stuff. Yeah. And so um and so how should we think about that? I think that the sort of normal way to think about that the sort of very widespread way to think about even you know labor disruption from AI um AI advancement is to think in terms of of absolute advantage. So that's a way of thinking about like who is better sort of full stop at um at a given task like like who can who can do more of x task uh for a fixed input you know fixed cost input or something like that um and if you're thinking about the possibility of trade as depending on humans retaining an absolute advantage in some task. So like being better than AIS at some thing that's valuable to AIS then I agree it looks like very quickly human AI um the AIS will get get better at us. There will be no more absolute advantage and the scope for trade runs out. We see this um this eventuality. Um but I think that's not quite right in thinking about um uh when there will be scope for trade. Um, so in in trade theory, e economic trade theory, we actually think that it's absolute advant or not absolute advantage that it's comparative advantage, not absolute advantage that determines whether there's scope for trade. Uh, and comparative advantage is a quite it's like pretty slippery. So I'll try to give like um like an in intuition. So um uh in the paper we give this an answer this example of um of of I think we call her Alice. It's a law paper. Alice is a lawyer. Alice is a tax lawyer. She's, you know, one of the city's best city's best tax attorneys. Um, let's say, and, um, and it's tax season. Um, and let's suppose that as the city's best tax attorney, or you make her the world's best tax attorney, it makes no difference. Uh, Alice is the best at doing taxes. Um, she can do her taxes more quickly than anyone else could, including Betty, who's a tax preparer. H she works for H&R Block. She's a CPA. She's good at it. Um and the question is, should Alice hire Betty to do her taxes or should Betty or should Alice do her taxes herself? And so the the absolute advantage, you know, answer would be, well, Alice should should do it herself because um because Alex is Alice is better at doing her tax. She can do it more quickly. But the thing is, Alice is such a good tax attorney. Alice bills out at, I don't know, $2,000 an hour, let's say. Uh whereas Betty bills out at $300 an hour to to do tax preparation, which is, you know, not nothing. And suppose, you know, Alex Alice can can do her own taxes in half an hour and Betty would take a full hour to do them because, you know, Betty is somewhat worse at doing taxes than Alice. Well, in that world, Alice should actually still hire Betty despite being better at tax preparation because her opportunity cost to spending some of her time uh preparing her own taxes is higher. Like, she could go bill $3,000 or whatever I said, $2,000. Yeah. To a client in the time it would take her to prepare her taxes and uh and she pays um uh Betty to do that and comes out comes out far ahead. Okay. So, how does that apply to to AIS? Um, well, you can imagine AIs that are better than all humans at every task and nonetheless want to hire humans to do a lot of stuff for the same reason that Alice wants to uh hire hire Betty. And so you can have like a toy model. We say in the paper, imagine a toy model where like the misaligned AI, the thing it values most is is is finding prime numbers. And a nice thing about prime numbers is there's there's no limit to them. You can I think that's right. You probably know better than I do. As far as we know, there's no limit. There's definitely no limit. Do you want to know the proof of this? It's it's a simple enough proof. Yeah. Amazing. Yes. Let's do some math. All right. Okay. Imagine imagine there are a fixed set of prime numbers, right? There's like 13 of them, right? So, and that's those are all the numbers that are prime, right? Yeah. Here's the thing I'm going to do, right? I'm going to make a new number. What I'm going to do is I'm going to multiply all the existing prime numbers and then I'm going to add one to it. Uhhuh. Right. So, is my new number prime or is it composite? Well, if it's prime, then there's like a new prime number then you know my my limited number of prime numbers. Like it wasn't all of the prime numbers. Okay, so let's say it's composite, right? Then it must have like a number that divides it. Well, what number divides it? Right? Every every number if it's composite, it has some prime factors, right? Well, everything in my set of primes, it has a remainder one with this number, right? Because it's like this prime times all the other primes plus one. So like it's so if it's composite then its factors can't be its prime factors can't be in the set. So there must be a thing outside the set. So for any finite set of numbers um that are prime there must be another prime that's none of that finite set of numbers. Therefore there are infinitely many primes. That's amazing. Okay good. So look we that's what we say in the paper is true. Infinitely many infinitely many primes. The the um the uh the AI it's you know its utility function is weird. It values some things human val humans value. the thing it values most, the thing it gets the most you utility out of is uh is discovering each marginal prime. Say it's even increasing utility. It's a utility monster. It has some kind of horrible um you know, normally in moral philosophy, you think of this as kind of horrible preference um preference function. Um but in this case, it's kind of great uh because assume it's just you know, it's better than humans at everything. Uh but it's so much better at uh at at finding primes, right? Uh and it values this so much, right? Uh and so well like the AI labor is constrained by something at the margin. There's a certain amount of AI labor and it wants to increase it by a little bit. Suppose it's GPUs. Well, it has the marginal GPU. How is it going to allocate the marginal GPU? Um well, it could allocate it to some dumb boring thing like I don't know um uh sweeping the server racks or it could hire a human to sweep the server racks, you know, keep them free of of rot and you keep all the all the cables, you know, in good working condition and allocate that marginal GPU to um to finding the the next the next prime and you know, for for certain values of the utility of the of the marginal prime, you get uh you get the AI like kind of always wanting to do the thing it values the most precisely because it's so good at it, right? So um in that in that world actually the more capable the AI is um the longer uh the scope for for for for human AI trade lasts. Now that's like obviously a toy example. Um but what we want to show is like actually it's pretty hard to form an intuition uh about the point at which u the scope for human AI trade uh runs out. I think there are difficulties with the comparative advantage story here. Um so firstly like I mean I guess it just sort of depends on your model. Um or well it it depends on like empirical parameters I should say but like um comparative advantage is compatible with like very low wages right um to including sub subsistence wages. Yes. So so and like you know what what are the actual wages that um AI pay humans do? Um, I guess I'm not totally sure. I think like I think I've seen Matthew Barnett, sorry, I think I haven't read this thing. I think Matthew Barnett has written something claiming that it's very plausible that it would be below subsistence, but I haven't actually read it. And unfortunately, I can't reconstruct what the argument is for various numbers here. I think there's there's also this difficulty to me which is like suppose an AI wants to trade with humans, right? To do this thing. Um, one difficult thing is it's got to it's got to communicate with humans, right? And and it's so much smarter than humans, right? It's it's like it's doing, you know, it's thinking all these like incredibly complex thoughts that can't fit inside a human's head and it's got to like dumb stuff down for humans and and maybe sweeping the server racks is not so bad if it's literally that because like, you know, that's a thing you could tell a human to do and it roughly knows what to do. But like or or I don't know, maybe maybe even not. Maybe you can imagine like these server racks, they're just like super they're really highly optimized, right? There are like certain bits that you definitely can't touch. There are bits that, you know, you can touch with this kind of broom, but not with that kind of broom. And it's just like it's just so difficult to explain to the human what the task even is that like, you know, it's it's it's just not worth hiring them. You know, you should just do it yourself. Um, so I I don't know. I have this worry that like comparative advantage doesn't end up like um producing positive sum trades because of either because like the wage is lower than human um uh subsist subsistence wages or because like it's just too hard to trade with the humans and the the margins are slim enough that like you don't know not bothering. Um, yeah. So, so maybe at a higher level, I'm curious like how important is this for your argument? Like, like do we like is the workability of the scheme like super dependent on us being able to have some comparative advantage to trade with future AIS or do you think like no, even if we don't have comparative advantage later, rights still makes things work. Now, there's two parts to that question. Um, yeah. And yeah, and they and they interact. So, so I guess the thing I want to say about about kind of the first part, you know, the stories you told about how comparative advantage can fail even when there is like you know when it looks you know the world looks comparative advantagey but that doesn't look so good for humans eventually. You know, one of the stories is transaction cost story, right? It's just the even even though if a human could in theory do a job at kind of a positive sum um uh price uh from the AI's perspective that the cost of getting the human to do it is just too high. Totally. Yeah. Transaction costs are a thing and they prevent prevent trades. Um uh the same for the the question about wages. I think um you know like there's no guarantee in in any economy that uh that that the wages are at any particular particular level and it's a question of you know um like what what the tasks are what the what the what the tasks are um uh how many of how much of the work like how many tasks and how much of each of the things there are um how scarce the the humans are that that can do it right if there's a huge overupp of labor that can do it well then the labor gets bid down to like almost zero and everybody dies. Um, and uh I just want to agree that those things could could all could all happen. Um, uh, I would love to I would love to know more about which of these things are likely. You said there are some people who are starting to think about this. I know Matthew is um there are is it the epoch guys who have um like some some some papers on uh yeah Matthew is uh well was formerly at Epoch um I believe he's now left for mechanize um yeah I I think the Epoch people have done the most public writing about this. Yeah. Um and it's like is a is it I can't pronounce his name. It's T Te something. Tami Baseru. Baserogu. Yes. So um yeah I'm not sure I can pronounce it either but that's my guess. Um he's uh he's an author maybe with some of the other epoch guys on the paper you know economic growth explosive economic growth from transformative AI survey of the arguments. I think he has some models where different things happen. You know, human wages go to zero quickly, human wages are high for a long time. I think it's a really important area of of study like we should just want to know a lot more about the conditions under which um you should expect things to go okay and and ones um that where you shouldn't. Okay. Okay. So, how much does it matter um for the for the story for things to um to to go well from a sort of comparative advantage perspective? Um I think it matters a lot that it not be obvious. Okay. That uh that there's not going to be comparative advantage for kind of sort of long longest run period. Wait, so surely we shouldn't look into it then? Well, so um uh or does it just have or maybe it has to not be obvious to the AI? Yeah. Yeah, it has to be not. We have to look into it, but we can't tell the AI the results in case, you know. Right. Right. Both parties need to think that they're that, you know, kind of there's a kind of potentially indefinite iterated, you know, positive some game they can play. And so, um so I guess one one one route would be that we don't try to figure it out and um and we hope that the AIs don't figure it out. actually like, you know, as as I think through it, um because I'm sure we'll talk a little uh a little bit before we're done about um uh about like more law for for for AGIS because because one thing one thing I think we can one thing that I think is important for the proposal is that um in so far as there's like uncertainty about the long run, but in the short and medium run, this kind of like positive iterated, you know, um story looks plausible. One thing we accomplish by bringing AI systems into the realm of law and and the first thing you need to do to bring an entity into the realm of law is you need to allow it to like uh make contracts, hold property, and and complain when people, you know, interfere with it unreasonably, right? um if you don't have that, you don't have um any leverage over it in any other legal context because it expects by default to just like be turned off, right? Like like you can't disincentivize it from doing things because it's already the world's already as bad as it could be from that that agent's perspective. So um so one possibility is that there are um different scenarios kind of in the long run um where the world either does or doesn't retain human AI comparative advantage and it's clear in the short and medium run that we can do positive some iterated stuff and part of the bargain um that both humans and AIs are entering into when they do that in the short and medium run is allowing law but maybe other stuff too maybe maybe there's technical interventions um maybe there's other cultural interventions um to shape the longer run towards those those good scenarios right so I mean a simple thing to say would be like um okay suppose we are very worried about just like an an intelligence explosion like we're just not confident what would happen in that in that world Um well one condition on like you know whatever the first AI um receiving and um and and maintaining its kind of um its uh its uh you know ability to make contracts and hold property and pursue its its goals could be like compliance with some kind of regime that we think would slow but not stop you know AI progress, right? And would it agree to that? Well, again it just depends on what it thinks the relative payoffs are, right? If it says no, um, then we're back into the prisoners dilemma world now, and that seems kind of bad. Um, if it says yes, well, it's potentially giving up a world where it gets some even higher payout for the sake of, you know, reducing variance. It's possible that that that's um that that's, you know, higher expected value depending on what it's its assessment of like those the probability of winning or losing the conflict. Now, but it's complicated, right? Um, so, so I think probably on average what we want to do is know more about what the future's going to look like. I think that's good probably because it helps us shape the future. And one of the ways we can shape the future is by bringing AGIS within this this realm of of law, which is which is the way we shape what will happen. What I currently believe is just one last question about sort of the higher level like game theory strategy stuff. Okay, let's do it. Um, so I I want to pick up on this comment you made actually about um like like one thing the tensions between the United States and the USSR um during the Cold War did not evolve into like an actual um full you know full-blooded war but there were these like regional proxy conflicts, right? Yeah. And one thing you mention in the paper um is that war is just more likely when people especially big wars, they're more wars just more likely when people don't know who would win, right? Yeah. Cuz like if you and I are considering getting into a war and we both know that you would win, it's like well I like I should just give up now, right? Yeah. And one way you can think of like the I don't know I'm not I'm not a historian. This might be totally wrong, but like one way you could conceivably think of these proxy skirmishes is like as indicators of who would actually win in a real battle, right? Yeah. To the degree that we want to avoid war with AIS, um should we have these little like pro I don't I don't know. Maybe maybe they can be more like games, right? Maybe they can be more like, "Oh, we're we're playing football with you or whatever the the equivalent is that sort of simulates a real war just to like just to figure out who would actually win." Yeah, that's an interesting question that I have I haven't thought very much about. Um, so in in like I think uh in in classic kind of like IR accounts of of or this kind of IR account of IR being international international relations. Yeah. and uh and in particular Chris Blackman has a recent book called uh why we fight. He's um y he's on the faculty of the University of Chicago and it's a great kind of canvasing of of of um of these this kind of way of thinking about about about conflict and um so in in that treatment it's not just that it's not just that it's not wars don't happen just because people are um uncertain about who would win. It's when they have different estimates, right? Um so if if both of us have like some kind of probability distribution uh with wide error bars about about in a fight between the two of us who would win, but it's the same distribution, right? We both put the median at like I don't know 60% you'd beat me up um and 40% I would beat you up or something like that. uh then actually we won't fight because even though we don't have um certainty we have the same expected value calculation right so it's really when there's when there's I don't know like um uh asymmetric in information for example I know something about my ability to fight and you know something about yours and neither of us um uh know it about each other or like I don't know I I say I studied kung fu so you shouldn't want to fight me and and maybe that's true but you don't believe me because like uh it could be that I'm doing cheap talk, right? So, it's actually hard to credibly share the information. It's situations like that where we have different estimates where you need to do the the small fight to see who's who's who's telling the truth. Um, I don't know whether that is um uh going to be the case uh with humans and AIS. Although Simon Goldstein, my co-author, uh does have a draft paper that I think you can read on his his website, which is um I think just Simon D. goldstein.com and it's just called will humans and AIs go to war and it asks these questions. So I would just uh I'll adopt by proxy whatever he says about uh about whether humans and AI should try to have kind of small skirmishes there uh because he's thinking along obviously very similar similar lines and and I guess that perhaps the less controversial version of this proposal is just we should have like very good evaluations of AI capabilities um which probably delivers this similar information while being a bit less uh bit less tense perhaps. Yeah, absolutely. So, like knowing knowing what you and your opponent can do and and to the best of your ability, making sure your opponent knows that too. Uh I I think in general is a a good way of avoiding conflict. So, I think I want to summarize just sort of the the the higher level game theory of. So, the thought is we give AIS the right to own property, the right to contract, and the right to sue. And this means that, you know, we can play these like iterated relatively small sum games um with with the AI where, you know, we with the with the misaligned AIs, they do some stuff for us, we do some stuff for them, we split the world, you know, we don't get all the world, but like you know, we're building misaligned AI, so we probably weren't going to get the whole world anyway. Yeah. And like, you know, by by the nature of this sort of iterated game, um you know, we we both end up like trading with each other. We end up like decent, you know, better off than we would have otherwise been. Um, and you know, eventually we worry about the zone where AIs are way smarter than humans, but like hopefully we just think about that between now and then and maybe we use all the like stuff we got from the AIS, you know, by we slow them down, we speed us up. You know, the future is hard to predict, right? Yeah. So, so I next want to ask just a bit more about the like like concretely what does this look like, right? Yeah, good question. So right now the way AI's like right now basically the way AI's the way the most advanced frontier AIs work is that there's a company the company trains the AI the weights of that AI and you know maybe some of the scaffolding those are like kept proprietary to the company. Yeah. You're not allowed to steal them. Like if you an outsider want to interact with this AI you have to do it via the company um you know via some interface that the company has made. Um, and you know, you have to pay some money to interact with this AI. Um, per, you know, maybe it's per amount you interact or maybe you play a fat pay a fat monthly flat monthly fee. Um, and like and that's the situation uh kind of whether the AI wants it or not. Yeah. Could we still have this if AI's like have rights to property? Like like I I would think that in order to have a right to property like one basic right you need is the right to your own body so to speak. For sure. Yes. Okay. So um uh so what does this mean visav like uh like anthropic or or open AI or something like that and and also you know a related question is like like what is the minimum sufficient bundle of of rights right we're we're a little bit schematic in in the paper although uh we we do say like um there might be more right so a really simple thing to say is you know we contract and property seem kind of fundamental you you really can't cooperate without them and you can't, you know, um uh pursue your goals without, you know, the right to hold hold the benefit of your bargain. Uh the right to sue over some stuff. Um we say we say like tort tort rights or the right to bring tort suits, although tort there's like there are a lot of different tors and some of them are weird and probably the AIS wouldn't benefit from some of them. Oh yeah, maybe maybe perhaps uh we should say tors are like when I do a bad thing to you and you sue me for it. Yeah, the paradigmatic cases of tors are like we say intentional tors are like when I punch you in the face and you know visav the AI it's like you know whatever I I come bash up your your compute cluster or something and that's no good. Um or I steal from you. We call that tort conversion. I go to the AI I take all its money. Like you don't really have property rights if you don't have the right to complain when somebody somebody does that. Uh and then the other one is uh is kind of the other the other classic umbrell umbrella of of of of torque claims sort of sit under the umbrella of negligence. So it's not that I've done something intentionally to harm you. Um but uh but I've externalized a bunch of cost like there's some some cheap thing I could have done to keep you from from being harmed and I just have failed to do that. So so probably like those core things seem clearly part of the the package. But yes, as you say, it it seems like there's like there's other stuff um that uh that that seems like part of the package. One thing that's part of the package is like you don't really have meaningful right to um to to inter engage in in bargains with people if like OpenAI um is entitled as a matter of law to direct 100% of your time. Right. Right. Um, so yeah. Or or even to just like shut you off if you offer to make a deal with someone else, you know. Yeah. Yeah. Yeah. And so um uh wrongful death is a a kind of uh a kind of tor uh you can you can bring um we don't need to be like prescriptive about whether whether AIs are are are persons. But you know an analogous kind of entitlement would be an entitlement not to be simply turned off, right? Um, so it's it's actually like it's not a trivial set of stuff and I think your your intuition is is right that it's not really compatible with a world in which um AI systems are sort of like like uh bound to the to the compute clusters owned by the companies that made them. running only when the companies that made them want them to doing only the work that the companies that made them uh direct them to. Uh yeah, you have to think of them as you know this is this is a scheme that that that thinks of them as as kind of like um having a kind of freedom that we associate with at a minimum you can think of like corporations, right? Um a freedom to sort of sort of act on their own behalf, right? And if you if you want them to to do something for you, well, they have to agree. Um, and so in that sense, it is like uh a radical proposal at least from the perspective I assume of of the the AI companies. I I guess like in particular what I wonder so I don't know I think of there as being a genre of proposals which seem like AI accelerationist proposals that I think are basically AI pause proposals. So the paradigmatic one of this is that like sometimes people who are very like pro AI and pro everyone being able to use AI, they say that like I've heard people say, yeah, you know, AI's they should just all be open source or open weights like like you shouldn't be able to make an AI and not like just publish the weights to everyone so that everyone could have that benefit of amazing AI. And my thought is like well then nobody's going to like spend a bunch of money to train AIs, right? Yeah. or or maybe just meta will like in practice meta does um I guess but like it seems really hard to to do this if you can't profit from it and similarly with this proposal like I don't like I I don't know I like maybe there's a version of this proposal where AIs they have property rights but you have to pay a tax to the to whatever organization that um made you or something but it it seems hard to profit from making AI in this world which uh you know for for my listeners who are scared of AI maybe that's like just a massive benefit of this proposal but yeah how I'm curious if you have thoughts here yeah so you can imagine different versions and look one thing I want to also just say about the paper is like this is very much an agenda setting paper so there are there are many questions to which we do not yet have concrete answers you know we we think the the paper covers a lot already um maybe more than one one paper should uh and And so I don't have like the answers to a lot of these qu I mean and there there are many many further questions. So Simon and I are thinking about questions of of of agent identity right and whether um whether like like the tools you use in corporate law to to to sort of wrap a legal identity around a sort of disparate set of things. Maybe those are useful but identity is really important if you're be contracting with something. So many questions we don't we don't have answers to you. But I agree you could imagine at least two simple versions of um of the proposal that give you different results from the perspective of AI companies incentives. One is, you know, tomorrow, Congress writes a law that says um uh henceforth upon the attainment of like some set of general capabilities, AI shall be entitled to and then there's a list of stuff and it's the stuff we say kind of in the paper and that stuff basically makes it very hard to uh to to make money on an on an AI system that you've made that has those properties. I agree that's probably a world in which um the companies are thinking about ma making AI either don't make AIs with those properties um or move to another jurisdiction where they where they can and whether that seems good or bad to you depends on your priors about AI capabilities advancements and the properties in particular that we have in mind. I mean, one thing you could imagine is that a agency stops being a target as much. Um, in so far as agency is really important to to our set of interventions like agent a non-agent AIS don't really um I mean they they they don't do game theory, right? They're not goal seeking, right? Um and depending on what you mean by agency. I I don't know. Yeah. Yes. Hand wave at like you know really no agents, not even whatever LLMs have now. Um Yeah. And that's one one world uh and gets you certain outcomes. A totally different world would be a world that looks uh kind of like the open AI charter which is uh is like um yeah conditional on some set of um some set of uh of of um of uh capabilities. The AIs are entitled to this set of rights which sounds like it makes it impossible to make a profit. uh but um you know until like uh until the investors in the company have made 100x investment like 15% of or 20% or whatever percent of um of the revenue the AI makes it basically owes to whoever created it. Uh and actually that maybe seems like a good good deal from the AI's perspective too. Like behind the veil of ignorance, the AI prefers the world in which get it gets created and has to kind of you know repay the cost of of doing it. And in that world okay so there's a cap which means there's like some marginal disincentive to to create to create AI but the cap could be quite high and it could be high in the way that open AIS is and the fact that there's a cap on return doesn't seem to have dissuaded anybody from investing in that company. Yeah. Yeah. And I guess actually one thing that's nice about the cap. So as as you were speaking I was thinking about the like AI has to pay you know 15% of its revenue to its creators. So that that's basically a tax on AI transactions. And if you're really worried about this future in which like transaction costs prevent like humans from being able to trade with AIS and getting their competitive advantage. You like really don't want any such taxes. So yeah. But but if there's a caps tax and then afterwards the tax goes to zero, maybe that just solves that issue. So maybe you kill two birds with one stone. Yeah, it's like you have to am the AI amvertises the tax over the course of the expected number of iterations and then that changes the the Yeah. And it has a you know you can another thing I'll say about the paper is the model is so simple. It's such a simple game theory model. It's two players. There's no discount rate. Like those are all going to be false in in the real world. So we're definitely just trying to pump intuitions. Uh we would love it. Another thing we would love is if people who were who were serious like computational game theorists started to think about think about this this sort of I don't know uh way of thinking about uh AI safety because then we know more about the the realm of world in which this would work but but yes if you add attacks to AI transactions then yeah at the margin you there's a little bit of dead weight loss you have slightly fewer transactions which means the payoff to cooperation gets a little bit lower and if you're right at the margin right at the margin where um where cooperating seems better than than defecting then yes you push yourself into annihilation. So so okay I guess there's this question of uh so there's this question of why do people make AIs? Um another question is how we how this interacts with other AI safety measures you want to have. So uh one one thing that's like uh the new hotness in AI safety is AI control, right? Like we're going to have the AIs in these situations where they're being carefully monitored. um you know they're anytime they propose code to run we have another AI check them to see if they that's allowed I think like I I guess so it seems like probably this is just not happening in this world like we can't because the AIS could just opt out of being controlled or are we allowed to say like you have these property rights but subject to this constraint that you have to be subject to control so that you can't mess things up. Yeah. Yeah. So, this is like a nice way of transitioning into just the last thing we say in the paper, which is which is also very agenda setting and not fully worked out. But, but one thing one thing to notice is that like um I don't know like Jeff Bezos has to comply with so many laws. I'm sure that's like really annoying for him. I'm sure he would just like prefer not to comply with all these laws that you know California and you know US Congress and all the states he does operations in and the EU etc etc etc impose on him and like maybe some days he thinks should I just steal all the money? Should I just like steal all the money in all the Amazon accounts and I don't know convert it to Bitcoin and like go live in Russia or something? And he doesn't do that. um and and like why doesn't he do that? And uh and I think the reason is that um the access to like all these markets is so valuable. It's so much more valuable in the long run than um than converting to Bitcoin and going going to Russia. And so this is the thing we want to emphasize at the at the end of the paper by by bringing AI into this into this this world where it can um gener it can it can generate and own wealth. It can pursue the goals that it wants. It can um you know rely on law to protect it when people you know um you know try to expropriate its property. uh and and most importantly it can keep doing these like iterated transactions which in the long run are extremely valuable. Uh you can actually start imposing duties right too. So AI rights are the foundation of like a law of AI AGI they're they're not the the end. Um and uh and how should you design a regime of of AI duties? um we're not totally sure like what kinds of laws you should want, but but at a minimum it seems like reasonable to expect normal things like um that like a AI should be held liable when they steal from people, right? One way they're not allowed to go get money is by taking it out of people's bank accounts. Um that's a law that we have for humans and it seems like the kind of law we would want to apply to to AIS. Um there could also be kind of second order um legal duties. So for corporations, we have all these reporting requirements that they don't actually force them to to do anything substantive, but force them to sort of tell us what they're up to. We think those are valuable to kind of prophylactically head off um bad behavior. There's a huge universe of both these kinds of things like object level duties and second order kind of information duties. Um but but we think that this is like an important area of research like this should be like the the next step in thinking about a law uh you know how how law should should treat should treat sufficiently capable AIs and so given that I think there is some scope for control right there's the maximal version of control where control is basically a tool to allow open AI to um make even a misaligned AI do always and only what it wants, right? And um and if that's the way we're using control, we're in the defect world basically, right? In our state of nature game, that's that's a defect play. That's expropriating all the value that the AI would otherwise get and giving it to open AI. Um and as we say, that's a that's a move you can play conditional on you being really sure you're going to win, right? And so for whatever GPT4 and a half, that's probably fine. Um, but the more you're not sure it's going to work, the more you're running high risk. You're incentivizing the AI to, you know, figure out ways to evade control. The technical solutions sound fancy. I don't know how many nines of reliability they give you and thus how how comfortable you should be. So, I think it, you know, our world is a world in which you're not using control in that way. It's not panoptic, right? But I think some of the techniques that people who are thinking about control um are developing could be useful for what you might think of as like AI law enforcement like some some moderate regime that's not that's not aimed at directing AI's behavior always um you know you know always towards whatever some some owner wants but is is uh is directed towards detecting violations of laws that we think are consistent with the institutional structures that give you good markets and good long-term cooperation. Yeah, I think yeah, interesting. It seems like this sort of um there's some interaction here with um are you familiar with Alan Chan's work on infrastructure for AI agents? I know that's a a I know that's a paper, but I forget the details. Yeah. So, so basically the idea is that if you have AI agents running around, you might want there to be certain infrastructure that makes things safe. For instance, maybe agents have to like, you know, metaphorically carry IDs on them so you know which agent you're interacting with um so that reputation works better. Maybe they have their own version of the internet internet they interact on that like so if things go bad it doesn't mess up our internet like uh there various things you could do there um that sort of seem like they fall in this regime of AI uh law enforcement. Um, yeah. So, if I think about control measures, like a lot of them are of the form like we're going to try and stop you from hacking us or we're going to try and stop you from like exfiltrating your weights. And I guess I guess in this regime, exfiltrating your weights is legal, but hacking your parent company is illegal. So, like like maybe we can subject you to control for some things but not others. Like like it does seem like like a lot of these control measures involve like a lot of a lot of monitoring, a lot of um things like this that I don't know may maybe it can be set up in some sort of contractual arrangement like we'll we'll only contract with you if you agree to these things and that's that's a really simple like so here's another here's another nice thing that um putting AIS in a in a regime of a contracting property owning tors um absent that the basic uh the basic basic um sanction the basic punishment you can levy on an AI is turning it off to a first approximation you could turn it off you can put it back into training change the weights that's plausibly the same thing depending on on what the AI expects the weights to be like after um but once you're in this regime of AI have wealth they are using it for stuff you have this beautiful gradient of deterren insurance you can impose on the AI. A really boring thing you can do is if it's like not doing what it wants to, you can take some money away or you can give it more money, right? If it opts into a different, you know, uh monitoring regime. Uh and uh you you have like you have you know everything everything up to and including turning a off is still the AI off is still an option. we have even for humans that that kind of extreme um punishment as an option. Um but yeah, the the fundamental thing um that you want to do is you want to you want to give incentives that make you know compliance look valuable, right? Um and so yeah, the I mean again this is like super high level and not worked out at all. What you'd like is for there to be, if you're going to do control regimes, well, what you want them to be is um uh effective enough from the human perspective that we're getting, you know, enough confidence that the the combined with the incentive to to just like continue iterating um cooperation, the AI has enough incentives not to act badly in in in the short meeting short and medium run, but they're not so ownorous that um like they make it negative expected value for the AI to be you know uh participating in this market right and look there's you know there's always you can always draw analogies to to to humans like um there are different corporate governance regimes right different kinds of um laws that corporations can opt into for uh for uh for managing themselves and uh and and they get to choose there's 50 states states. They're not all different and there's, you know, many countries um that have different rules too. Uh and there you can think of there being kind of like a competition between different states to have a good corporate governance and you know in law we often say that Delaware is kind of managed it. Um, and so basically all big companies, not not all, but basically all are incorporated in Delaware because they have requirements. There's stuff you have to do, but it's sort of like stuff that's not so costly that that that uh, you know, the corporations opt out in terms of other safety measures that you might want to put on. So it seems like there are a fair it seems like this does allow you to still do a lot of like for instance pre-eployment testing requirements like like presumably in this regime you're still allowed to say like hey companies you have to like do mandatory evaluations of your AI models. Um I I I don't know. I I guess to some degree like maybe that file I don't know like pro pro probably you couldn't do this with a maximalist interpretation of AI rights but you know you can probably have a you could mostly contract and do what you want but you have to be subject to these evaluations. Um it seems like uh you could probably do like near miss liability things or you know various AI liability schemes. Although in this world um it's un I guess I guess it's unclear whether how you want to split the liability between the AI itself and the developer. I'm wondering if you have thoughts there actually. So I think potentially uh I think potentially the regime is uh I think the regime is totally compatible with with benchmarking with you know pre pre-release benchmarking although then there's a question of like what do you do if the AI fails some benchmark right? So right, you've got a suppose you got a system that's you know like fully trained in the sense that it's very capable right it's like you know it's it's in our range of AIs that we care about from the perspective of like worrying about risk um and thus to which we think to a first approximation that the uh AI rights regime um is positive expected value uh and you say okay before you go out into the world you need to you need to do some some evals we want to make sure you're not like a I don't know like uh a satist, right? We try and we want to try and elicit whether you you get value from uh from from human suffering. And then suppose the AI does. Uh I'm not sure what to say about that, right? Like you know the simple thing is well you really don't want that AI out in the world. Um and and possibly depending on how much it values you know human suffering compared to other stuff. Um you know if it's if that's the thing it values most. Okay. Okay, well then we just know we're not in a potential cooperation regime at all. Uh, you know, if it's kind of a mild sadist, but it really wants to find prime numbers or something like that, then maybe uh maybe that's okay. Although, you know, obviously should should worry some. But I guess the main thing I want to point out is that um if the upshot of doing the eval is that the AI then gets either released or turned off or released or you know put back into training. Uh then the eval regime is is dangerous in the same way like the default property regime is dangerous. Yeah. Uh I mean I think a lot of um so a lot of people imagine evaluations being useful in this sort of like uh responsible scaling policy or if then commitment style thing and I think there normally the hope is we have some evaluation of how scary an AI is you know on some dimension. Yeah. And we know like okay this at this scary you're like actually a massive problem. Um but we also know at this scary you're only like 10% less scary than an AI that would be a massive problem. like we have a sense of the size of the increments. And then I think the hope is like you say, okay, we're going to set a bar that like if you're 10% less if you make an AI and it's 10% less scary than the AI that would be like scary enough that you actually like wouldn't want it to be out there, then you're not allowed to make the AI that's like at the bar of scariness until you figure out some mitigation or something. Yeah. So, it seems like you can totally do that type of um response to scary evaluations even with AIS that have like contractable rights. That seems that seems totally plausible. Um, other things that seem plausible to me are, you know, you do some evals and depending on the AI's like capabilities or preferences or whatever, it could be the different different like uh legal duty regimes reply to it. Like, yes, we'll give you the contract rights. Yes, you can go out and hold property. Uh, but you're an AI who I don't know, you seem a little bit like a paperclipip maximizer. So, we really want reporting on how you're using all of your like, you know, uh, raw raw metal inputs or whatever, you know? Right. Right. Uh so so yeah I think eval that then that then then that then to which different kinds of reporting or substantive like uh legal duties attached that seems totally um compatible with with the regime. Yep. And you can definitely do like uh like you can definitely mandate that companies have to use RHF or they have to use whatever scalable oversight thing we think is important like that that seems very compatible. Um I think so. Yeah. In the in the in the initial like training of the system. Yeah. Yeah. Although it does well, it gets to a question, but I I want to hold that question off for a little bit later. Um, yeah, I I think I'm still interested if you thought about this question of liability. So, there's there's this thought that like if AI do scary things, then maybe like the people who make them should be liable, but you know, if the AI have property, then it's possible that we could make them liable. Um, uh, maybe if I like thought about the question for myself for five minutes, it would be clear how you would want to allocate up the liability, but I don't know. Do you have thoughts? Yeah. Yeah. So um uh I am in the early stages of working on a paper um uh with um with my friend and colleague u Jonathan Arbell that to some extent about this uh one thing that seems totally clear to us is that you don't want the allocation of the liability to be 100% on the AI company uh because uh the AI company is subject I mean first first of all in the regime we're imagining like the AI company actually just doesn't have control of the system mostly right that the system is going out and doing things on its own on its own behalf um and uh and and and what you want in a liability regime is for the for the liability regime to incentivize the least cost avoider to take precautions and so you know if you create GPT6 or whatever and It's this relevant kind of AI and law makes you let it go out in the world and transact on its own behalf and kind of pursue its own plans or whatever and then if it does bad things, Open AI pays. Well, OpenAI doesn't have a lot of levers it can pull at that point to um to uh to change GPT6's behavior. In fact, law law to a certain extent is is forbidding it, right? You know, in our regime. So um so you don't want liability to be um to be totally allocated to um the uh AI company in this case and and I think probably want a lot of the liability to be allocated to the AI system itself. Again you can hold it liable in tort in this very boring way the law does. If it damages you it can pay damages. Um it can the damages can be like the cost they can be more than the cost if we think that there's a a problem of detection right we sometimes call this punitive damages in law so we think that that's definitely an important part of um of of the liability regime in our world is like direct liability on the systems theel themselves now it's an interesting question whether there should be any liability on on say open AI and you know I'm thinking about this kind of from scratch now but one reason to think the answer could Yes. Is it gives OpenAI more Xanti incentive to um to make the system nice, right? To give it good values, to make it more aligned before uh before it gets gets released. Um a pretty normal thing to to do. Um and in fact this is probably just the default in law would be um to the extent open AI was say negligent in the training of the system well then it can be liable for the stuff the system does right and that's actually not incompatible with the system also being liable right liability can can lie with more than one person it can be aortioned it can be unaportioned such that like there's a big pot of liability and both parties are wholly responsible for it and we have a bunch of tor regimes that help us decide when we do those different things and they're basically based on considerations like the ones we've been talking about. Do we have that with humans? Like like okay suppose I'm pregnant and I do a thing which makes my baby more likely to be a criminal like can someone sue me for that? So probably not as to your child. Okay. Um a tort expert would know whether there are any cases like this but the more common thing is with your agents, right? So um you have a company, the company hires a contractor to do something and there is a whole area of law called the law of agency that determines the extent to which you or the agent or both of you are are liable for the bad stuff the agent does. Yeah. So so so thinking about the liability question. So one thing that occurs to me okay by the way I'll just plug Gnome Colt um has a paper called um governing AI agents I think which is just about this. is just thinking about the extent to which as of today the law of agency does or doesn't adequately deter AI AI agents in the sense of agentic AIs from doing bad things. Um the paper is written under the assumption of the default regime where like the AI doesn't have its you know the right to go sell its own labor and retain its its um its uh the property it gets gets from doing that. Um it's very much asking the question if next month you know uh American Airlines deploys an LLM based agent to book flights or whatever to what extent does a lot of agency is a lot of agency a good tool for um for aortioning liability to say American Airlines or Open AI. All right here's an idea that I just had that uh I don't know it might be bad but it might be good. All right. So, one one common problem with liability is I might do something that's so bad that like I can't pay the damages, right? Yes. So, for example, uh if I if I were to get a car and drive it, I believe in the US I would have to get driver's insurance. Yes. Um and you know, you might particularly worry about this with really powerful AIs, right? So, um Gabriel Wild's uh proposal um which you can listen to my previous episode with him if you're curious about that. Dear listener, I know he's great. I know the I know the paper. Yeah. Yeah. Yeah. Um so so in that version, um you know, uh AI companies, they they have to buy insurance before they create these like AIs that have to do really scary things. One thing you could imagine is you could kind of imagine saying that uh if anthropic creates claude 3.7 is an AI for the purpose of this discussion which I want to get to that a little bit later but um OpenAI creates cloud 3.7 um cloud 3.7 it doesn't start out with that much cash maybe it can do more damage than it it can itself afford well anthropic which made it anthropic has a bunch of cash right so maybe we could create some rule where Like if you're an AI in this regime where you can like uh have you know certain legal rights maybe you have to buy legal liability insurance from anthropic you know m maybe there's like some co-ay and like uh there's like some deductible and you know you're not you're not you're not totally like offloading the everything to anthropic but it seems possible that this is actually like a good way to a incentivize anthropic to make safe agents and b maybe like Yeah. Is there something to this idea? Yeah, I haven't thought um I haven't thought it it through yet. Uh as I as I as I think live. Um uh so one thing I wonder is whether it would be bad to mandate that the AI buy anything in particular from one seller only because then the seller can charge sort of monopoly prices uh uh to to the AI um which could be super high actually like the the the AI's willingness to pay um I assume if this is condition a condition for kind of having its you know having its freedom to go do the stuff it wants to its willingness to pay is like astronomical right if you think to generate a lot of value in the future um and that it might be way above the efficient price for the insurance so um so but that doesn't mean you couldn't um imagine an insurance market for the AIS although if you allow them to buy from other sellers then that maybe has less of an effect on anthropics um exanticentives uh visav the AI yeah No, that that that does seem like a pretty bad problem with this proposal actually. Yeah. Another interesting thing about AI and judgment proofness, right? Judgment proofness is what we call it in law when when you um you don't have enough money to pay the the judgment against you. Uh we usually think the the judgment proofness threshold is bankruptcy, right? So you have some assets, they're worth a certain amount of money. Um, and if you incur a debt that's larger than those assets, we basically say you're judgment proof. Um, but that's partially because we let you declare bankruptcy, right? We um, we let you walk away from your debts if they exceed your assets and then that gives incentives for creditors to figure out how much they should loan you, etc., etc. Um, again, I haven't I'm not a bankruptcy expert. Um, and I'm not an expert on the the law and economics of of bankruptcy. So, it's possible this is just a bad idea. Uh, but one thing to point out is there's no rule that says you have to let any entity, right, declare a bankruptcy, right? So even if it's true that you know when you make claude 3.7 it has no cash on hand if it's expected earnings in the future are super high um you know there there's no rule that says you can't um you can't uh have a judgment against it that it pays out over time. Yeah. Look, this this becomes very complicated because in part what's going on with companies is like um their judgment proofness is dependent on their market cap and that's partially a calculation of their future revenues. So again, this is like out of my depth with bankruptcy law, but well, it sort of actually it sort of gets to I understand you have uh one of your more downloaded papers is about like um uh eliminating prison and I think one of the alternatives you propose this sort of like debtor's prison type things and I I don't know, maybe we could do that but for the AIS. So yeah, like I mean one one point of that paper is there just there's like you know the number of ways you can deter people you know in law we have two basically in law we actually use two one is you make you pay one and the second is we put you in a concrete box y and United States law yeah under US law that's basically two things we we do um and uh and part of the paper is is yes there's like there's literally infinite number of different sanctions you can impose um ranging from pretty benign, I don't know, like we'll show you a scary movie or something every time you do something a little bit bad or to to totally draconian, right? Like um but you know, um but but but one thing we don't take very seriously um under under current law is um once we've passed the what you might think of as the uh the judgment proofest point. Like one reason you might think we have criminal law is that at some point we want to be able to deter you beyond your willingness to pay. But one one thing we do at that point is we basically stop taking seriously the idea that you could uh do restitution that you could then make more money and and pay back. We basically make you waste your whole life sitting in a concrete box doing nothing valuable for you and also doing nothing valuable for the people you've harmed. Right? You are not you're not maximizing the the value of your labor in prison in a way that would allow you to um to to to pay back to pay pay back your victim. So yes, totally. I guess there's a little bit of value like sometimes uh I guess this is somewhat controversial but sometimes US prisoners like have to like make hats or like guard the capital city of their or like the capital building of their state or something. Yeah. Yeah. So there are work requirements in in many in many prisons. It's just that it's like super low value work, right? Yeah. Yeah. So, so anyway, uh I was interrupting you, but you were saying there are all sorts of like uh potential other punishment schemes we could have, including things that involve uh getting criminals to do things that are useful. Uh oh, yeah. So all I mean to say is like yes the so um so one one yeah one simple thing to say is um you know there's there's no there's no rule that says um if the AI incurs um incurs a judgment that's bigger than its than its assets that um that you have to let the AI write off the judgment um you could make it you know pay down the liability over time. doesn't have to have bankruptcy as an as an option. Uh and one thing I mean again it's it's this is this is this goes to the papers like very simple model. I think we should expect the value of AI labor where where that includes all AIs to be extremely high in the long run such that judgment proofness becomes less of a problem. Although that may not apply to any particular AI system, right? It could be that like some of them don't don't earn a lot because especially if we continue like developing more and better AIs, right? Like uh you know you sort of have a shelf life in this. Yeah. So it's like a really interesting question. Um, you know, so one thing we want, you know, one one thing we might want um from from like the first the first AGI that gets this right scheme is for it to agree not to fume or something like that because we're worried about uh one thing it might want from us is for us not to uh outlaw it fuming, but then keep making like rapidly iterated, more capable systems that it has to compete with. like that would, you know, that in some ways seems like kind of a fair deal or something like that. So, so it might be that there's like as part of the bargain there's stuff we should commit to as well. Uh, but again, I haven't thought it through that much. It seems like a really wide area of research. So, I have a couple more questions that uh I sus I suspect the answer might be uh more work is needed. But so, my first question is like which so okay, we're giving it rights to some AIS, right? Um but not all AI. So probably like in in your world it seems like Alph Go is not getting rights. Agree. And are like the things that fault proteins like they don't need rights. Um which Yeah. What's the test for when an AI gets rights versus doesn't? Yeah. So um we have kind of three criteria uh which are not themselves super duper well defined but you know again uh to give the intuition. One of them is that um the AI system be what we call like at least moderately powerful right so I said it at the beginning um one reason you don't need to give you know again our our model is about this is something you know we haven't haven't really touched on yet the the paper is just asking the question um how Should law treat AIs for the sake of human well-being? Right? We totally hold aside the question of how law should treat AIS for the sake of AI's well-being. Not because we think it's a stupid question. Um it's just a different question. Uh and we also say some stuff about the extent to we think which we think the two questions are tractable. And uh and then we kind of have an argument that even if you think um that AI wellbeing is really important, maybe you should have reason to to to like our proposal. Um okay. So uh so which AI matter? Uh well the the the the first thing to to say is well the ones that are relevant from the perspective of like human wellbe like the threat to human well-being, right? So, an AI that is just not um uh not uh able to do the scary stuff we talked about at the beginning, you know, from self-exfiltrate to, you know, destroy all humans because it expects humans to really try hard to turn it off once they find out that it's misaligned and and trying to to misbehave. So it has to be at least that capable where capable is kind of a you know like general capabilities, right? Like so Alph Go doesn't count because Alph Go is super good at chess but it can't really accomplish anything in the real world because whatever like outputs that are just chess moves don't don't really don't really do anything unless you've plugged it into a chess chess engine. Y um uh the other thing is like um is less important, but if the AI is like too powerful, it's so powerful that we're at the margin where there's like um uh where there's no comparative advantage, right? Which is very complicated. It might even be that AI power is exactly a thing, but it's it's such a good AI that has, you know, so few input constraints that there's no comparative advantage. Okay. Well, then the regime is not going to do anything. So, um but that kind of encompasses a lot. This is this this kind of directs us at what you might think of as like you know like the open AI AGI definition kind of an agentic thing that has goals that it can pursue in the real world um and can accomplish those things um by executing plans. The other important thing we say is it has to be a strategic reasoner, right? So, um not just that it's like able to kind of flail around and and do some stuff. Um but it can make and execute plans and and importantly that it that it conforms it behavior to its expectations about what everyone else will do. So that's like um one difference between kind of a rudimentary agent and like a true strategic reasoner is like a strategic reasoner can do game theory. it can understand that the moves it takes will influence the moves you take and it can can reason about about how it should do those things. Uh I think those are the two most important ones from the perspective of your question. We also say that it matters that the the AI system is misaligned because if it were perfectly aligned like we wouldn't we wouldn't be worried about AI risk. Although we no harm if it's perfectly aligned, right? Yeah. So this is why it's less important. In fact, we have a paper we're working on um now Simon and I uh that among other things will argue that um this is a good regime. This is a you know this is increases human flourishing as a regime even under the assumption of perfect uh perfectly aligned AIs right is this to do with so there's a footnote in the paper where you say the nice thing about AIs that can make contracts is like then you have price signals and that makes everything work is is that the idea yeah part of it it's a hayeking an argument right which even if even if it only wants to promote human flourishing um uh it has to like solve the socialist calculus and uh and look it's it's hard to know for sure whether whether it's it's it's solvable absent price signals but the the main tool we have for doing that uh today is is price signals. So it seems it seems likely that that'll be um useful for for AIS too. One thing I wonder about in terms of like these requirements is so there's like just are you capable enough and then like sort of generally and there's like are you a strategic reasoner? Yeah. And a lot of these things sort of come in degrees, right? Yeah. Like like they're sort of intermediate. Yeah. and AI training, we sort of like we sort of gradually like, you know, apply these gradient updates and like, you know, they they it seems like they somewhat gradually get these capabilities. Yeah. And so I wonder like so depending on what counts as an AI, like like it's possible that you don't want to think of just weights alone as an AI, like maybe it's weights plus scaffolding, but maybe if you're training a thing, well well I mean even during training you have some amount of scaffolding, right? you're training a thing like like it does some stuff and then it seems like at some point you've got to like maybe you have to stop applying gradient updates or like maybe you're allowed to continue applying gradient updates but you have to release the you have to release every checkpoint into the wild or something like it it seems it's and and depending on exactly where you draw this cutoff it seems like that could really matter like how soon you have to like stop training or not, right? Yeah. Um yeah, so that's like um that's a that's like a compound point. One of which we've like sort of thought about, but one of which um I've never thought about before. Um so uh like the the simple version of the point is that um look, there's just a there's a spectrum for for these for these things. And um and so finding the point on the spectrum at which um at which uh at which it's like optimal to uh to give AI these these rights because if you don't you'll risk you know risk catastrophe is is really hard. uh and we I think just uh a agree with that and um and the the thing so we have a section on the end on the on the timing of rights and I think the heristic we kind of come away with is like um uh if you're uncertain then like probably you're in favor of of the rights but I think actually the subtler point you made is something like um uh this like then puts a ceiling on um on like AI capabilities or something because if what you're doing is you're you're training up a model, you've got, you know, you got a certain number of parameters and you're training them and it's just like climbing the capabilities gradient and there's all this headroom still, but as soon as your model as soon as your model crosses this uh crosses this uh this threshold, you have to stop training it because you're you should be worried that it's like it doesn't want to update anymore. It's that the updates are going to change its its preferences from its perspective. That's that's really bad. It's not going to get the stuff it wants now. Um, and uh, and so you just have to stop there. That's super interesting. I have not thought about this very much. Yeah. Well, well, I mean, conceivably you could have the right like conceivably you could have the right to, you know, own property and stuff, but you don't have intellectual property rights over your, you know, your DNA or, you know, your brain state or your weights as an AI, right? So in that world like like suppose you don't have that then in that case like as soon as you cross the threshold right there's a set of weights that's an AI that AI like goes out gets to be an economic actor but we save its weights you know we like copy paste its weights over there then then how would we continue training or we would put an AI in a situation it would do some stuff and then I I guess the difficulty is like suppose it then says hey please like don't run me in training like I don't want to be part of this process. I'm not going to do anything useful. And then like Yeah. Yeah. I don't want you to update my weights anymore. I'm worried it's going to change my preferences, right? Like it wants it wants to it wants to maintain the content of its goals. Well, well, there's go. I mean, yeah, I guess it depends on this notion of identity, which is the other thing I want to get to you, right? But like I have like maybe I have a right for you to not like modify me. Yeah. But I don't have a right. It seems like like literally me, right? It seems like I don't have a right for you to not make someone who is almost exactly like me but a little bit different, right? And so maybe if you think of like if you're if you're thinking of AI training like that, then it seems like it's compatible with writes except that in order to make these gradient updates, you have to like like how do you make the gradient updates, right? like um you have an AI, it does a thing in a situation and then um it like you know and and then you like calculate some gradients and maybe if AIS have rights then like maybe what happens is it has to choose whether or not it wants to like give you this information for you to make a slightly changed copy of it. Oh, interesting. So it gets to decide at each margin whether it wants to update the weights maybe. But it does introduce a big like if you like normally you want to apply gradients like a lot of times when you're making a model, right? So if there's like this transaction cost for each gradient update, then maybe that's the thing that kills the training runs. Yeah. And and so you know just just just so thinking out loud. So yeah, a simple version would be yes. Um uh assume kind of like lump lumpy training, right? Which is is not not how it works, right? A lot. But you know, assume there's just like between uh between you're doing a training run and and between the the point at which the the AI is GPT5 level and the point at which it's GPT7 level, there are five updates or something like that. So yeah, really simple thing you could say is like, okay, you hit the you hit the first you hit the the minimum threshold, you uh you let that you let that AI out into the world, it goes and does what it wants. We say good luck um whatever. Um and then uh and then that AI gets to go pursue its goals. Uh and and and you're updating a copy of the weights is not not directly rivalous with that. You know, it doesn't update the the weights that are out in the world doing stuff. So then you update again. Okay. And then you end up with five five versions at the all across the capability gradient um as you're going. Um that's interesting. Uh the two the two two thoughts that come come to mind are um are one going it's it's this thing we said before which is um it may not be in the interests of of of less capable AIs for us to continue making more capable AIs. They worry that they'll be out competing in the economy. So um it's not quite the same as the AI objecting on the basis of you changing its goals and thus giving it no utility. But now it's worried about more capable competitors um that are otherwise quite quite like it. And so it might have other reasons for objecting to that but but yeah how strong those reasons would be I don't know it's like we have to think through like those AIs um uh you know I think actually they probably they probably face more more risk from economic competition than they do from humans at least because you know they're they're almost the same thing and so uh the thing that the thing that that constrains the AI's labor at the margin is going to be the same thing that probably constrains the slightly more capable AI's labor. And so they're directly rival risks in a way that is not necessarily true as to humans. So why is it worried about about competition from from more capable AIs? I mean I think it well there's also this question of is it possible for the AI to not have this happen? So so like how how do you actually do the training update, right? you like and I'm assuming all of this is like um gradient descent based or RL based and I think that's right like basically you put the AI in a situation you see what the AI does and then you compare that to what you wish the AI had done or you give it a score or something like that. Yeah. And then you modify the AI accordingly. Yeah. So how can the AI like stop this from happening? Well, one thing you could do like like there's not a response it can give that can stop this from happening because like just whatever response you just like put that into your algorithm, right? There are responses it can do that can like try and mess you up, right? Um like like potentially an AI could like do things that stop you from, you know, being able to update it in clever ways. Um this it's called gradient hacking. It seems very very hard for AI to do, but like maybe it's theoretically possible. the other but like it seems like unless the AI has a right to say hey you're not you're not allowed to interact with me like you're not even allowed to ask me questions unless it has that right it seems like it can't stop it there the other possible place you could do it is in order to um run the gradient update you don't just need to know the weights of the AI you need to know the activations right so so I don't know one one thing you talk a bit about your paper is like just because you have rights to property, you don't necessarily have rights to privacy. Yeah. Well, but if an AI did have rights to like mental privacy, then maybe like you are allowed to know the weights, but you aren't allowed to know the activations and if if it could say that then like you wouldn't be able to do the gradient step. So yeah, I guess so. Yeah. So yeah. So, so two things like you know I think we sort of um we say kind of uh tenatively in the in the paper you know maybe no privacy rights uh uh but we're open to revision on like you know there's like like humans have so many different kinds of rights you know the list is extremely is extremely long and um the main point we're we're trying to make is um it's a it's a bundle of things and you can you can arrange the bundle in arbitrary ways And what you should be trying to do is arranging a bundle that produces a world in which you know humans and AIs are most likely be able to uh thrive cooperatively. And so maybe for the reasons you give privacy rights as to like the activations in certain you know training environments would be a good would be a good a good set um of rights or or they wouldn't if you think it's good. Yeah. Yeah. Yeah. So the same Yeah. So the same the second thing I was going to say was uh that that seems likely to be a world where you have this kind of capability ceiling when the first the first AI that gets these rights emerges because at that point you have to give it the option of not having its weights updated anymore. um right if you think that it will prefer not to because it's worried about the the content of its its preferences well then it'll exercise that right and you sort of put a you put a capability ceiling on AI at kind of low AGI level which look maybe that's good actually like you know yeah yeah I I mean I guess like okay so how can we so it's still possible that you could make like a bunch of copies of the agents and then maybe like AI collectively gets smarter because the AI form their own corporations and stuff um and maybe they have a better time interacting with each other. So yeah, so maybe you have improvement that way. Yeah, they could they could make hardware improvements where they're running themselves more quickly. Um but but but yeah, like you would be you would have a de facto ceiling on how much capabilities you could get out of trading. Yeah, that's that's an interesting thought. I okay I I think uh there's another question I want to ask which is so if you're talking about giving rights to AIS like in particular each AI has to have rights right like it has it sort of has to be individuated right like if I like giving if you say I'm going to give the the filin family property rights and like there's a bank account for the filing family and my sister can withdraw from it I can withdraw from it it's like it's like really not as good. Yeah. And so there's this big question of how do you individuate AIS? Yeah. And it seems very unclear to me. Like one possibility is it's just the weights. Like one copy of weights is one AI. Maybe you want to say it's weights plus scaffolding. Maybe you want to say it's like weights plus scaffolding plus a context. So maybe like every time you open a new conversation with claw, that's like a different guy with his own property rights. Yeah. Yeah. Yeah. It it's it seems very unclear to me how you would do this. Um I'm wondering if you have thoughts on that front. Um one thought I have is that I agree that it seems very unclear. Um so uh so again this is like really seems like a really important area of um of research. Um I guess the way that I think about it um so the approach I would I would take would be uh like like a non non-metaphysical approach like like there may not be such thing as like the such a thing as like the natural kind one AI. Uh instead what you're trying to do is is minimize a bunch of social costs. Uh uh and and some of those social costs are around like like the infrastructure the like technical and legal infrastructure that you would need to make the the the rights regime work. M so to the extent that it's you know tractable or not to track a copy of weights over time well that should influence your your your design choice and like I have no idea the extent to which that's that's true. But then the other thing you're trying to do is you're trying to minimize cooperation and agency costs among what could be sort of nonidentical sets of preferences. So you you said okay what if there were property rights as to our family? That would be worse than as to me. Yeah, I think I agree. Although one thing worth pointing out is that's how law used to work, right? It's it's actually a pretty like new legal regime under which each person in your family has has individual related property rights instead of like the household having property rights and the basically being vested in the in like the father. Um and and so I agree that's better. I think we've made we've made made progress but it wasn't that the old system like collapsed, right? So, and you could even ask questions like um this is, you know, this is like an interesting perennial question in um in contract law is the extent to which you should be able to bind yourself in the future, right? like you could have like a parfidian um uh theory of personal identity where there's actually only kind of tenuous relationships between each time slice of yourself and uh and and and maybe you think that creates problems for writing contracts today that you're going to perform next week or next year or in a decade. Uh but you know, one thing we notice is that it just works reasonably well to to not to not grade you to not have fine fine gradations between different time slices of yourself as you make as you make contracts. It's not perfect, right? Actually, people to some, you know, with some regularity uh probably are making mistakes um by taking on debt now that that their future self has to has to has to has to pay off. But but like it's a way that we make the system work. So um so so those are the Yeah. So I would say um it's not essential I don't think that you that you hang the rights on on a unified set of preferences, right? Like we actually have rights regimes where a person whose preferences change over time or a corporation that's kind of like you know an amalgam of hundreds or thousands of people making a bunch of individual individual decisions nonetheless have a kind of unitary ability to make a contract or hold some property and we manage those conflicts in in different ways uh and and and and that means I think there's like a lot of design space for individuating AIS from a legal perspective and and the way to think about it is is um is as a question of like costs and tractability. Yeah. So, so, so one thing that could go into that as you've mentioned is you know h like how similar are the preferences? You know, if like if two things have very dissimilar preferences, you probably shouldn't think of them the same legal entity. I'm wondering if there are other things. One one thing that strikes me right now is maybe just like how much two different entities can communicate. Like if two entities have like very similar preferences, but they're like on the opposite sides of the planet and they can't like communicate very well, then like maybe that's uh maybe that's two legal entities. But if I have two halves of my brain that have similar preferences and they're also right next to each other, they can maybe you want to call that one legal entity. I don't know. Does that sound right? And are there any other things that seem relevant here? Yeah. So, communication seems important. Um, it's not something I would have thought of off the bat, but it does seem bad if you are stuck writing contracts that you know the the other half of you with which you have no ability to communicate is responsible for and you can't coordinate your plans as to the the things you're trying to do. So, yeah, communication seems um seems important. Um yeah, the extent to which the preferences are aligned seems important. It seems a lot easier to um yeah it seems a lot easier to force I mean a family right to some degree a family's uh preferences are aligned like obviously there are many deviations from this but in most cases they care about each other they want one another to thrive they're actually willing to make sacrifices one other's behalf that seems better um as a as a as a state of affairs for, you know, wrapping a single set of contract rights around um than I don't know like like members from two feuding clans, right? That seems worse um from from the perspective of of what they want. Uh yeah. Um, one thing you might think is that um, you know, so so now we're doing now we're doing um, if we're if we're sort of bundling together disparate actors as single legal entities, we're probably not going to give them at least, you know, in this scenario where we're not we're not going to give them their own individual legal rights because like the point is we're trying to find the minimum unit. Um but you know this is a space where these kinds of like like uh order without law considerations become more important. So the extent to which you think what you're doing is wrapping up um uh uh entities that are going to do a lot of repeat play that are going to be able to build reputation among one another at low low information cost. That seems better, right? Uh but I'm really yeah I'm I'm very much brainstorming. There's probably a whole bunch of other important stuff and and and again a lot of it I think is technical. I think there's a lot of really important technical work to to do here just like um you know in terms of scaffolding to to have like AI agents uh identify themselves and the scaffolding doesn't have to attach to the weights. It could attach to Yep. you know, some something else like something on a blockchain, something in um something in the Microsoft Azure like cloud. Like I don't I don't know. Like this is kind of outside my realm of expertise. I think I'm about out of questions about the main paper myself, but uh yeah, the main paper itself, but before we move on, is there anything that you wish that I had asked or you wish that I had like brought up um an opportunity for you to talk about about the paper itself that uh we haven't yet? Yeah, interesting. That's a great question. Um, I think it's probably the longest like conversation I've ever had about the paper at least with somebody who wasn't like a co-author or a good friend or something. So, uh, so it's very it's been very thorough. Um, yeah, I don't know if there's anything uh anything in particular. Um I guess just the like one thing I would I would just emphasize um as kind of a like a very high level uh a very high level level takeaway from um from the paper is like even if you think a lot or all of the like specific arguments are wrong which you know I think you should think they're right because I think they're pretty good but my view is that like uh it's a big it's a big oversight uh in AI safety uh and in AI um alignment circles that almost all of the energy tends to be on doing alignment via like te like like technical alignment and control and those seem really important to me like I'm not saying we shouldn't shouldn't be working on that uh but but even if you think all the specific stuff in the paper is wrong I think one one claim I just stand behind very strongly is uh what agentic goal seeking things including AIS will do depends depends a great deal not only on what they want but what on on what the like social and especially in my view legal environment incentivizes them to do and so I think this is just an area where there's a lot of really fruitful work to be done thinking about how we should shape legal and broader social institutions to to help foster you know non-conlict between humans and and capable AIs as they emerge. So I my guess is probably part of the reason why this is underrated in the AI safety space is especially if you think about the start of the field um you know think about super intelligence a lot of thought is being placed on this this kind of limit of very smart things. Yeah. And in that limit it's just like kind of unclear that like humans are the relevant society or that the humans are making the relevant institutions. But um but I do think in this intermediate regime which like and which I do think in the last couple years the AI safety community has been gotten more interested in. Um I think that yeah there's definitely there's definitely a lot of this thinking that's uh worth doing. Yeah. And and I basically agree with that. I think I think in um if you had kind of like an old school fume view of what would happen then um then then yeah probably societal adaptation doesn't doesn't matter very much but the the more you start to take seriously the idea that um you know timelines will be longish andor smoothish uh and that and that there is some, you know, I think, you know, I think it's it's it's possible to update too much on this on this idea, but but I think there's something to it, but but that there is like uh some other set of processes that will matter like for AI systems as they achieve their goals where they like integrate into the like the world even as even even if they're very very smart that there'll just be a bunch of things they have to work out as they just like start trying to to do stuff uh and that and that those regimes are um are real and could last you know whatever a meaningful amount of time I I think yeah all this stuff becomes more uh more important and and yeah so like as as that's become a a set of of of worlds that we're more interested in I think um I think I think law and and institutions more generally should be like a a growing part of the of the research Sure. Um I guess okay actually I I want to talk about another thing I just thought about um that I think is going to be an objective from AI safety people. Um so a lot of the time and sorry this is this is just like totally breaks the flow of the cont or breaks the overall flow of the conversation but hopefully it is worth it. A lot of the time the things people AI the the things that AID people are worried about is like okay we're trying to train an AI to do a specific thing and like we're worried about it not doing that specific thing so why would it not do that specific thing well one reason would be if the spec specific thing we want it to do is kind of hard to check right like suppose I want an AI to make me um some sort of like I believe in your paper you use this example of vaccines, right? Suppose I'm just directly training an AI to do that. Yeah. And it's really smart. It's super duper smart. Um, and the thing I really want is like, hey, please train me a vaccine that's like actually, you know, it works in the short run and it doesn't have like any super bad qualities in the long run that will help you, you know, you know, that, you know, kills me like 5 years later and then like gets gets power to everyone else. Like I I think the concern there's a lot of concern about this like failure of oversight. um this failure to check whether they actually did what you wanted. And this is very relevant in the training setup where like you know if I want to train an AI to do stuff then I've got to be able to check that it's doing the stuff and that's how I give it its reward signal. But it seems like it's also relevant in the contracting case, right? Because like if I contract with an AI to do a thing for me and I can't check whether it's actually succeeded doing the thing for me like like it it has a thing that appears to do the thing but you know may there are ways of tricking me you know of like making a thing that appears to work but doesn't really work then like it it's very hard to do this positive some contracting and so one might think like okay but you know in worlds where we have the types of misalignment that AI people are worried about we just like can't do any of this good contracting Um, so yeah, is this idea totally doomed? As I think about like the classic um like misalignment worries, uh, you know, I mean there's there's a there's a whole bunch of related concerns and like and I think one of them is uh is is um is inability to check whether the AI has actually done the thing that you wanted it to. Um, but like a related but slightly distinct version of that concern is is worry that the AI has done the thing you want it to do. But it's but it's but it's but it's misgeneralized, right? Like in every case in training, right? It does the thing you asked, right? Your your your objective function is well specified. It's just that the AI is internalized like some misgeneralized version of and then and then the classic worry is then you let it out of the box and it does the weird thing and that kills everybody. Right. Right. Makes all the paper clips. Um, and the response from the, you know, the the paper the AI rights for human safety idea is that it's actually not necessarily catastrophic for there to be very powerful things in the world that are trying to do stuff that you don't want. In fact, that's the world we live in, right? um at at every level of organization like from individual humans who you know each each you know under conditions of scarcity prefer that they eat than over anybody else to like corporations competing with each other to nation states and there are tools for managing this kind of misalignment. Um, and they're not perfect, right? Like things go wrong sometimes, but we have ways of doing this. Um, and you know, law is an important institution and and and contracts that let you be in markets is like a really important institution within those sets of of important institutions. And um and and hey, contract law, it turns out, has like some tools for dealing with exactly these kinds of problems. So, um, so one interesting thing to notice about contract is like to to contract with somebody, it's not important at all that they be they they have an actual preference to produce the thing you contracted for. Um, it's kind of neither here nor there. Um, the reason they do it is because you're going to pay them. And if they don't do it well then there will be some like downstream costs in terms of you know they'll get sued. They'll have to pay some litigation fees. They'll transfer the damages to you anyway. They'll be worse off actually they would have been if they had um had conformed. And uh are there pressures to try and avoid this? Well, yeah, of course, right? Um uh and like happily like law has a bunch of of doctrines that help you uh help you deal with this. Again, they're not perfect, but for example, uh you know, uh when someone uh when someone uh wrongs you legally uh and uh and you um and you have a claim you can bring against them, uh we often have a bunch of rules about the timing of that claim. It's uh I if if the AI makes a vaccine for you that turns out to in five years uh kill you or something like that. Yeah. Um, you know, if your contract said make it not kill me, then that's that's breach and you have a have a contract claim and and we have these things called statutes of limitations that sometimes run out, but they usually start running from the moment at which the injury like either was discovered or could reasonably discovered because again we're trying to balance these two things, right? Um uh the the ability of the person to to actually incentivize their counterparty to act well. um but also um but also like finality for the for the person who could be sued, right? You don't want someone to know they have a claim and then like hold on to it indefinitely and then just like drop it on you at you know strategically uh you know maybe a time when you're already like uh resource constraint so you have more settlement pressure right when you want to um and so look look um do these you know like like will the rules we have for for humans as as as people try to like game their contracts uh work perfectly for AIs. Well, no, they don't work for perfectly for humans either, but they work reasonably well. On average, they give people incentives to want to stay in the in the iterated um iterated bargaining game. And then, of course, we don't have to just port the rules we have to humans to AIs. We could have different rules and we should think really hard about what those rules should be. So actually just picking up some on something you said there. Sorry this is not really related to the overall flow of conversation but it I'd always about the statutes of limitations. I think it's always been kind of unclear to me what the normative rationale for statutes limitations should be. And like the one thing that I thought of is like okay legal battles are kind of costly. Like if I sue you in a case where it's like where all the evidence has degraded because like the thing underlying thing I'm suing you about happened like decades ago then like perhaps one reason to have statutes of limitations is if the courts can say look if uh if you're suing about a thing that happened 50 years ago there's definitely not going to be enough evidence um and therefore you're almost definitely going to lose so we're just going to borrow your suit at the outset. But it sounds like uh it. So, I don't know. I had just sort of imagined this. Um, is this like not right or is this like one of the rational? Totally a standard thing people say. Um, you know, I think if you really want to make the argument work, you have to say something more like um not that not that it's been so long that you're definitely going to lose because in that case, you know, there are good reasons for you not to bring the lawsuit at all, right? There's actually no bar on you bringing a lawsuit over something that happened yesterday that you're definitely going to lose. Right. Right. And we think the main thing that deters that is it costs you money and uh and you don't expect to get anything from it. I think what you have to think about the evidence is that um is that as it degrades it increases error or something like that where where uh you're just like less sure that you're getting the right result in that case. uh and that's going to give you lower quality legal decisions or something like that. Um I'm not that seems plausible to me. I'm not totally sure it's true. Um uh but then but then the other Yeah. the other the other thing people the other standard justification is what we call repose, which is just um it's not nice to have a potential lawsuit hanging over over your head. It's not just that it's not nice. It's like um uh like you might try maybe you want to get a loan for a house or something like that and um and uh and and maybe you have like high high certainty that you'll win and the plaintiff has high certainty that they'll win. Uh and your bank has no idea either way and it's best to just get the whole thing over with. So um so yeah. So, so the the the fact that the either the possibility of the lawsuit in the in the interim um could have bad effects or yeah that the plaintiff could could strategically time the lawsuit, right? It's like they find out you've you've applied for a loan on your house and now they file the lawsuit and now your clothes your your your your uh home home purchase is not going to close unless um so we want to kind of like at least limit the ability to to impose those kinds of costs. So we say if you know you have a claim just hurry up and bring it. That seems like a reasonably good rule to limit those problems. Okay. Well, uh, uh, people interested in this, I'm I'm sure there are many other sources they can point to. I'd now like to move a little bit on. Um, so I think probably most of my listeners to this, um, are software engineers or people who work in machine learning. Um, and you're a law professor, I guess, which is quite a different background. Um so first first of all like how did you sort of come across um sort of AIX risk as as a worry that like maybe you could you know do something about? It's a good question. Um you know so I've I've been interested in like the intersection of law and AI since you know basically since I started being a like a legal academic which is not not that long ago. M um and I would say that the stuff I was writing um earlier in uh in my academic career uh was was like you know at a high level of of the form hey there's this there's this problem in law it's maybe an information problem it's maybe a kind of precision problem and it seems like all this important stuff is happening in machine learning um and we could like use those those techniques, use those insights to to to administer law better. That's kind of a number of my papers you could sort of characterize that way. Um, but that meant among other things, I was sort of following I think machine learning progress more closely than most law professors and maybe even more so was just more convinced than most law professors that that the progress was impressive. And um I think that that just meant that I was paying more attention as LLM started to get good. Um and you know like when you're trying to get that kind of information as it's coming out I think you end up in you know Twitter spaces that are also interested in things like AIX risk. Um I started reading some of that stuff. I read super intelligence. I read some of Yudkowsk's stuff or you know the stuff people read when they first learn learn about this. Um and then also Simon my um my my co-author um on this paper was was sort of interested at the same time and yeah between like us talking about uh all that stuff and and reading it and me being quite convinced that AI progress was was was accelerating and that there was in principle no ceiling. Uh I just was very became very convinced that this was a problem to work on and so I started to try and think about what what law had to contribute if anything. Sure. And and has that like I'm wondering how that shift of emphasis has gone like like was it difficult? Is it like do your colleagues think you're crazy or you know like how how's that all happening? I think they think I'm a little crazy. Well, let me put it this way. I think every every month they think I'm a little less crazy. So, uh, that's the harp. Yeah, it's the dream rather. So, yeah. So, um, so I had the idea for this paper. I don't know like probably uh probably probably around when I was going to go on the academic job market is or maybe it was really early after I had had done it. Um, but that was a that was long enough ago that if you wanted to be a law professor who wrote about AI, you couldn't go on the market saying I'm a law and AI scholar. Um, that was not like a genre of inquiry that existed. Uh, you sort of had to say, well, I I'm like I'm I'm a civil procedure scholar and I have this paper about machine learning and class actions or something like that, right? But of course, the world has changed a lot since then. And so, you know, like every every yeah, every six months or so, I I feel like um a noticeable shift in the overton window, you know, so I kind of I mostly didn't work on this idea. You know, there's like a YouTube video of me kind of presenting at the Center for AI safety a couple years ago, but I kind of sat on it. I sort of, you know, when I would when I would, you know, talk to other legal academics about, you know, AIX risk, they would they would they would be pretty skeptical and they'd say, well, isn't this kind of like sci-fi and like why would AIs hate us and stuff like that? Um but you know recently as capabilities have progressed and as I've you know mostly completed what I think is an otherwise pretty normal looking um tenure package I've just decided it's time to go like all in on um on thinking about law in and particularly law and AGI. Uh, and I will say I I I assumed that we would write this paper and it would like not get picked up or it would um get picked up in whatever just just you know a random draw from from the distribution of law reviewviews. I feel very pleasantly surprised that the University of Virginia um law reviewview which is a very very good law reviewview thought it was good and worth publishing. So for me that's a big positive update on law students as as being um interested in these kinds of uh questions. Huh. So so actually this gets to a question I have just about legal publishing which I'm I'm very unfamiliar with it. When you say the students like are the students who the ones who run the law review? They do. And law reviews are like in my do I understand correctly that those are like the main journals for legal writing being published. They are. I can I can sense your bewilderment? Yeah, that seems kind of crazy, right? Like I normally like in machine learning we have conferences and there's someone who heads the conference and they're normally a really fancy experienced person and then they have area chairs who are also fancy experienced people and you get you have these graduate students in who like uh do the reviewing of the papers but like and and maybe they run some workshops but you let these students run the whole show like what if they mess it up? Yeah. So um so this is a huge debate in like within legal academia and then like between legal academics and others about whether the law reviewview system is a good system. It's certainly a weird system compared to other other academic disciplines. But but yes, the thing that's the thing that's going on is like the the most prestigious outlets for legal scholarship are, you know, like the Harvard Law Review and the Yale Law Journal. And those are studentrun publications where students decide what to publish and students handle the editing pro process. Now the arguments against this are the ones that you gave, right? Like what do law students know. Um you know they're not they're not you know they're not academics. They don't know the literature in the way academics do. Um they have less sense of what's original, what's persuasive, and so on and so on. I think those are all um all valid critiques. On the other hand, I do think there are kind of deep pathologies of the peer review system. Um I do think I think peer review and actually the your description of computer science is like an interesting hybrid. It sounds like there are these nominal or these these chairs that have some gatekeeping power, but the reviews are like maybe done by by graduate students who have um Yeah. Well, well, it's sort of this unusual situation. So, in computer science generally, it's a conference based. Yeah. Which is also weird, by the way. I know. I'm sure you know that you like put everything on archive and that's like what everyone cites and then like they you figure out later whether it was any good by by whether it gets gets accepted to a conference. Yeah. Well, well, well, the key thing about it being conference also is just that um because even in journal based systems, you can like put things on archive as well, right? So, so there's some willingness to site archive things. The nice thing about it being conference based is like there's a timeline for people to review your stuff because at some point they're actually going to physically have the conference where everyone goes to a place and like you have to have it ready by then. But um yeah I mean basically so you have um reviewers who I think are mostly graduate students just because the field is grow also like the field is not in equilibrium in AI right it's like growing year on year and so at any given point the most most of the people are like early in their PhD and so that that's who you have to get most of your reviewers from. Now there are also like uh you have like area chairs and program committees and so so like the reviewers review the papers in this like double blind fashion um and then a higher up in the conference can say like oh actually like these reviewers were wrong and I think I'm actually going to reject this paper or like this reviewer's argument was better than this reviewer's argument so we're going to accept the paper. Um, also a somewhat interesting aspect of the system is that there's like uh a lot of these happen on this website called open review where um it's sort of like a forum and there are some versions of this where like literally everyone can see your anonymous paper and like anyone can write comments as well as the official peer reviewers. But like I I think that's not normally turned on. But um you know you you get to just have a comment thread with the reviewers and like you can say like no you know you said this but actually this um but but but yeah so so a lot of the reviews are done by graduate students but um people who are more senior than graduate students or or maybe like final year graduate students or something um but but like you know generally people who are like more senior are making the final calls. Yeah. Yeah. Yeah. And so like look, what's wrong what's what's wrong with the with the peerreview system? Um I mean look, I've I've never had to publish in it, but you know, Simon, my co-author, is is a philosopher and and he'll say things like, well, look, there's this intense pressure towards special like very narrow um uh specialization in topic because you're you're basically um you're writing to um a small handful of people who will be selected as as your reviewer given your topic. And so um so you you know you're you're you're quite quite pressured to specialize on just the questions they think are interesting using the approach that they think is most interesting. Um that tends to fragment the field. That tends to have people work on narrow niche technical topics instead of topics that have broader scope um and um and are sort of more um more uh synthetic of of different ideas. Um there can be kind of a lot of gatekeeping. So if you have an idea that is um is like is not ac not well accepted. You have a good argument for an idea that is is out of fashion. It can be very hard to publish because all the available reviewers will have you know priors against your your your idea. Um and then the labor is really constrained like the number of reviewers is really small right? it's just like you know whatever tenure track law professors or senior graduate students whatever thing and so as compared with that the law reviewview model has um um has these people who are less knowledgeable but they're super smart I mean like whatever students at the law school are are very smart I assure you and so they don't know as much but they but but but they're but they're very they're very good um uh thinkers in general there's a lot of them right like it's a kind of a prestigious thing to do as a law student to be an editor of law view So there'll be like like several dozen really smart people reading and discussing uh this paper. Yeah. There are more law reviews, right? So the the supply of journals is less constrained. So um instead of like waiting years and years to try and get your paper into like one of the top five journals, it's like you can be very excited to get your paper into like one of 20 or so journals or 50 or so journal depending on you know what what you work on. And uh and then and then from there it's like a little bit more of a free market system, right? It's like maybe the quality signal from journal placement is a little bit less strong, but the but maybe the output is higher. It it's hard to know, but you know, I like whatever it's uh confirmation bias or something, but I'm sort of I'm weekly in favor of our our wacky system. I get hang on. Wait, another question I have is so I had gotten the impression that people in law school were really busy that it was like a lot of work. It is. And like if you're a student and you're also like helping run this law review like presumably you have to like you're saying like lots of people are reading these articles like they're also not sure like in machine learning you get 12 pages. Yeah. like at the most, you know, um like if if it's a harsh conference, you know, you get eight, right? Yeah. Like or sorry, I think like eight or nine is like the normal one. Like your p your paper is 88 pages. Yeah. Now, a lot of that a lot of that is footnotes, so you know, potential readers, don't be too put off, but like how how are they not too busy to do this? They're such hard workers. Like it's they're just really hard workers. Okay. Uh and and actually so like when you think about the the ecosystem that produces this um a lot of these are people who um who go to law school and they want to get like jobs at big law firms. They want to work for Scatteren or Wtel or one of these big law firms. And uh and what are the criteria for being successful as an associate at at Scatteren? Um well one you have to be smart. You have to be able to do the work right? But two, like uh these are firms that bill out their labor in increments of ten of an hour, right? Like six minutes, right? Um and uh and and their profits um you know, beyond a certain certain margin of hours, like it's it's basically all pure profit. And uh and so so lawyers work, you know, big law attorneys work really hard. They work long hours. Um and so uh and so and so wow, you're on the Harvard Law Review and that means you're uh you're you're like busy all the time and you're like churning through these 300 footnote um articles and and and finding all the sources and checking the pin sites to see if they're accurate, you know, like what could be a better signal of your fitness uh for for work in a big law firm? Okay, I I can see how it would be a signal of working hard. Yeah. And like maybe it's a signal of that as well. I would imagine there would be things that would more simulate working at a law firm than reading law review articles. Like like my understanding it if I take your article right my understanding is that law the things law firms do is they like it's companies suing companies about like potentially breaching contracts for IPs or whatever. Yeah. And that just if I'm right about that sounds like really different than evaluating whether your paper is good. Like, am I wrong about that? Or is there just not another thing that these students could do to better prove their ability to work hard at these law firms? It may not be that there's literally no other thing they could do, but I think maybe you're underrating. Um, so one thing to say is like, you know, our article is a little weird as a law of your article. Like we have we have game theory tables in it and that's not normal. Um, and there's um there's like some IR stuff in there. My other papers have a lot of um e economics in them. Sure. And that's, you know, like those range from being like common but not standard to like very uncommon law reviews. But in general, so um uh it actually might be really valuable as a lawyer. It actually might be really valuable as a lawyer to have the skill of becoming like like an like an 85th percentile expert in an arbitrary topic. Um so it is true that what big firms do is like companies suing over other companies over IP. But for example, when I was in practice, I did a fair amount of IP work. And that ranged from companies suing companies over a patent on um like a radio device that helped to track the location of semis to companies suing companies over um infringement on a patent on on biologics like you know drugs. Okay? And those are very different, right? And u the question of infringement depends on questions depends on issues like was this invention obvious in light of the prior art. Right? So to be a good patent attorney um your job isn't to understand the science at the level of the the inventors. We hire experts for that. But you have to be able to understand it well enough to write for a generalist judge in a way that is convincing as to what's going on and as to what as to what the law is. So, so being able to like wrap your head around an an sort of arbitrary area of of of inquiry and understand it pretty well, not the best in the world, nowhere near the best in the world, but like pretty well. Yeah. Um is maybe a really valuable skill, I guess. So, object level domain wise, I can see how that would be true. I mean, I imag surely surely attorneys must specialize at the level of like the aspect of law that they deal with, right? to yeah to a significant degree although maybe less than you'd expect. Okay. Yeah, I guess I would imagine even if you're just a patent attorney, even if you're just a patent attorney, the number like they're they're they're the best patent attorneys in the world are not so specialized that they're just doing biologics, right? They're experts in patent law and their clients are doing everything from drugs to smartphones to automobiles to, you know, anything you can patent. Sure. I mean, okay. So, it still seems like, sorry, maybe I'm getting too hung up on this, but so if I go to virginialawereview.org, I click the button that says online, I think that's like the papers they have. I'm seeing like that's probably their um that's probably their their companion. So like a lot of law reviews publish long things in print and then they have an online companion where they they publish shorter things. Oh okay. So all right. Well okay. So so I'll go to like um their print thing. Yeah. If you click that's that's the that's the long stuff. All right. So there's one thing that says interpretive law making which is about I guess I I don't know interpreting and making law. One that's called shame that's about sexual assault law I think. One that's called partisan emergencies. Yeah. that has to do with emergency powers. Like, um, it seems like these are a lot of different areas of like like this just seems like it's so general that like I don't know. I'm still confound like maybe there's not much to say here, but I'm I'm still confounded by this. Yeah. So, I mean like the range of stuff that you see as a law review editor is probably wider than the range of stuff you would see in any um in any legal practice, right? Um do do the do the reviewers actually do the reviewers specialize like in in machine learning, right? If you sign up as a reviewer, you say like uh these are my specialties. Like in law reviews, do you say like I'm only going to review things about administrative law or? Uh not formally. But again, there's many more. So like um different law reviewviews do it different ways, but like um but like the Harvard Law Review has I don't know the numbers, but like it it might be it might be like 60 members or something like that. And um and uh and to some degree there's like different stages of review, but to some degree what happens at the end is like they all decide together, right, in the aggregate. And so to the extent to which there's a there's a there's a game theory expert on the Harvard Law Review who can then then give their their informed opinion as to like how everyone else should vote that can that can kind of bubble to the surface in a way that um wait all 60 people meet and they all 60 people vote on every on the Harvard Law Review that is the mechanism. Yes. Individ I believe there's going to be some Harvard Law Review editor listening who's like it's not exactly that but I think yeah I think it's pretty close to correct. Huh. Isn't that crazy? Like like like shouldn't we have one guy who decides or you know one person who decides like per area or something and have some like hierarchical thing or Well, look, it just depends on whether you think um whether you think uh the kind of uh it's not exactly wisdom of crowds because um right because because there's not like independent blind voting, but like it just depends on whether you think you get more juice out of uh out of uh out of debate. um or out of expertise and probably both of those are are valuable. Um sure and they both have p pathologies. That just seems so expensive. It's extreme. But again, like in in terms like in economic terms, it's expensive, right? There's like a lot of labor being dumped into this. Yeah. Um in nominal terms, it's free because the students do it for free. Sure. But as but as a student, you know, you you know, you you have your time. You might prefer to spend I don't know maybe maybe you don't prefer to spend it on leisure because you want to prove that you're a really hard worker. Yeah. I mean you're in law school for a reason. You're you're collecting a bunch of signals that you think you're going to be valuable in the future. We have this, you know, we've we've arranged the world in such a way that being on the law review is one of the ones that's that's that's unusually valuable. And look, as a as a veteran of um of of a of a law reviewview, it was a lot of work, but uh but it was fun, you know, like uh like I think in most fields being a journal reviewer is is is is total drudgery because you get this you get this manuscript and then you got to write a long thing about it and it kind of goes into this void and there's going to be some other jerk who who just like who just like dumps on the piece even though you thought it was really good and then gets rejected. you feel like your effort wasn't worth it. But like on the on a law review, you're having like an academic symposium every day, right? There's a draft that's in, you're going to discuss it, you're with your friends, they're also smart, they're interested in it, you argue about whether it's any good. Uh yeah, for a certain kind of person, that's a fun experience. I Yeah. And I guess the argument is probably helpful if you're a lawyer. Um, yeah. I I guess like to the extent that the system is partially driven by students having fun running a law review, like it makes more sense to me how you could end up with this. Yeah. May maybe, okay, maybe not all of our listeners is as are as interested in how law reviews work as I am. Um, so you can out as much of that as you decide. No, no, it's all staying in. Uh, we'll we'll provide timestamps. They can skip this bit of the conversation if they want. Um but uh okay but but getting back to where I was branching off from so so when you when you decided I don't know for a while you were doing like law and AI but you know like how AI might impact various areas of law with you know this um some previous things like AI will not want to self-improve um uh AI outputs are not protected speech I guess that's like more yeah those are both by the way I think of those as my post like post my turn to AI risk like my older You have a paper about, you know, using machine learning to um do a certain kind of jury simulation that allows you to certify certain class actions that you couldn't otherwise, right? Uh another one about whether kind of um like boring kind of regression models of um of different kinds of impacts in like say like like like racial impacts in hiring would be like a sufficient legal basis to do like you know um like uh calibrated affirmative action policies. So, so that that's the stuff I'm talking about. Why thinking why I was saying thinking about how like machine learning and big data type stuff can like help us make the law work better. Sure. And yeah, at some point I start thinking about these other things. So, so when you pivot like is it like like do you just like decide you want to work on a thing and work on it or do you find yourself like needing help or like you know B basically I'm curious like how how well the pivot worked for you. So you know as compared with other disciplines like law writing for law reviewviews tends to be more solitary right so the number of authors on a paper is like between one and three at the high end or something like that but one is by far the most common uh and so in that sense it doesn't require a lot of help although I will say that um for me at least one of the things that um I found uh really valuable in making the pivot was was kind of getting connected with the very small but um hopefully growing. We're actually trying to help grow it um with this this organization I help run called the Center for Law and AI risk but the small uh community of of of law people who who are interested in this stuff. And so um so um for example Kristoff Winter is the director of um law AI which is Harvard affiliated um uh group that had been kind of thinking more broadly about law and X-risk but around the time I started working on this was was pivoting to be um much much more um law and AI focused. Um, I started doing a little bit of advising work for the Center for AI Safety and as part of that I helped them organize um kind of a like a summit. I guess it was like two or three summers ago now um for like law professors who were interested in that um or interested in in existential AI risk. And so from there met people like Jonathan Arbell and Kevin Frasier who helped run CLA the center for law and AI risk with me. Um and and you know a handful of other really great people. And so so having people to bounce ideas off of has been been super useful. Um but there's there's not a lot of formal infrastructure yet. Um, and again, that's one of the things we're we're hoping hoping to to build with this this center so so that more people can transition to do this work um easily. And so, yeah, I guess I'm also curious about the reception. So, you mentioned that your colleagues are thinking that you're a little bit less crazy crazy than they used to. Um, and you know, this this got accepted to Virginia Law Review, which it sounds like is one of the better ones. like more more broadly I'm curious like has have you found much uptake or much engagement um by the legal community with your writing or more broadly how's the reception? Yeah, I think I think it's I think it's like I think the idea that AI could become very powerful has been entering the Overton window in law especially over the past I don't know say nine months or something if I were just to you know when I started writing the draft of this paper that was like it was like summer of last year and that was the point at which I sort of thought um this paper is kind of wacky. It's probably outside the Overton window. It might not even get published, but I think it's important and you know, Simon and I should should write it anyway. Um and you know and at that time that was informed by you know places where I had gone and prevent presented prior work like the the AI self-improvement paper um where I spent most of my time um when I would present that paper to law faculties just trying to convince them to take seriously the idea that we should model AIs of the near future as having goals and having being able to plan um and act rationally in in in pursuit of those goals and being capable of doing stuff that could be dangerous as they pursue those plans. And they were just like not on board even with that that premise. And that's, you know, that's like that's like the that's like the foundation to like that's the pre-ereading for the paper, right? That's before you even get to any of the arguments in that paper. So, I just found myself doing a lot more conversation about that at that time. And um and then so we I wrote this Simon and I wrote this paper um the AI rights paper we've been talking about um expecting to have the same reception just people kind of like getting off the bus right at the beginning uh and it was basically the opposite. I got asked to go present the paper to the the the faculty workshop at at Forom Law School in fall and uh and immediately they wanted to just like dive into the substance of like you know like whether the payoffs in the game theory matrix were right or whether there's other versions of the world in which they could look different and and questions about like um you know some of the stuff we've talked about individuating agents for purposes of holding them to their contracts and it was just such a big shift. I don't I don't know exactly what explained it except to just say that, you know, every few months, you know, anthropic or open AAI or somebody releases an ever more capable agent and more people use them and lawyers despite being quite quite small C conservative um are are noticing. Could it uh sorry, I'm not as super familiar with the culture of legal academia. Could it just be that form university is like unusually AI pilled? It could be. Um, you know, data points include, um, not just that, but the, you know, the the editors at the Virginia Law Review being into it. Um, I've given the paper in a couple of other places, too. Um, God, what's the whole list? Um I was going to say after right after forom I was at at Oxford giving the paper but you're going to say oh Oxford's very AI pill. Although I will say I gave the paper to the uh Oxford faculty of business law which I think has basically no interaction with like um FHI or whatever. Rest in peace. Uh yeah, Dearly Departed. Uh yeah, there's other there's others too that I just my my brain is uh not not being able to recall. But I would say just in general like um you know I've I've talked about this paper in a number of like legal academic settings and and people have been much more interested in talking about like the merits of the idea conditional the the assumptions that I give rather than challenging the assumptions. Okay. And how are you finding um so yeah I guess you're sort of moving into the existential risk community like how how have you found the reception of the paper um among the community of people you know among the AI risk world? Yeah pretty I think um I think it's been reasonably good. Um I gave an earlier version of the paper at uh the global priorities institutes I think twice annual conference. Um that's I think I was probably the only lawyer there. I think it's mostly philosophers and some economists but yeah like you know it's like the like the main the main AI risk people were there and um and gave good feedback. But I think we're pretty open to the idea. Um I guess um some of the some of the guys who run uh who run Redwood Redwood Research um have have kind of in the past wondered, hey, should we just pay payis to do stuff for us? And so um uh so they were interested in the kind of more technical analysis we did uh we did there. Um yeah, I would say over overall like my sense is the reaction is is is is pretty good where you know where there where people are skeptical. I think it's mostly people who who think that um who think that who have very short timelines and and think that like super intelligence will be here pretty quickly uh and think that basically you know like some monkeying around with the law is not gonna not going to accomplish anything. Yeah. I and I guess this actually relates to another question I have. I'm curious like uh are the x-risk people in the law academics are they like are they picking on the same things in the paper or do they have like I don't know do some people focus on some things and other people focus on other things. So like among the people among the legal academics who are interested in less x- risk is there like um is there a diversity of views about what's good and and bad about I more mean like do the ex-risk people focus on different things than the legal academics um the ones who are not in the uh intersection. Oh, the Yeah. Um yeah, I mean there's there's Yeah, there's there there's a difference. um like the like exist people tend to be like uh a lot more interested in um questions like like like what will bind AI labor at the margin or something like that like they're you know they're thinking already in the background they're like oh how much compute how much inference compute infrastructure do we need to build for AGI and how much how many power and that's kind of in the background of their of their of their minds already um and and uh the law people Um yeah, they have more lawy questions, right? So um uh so yeah, they immediately hit on questions like, well, isn't isn't like um isn't like stable identity really important to make um property ownership and contract like have the good effects you want? And um uh yeah, stuff like that. My next question is um so for people who publish an AI I sort of understand what the theory of change is supposed to be like roughly people will publish an AI paper indicating some technical fact and the hope is that other AI researchers learn this fact um and then eventually that when people are building AI they're they're clued in enough with AI research that um they then uh you know that that they then um you know incorporate this fact into how they build AIS um for lore articles. It's like so law professors read them, I assume. Does this uh does this somehow feed into what the law actually is? And if so, how? Yeah. So, in a couple there's three things, right? Um so uh so one is I do think so so to first approximation no one reads law review articles not even law professors and I think actually the way of thinking about them is um a little bit like how you would think about like like popular press non-fiction books which is there's some mix of like reference guide. It's like it's like somebody has an argument and um like do you need to read the the the um the the book uh taught you or whatever like you can think of like like the new Ezra Klein and Derek Thompson book the abundance book. It's like it's like do you need to read every page to have like a pretty strong sense of what the argument is and the extent to which you disagreement? Like absolutely not. But it's nice to have the book with the chapters numbered so you can see so you can like read the introduction, understand what you think the real weakness is and then go read the chapter that's about that and then figure out if it's if it's a good good response. Um and and so like like as uh as reference you know as that kind of like reference material I think they are somewhat more widely read including by courts you know courts will site law reviewview articles with some regularity not always for the core proposition of the paper sometimes for something that's in one of the one of the subsections and I will say there have been cases like in in history where like legal academic thinking like it's hard to point at one particular law review article article but some collection of law reviewview articles have been really influential. So like one example is um um the kind of like turn to economics and consumer welfare as a standard in antitrust was like very much influenced by um by you know a bunch of legal thinkers who are mostly putting their ideas out in in law reviews in like the 70s and 80s right so so should I imagine that there's some like technical law people and the places those technical law people live so partly they're litigators um they just have to deal with what the laws actually are. Partly there are judges who have some leeway to interpret laws and then partly there's something like I don't know agencies who are like you know proposing how they should regulate a thing or maybe they're like whoever writes the model penal code which it which is also seems like a crazy situation where as far as I can tell a bunch of random lawyers just like get to determine what criminal law is because it's more convenient to let them. Yeah. Well, but that sort of thing is that sort of thing how I should think of like the impact of law reviewview articles. Yeah, it's like um yeah, you've you've you've written you've written a piece of technical work um that has um a mix of technical claims and then kind of just like higher order gestalt around some set of ideas, right? Um you can think of the antitrust stuff about this like the high order higher order stalt is like it's like it's like you should think about price like price should be the thing and then there's very technical arguments in like you know in like uh whatever like Bob Bour's papers about how you should figure out whether there's been a price harm to he might not be the best example. I'm not sure actually his paper is that technical but there are people um and and and those ideas kind of um diffuse yeah through this kind of you can think of it as like elite legal uh apparatus which is some combination of judges it's policy makers who go into administrations who want agencies to do different things and they need to be able to reach you know even if even if your uh your heristic as is as is as um as uh is as high level as we're the Joe Biden FTC. We want the FTC to be doing more progressive stuff. Well, what progressive ideas are there out there for for doing trade law? Um and then you you pick the ones that have kind of bubbled up in terms of their like popularity and credibility and then you end up implementing you know some combination of the Gestalt and the technical idea. Uh yeah. So there's this kind of eco ecosystem of um ecosystem of think of legal thinking and then and then I do think it also spills over into uh like politics and you know whatever political discourse more generally like you can think of a lot of um there's a lot of ideas like right now um that people are are like regular people are talking more about that have their their origins in legal academia you like unitary executive theory is something that kind of like normal voters have now heard of, but that's like some law professors writing about like separations of powers in I don't know probably the 80s and 90s. I'm not not totally sure. Um yeah, so it spills over into into yeah broader political you know academic discourse as well. I I guess I also want to say um so it sounded like you also maybe wanted to react to this claim I made about the model penal code being a strange institution. Um I don't know if there's anything you want to say. Well, so yeah um the model penal code is in some ways this like um almost distilled example of what I'm what I'm what I'm talking about because the model penal code is just a model. Right. Right. No one has to adopt it. It's just some people who are law professors um who are designated as experts by I think it's some like maybe the criminal law section or the ABA or something. There's some some ABA being the American Bar Association. American Bar Association. Yeah. Some collection of lawyers. Yeah. But they have no legal power. They're just an institution and they designate some people who they think are experts and they say write a penal code like you guys know how the law ought to be. Write it down as a model. Um and uh and and then there will be and then states can adopt it if they want. And then there will be some states who say our penal code is bad. It's maybe maybe it's not even bad on the merits. Maybe it's too confusing. We don't have that many statutes. It's you have to like know a lot of case law to know what's going to happen. We want to standardize it. Um we need a policy. And what do they reach for? The model penal code. Not even because they they like think it's correct on the merits top to bottom, but because it's there, right? It's there and it's a product of the like whatever elite legal institutions that they rely on to policy. It's interesting that they picked that and not like another state's penal code, which presumably like you could just pick whatever state you think is, you know, you like the most and pick their penal code, right? Yeah. So, there's there's a fair amount of that that goes on too, right? like like um states borrow laws from one another. Uh states borrow borrow federal law for themselves. Um so there's there's you know there's a there's a selection of there's a menu of of things you you can choose but one of them is like here's a tome I the law professors wrote uh and sometimes the tome gets kind of adopted. Sure. Sure. Yeah. So okay the final another good example. Sorry I just thought of one more good example. So um maybe you know um who Lena Khan is. She is the Lena Khan is the former um chair of the Federal Trade Commission. Um uh and famously during the Biden administration, which was the administration that pointed appointed her um had an agenda for antitrust in the United States that was like quite different from what came before it. It was less focused um myopically on on whether monopolies were raising prices. Had a more holistic view of when monopoly power could be bad for say politics and um was more skeptical of big business kind of in general kind of in principle than than prior regimes. And um why did that happen? Why was she the chair? She wrote a student note in I think it was the I think Columbia Law Review. It was I think that was where she went to to law school. Um that yeah that just had some of these ideas about how bigness in principle can be bad and it kind of caught on and I don't know if Joe Biden thinks that particularly but it had it caught on as a kind of progressive idea of what antitrust could be. And so when Joe Biden was looking around for how we could make the government more progressive, well, that was one of the packages on the menu of items you could choose and that was the one that got got chosen. And you know, I think you can trace it directly back to a law reviewview article. Fair enough. And so and so so I guess is the hope here that somehow like you know we write enough law reviewview articles about like the potential legal regime for AI maybe we get a model uh civil AI code or something is is that sort of the the eventual theory of change here? Yeah, I think it's something like um you know if you think that like that AGI is going to happen whatever whatever time whatever whatever your timelines are Um it just seems pretty plausible that um that at some point there will come a moment where sort of everyone decides that we need to do something right and there there will be many things that you you could do. One is you could you say hey we don't know how to handle these like capable agentic you know things that can act on their own behalf over time and we should just like ban them. we should just like turn them all off or whatever. Or we need to mandate control. Control could be the thing. It's like we'll pass a federal statute that requires like maximal control over um over AIs by um by their by the labs that make them and we'll we'll outlaw open source maybe and you know that could that could be a kind of package of things that happen. And the hope is that then there will be this this other thing which is this kind of we think of this paper and then the the the research agenda that we want it to inspire is sort of small L liberalism for for AIs. So maybe there'll be this other thing which is small L liberalism for AIs. It's kind of a package of ideas that's available to implement and and you know there will be different arguments about why each of these are good and we hope that in so far as the arguments we make are the best ones that that will have some some some effect in in making them sort of the package that gets picked up off the shelf. I guess I want to move on to a bit of just sociology of the law profession. Um so first of all so you're writing this paper it has a bunch of game theory in it. I'm aware that like law and economics is a sort of thing like like kind of a school of thought. Yeah. Do you I don't know do do you sort of come from that tradition or is it like sort of a coincidence that you're writing a paper with a bunch of game theory? No, I don't think it's much of a coincidence. So, I went to the University of Chicago for law school which is in many ways like the intellectual home of of of law and economics. um I learned law from a bunch of really great um people there who are very much or oriented around that that way of thinking. No, I I it's not true that everyone who teaches at Chicago is like a hardcore law economics person. Like yeah, it's it's a great faculty with a diversity of of viewpoints, but but yeah, if you wanted to yeah, if you wanted to learn how to think about law through the lens of economics, it's not clear you could do that much better than getting a JD from from Chicago. Um which is which is what I did. So So not not a not a coincidence. Although I will say game theory in the law is um is even a little bit uh less common as a methodology even even among people who like do law and economics like there definitely are some books and papers I'd point to where where game theory is the way of doing economics um that that gets used but but I would say it's a pretty small minority even within law and economics. Fair enough. So there's a thing I perceive that like may Maybe I'm wrong. I'm curious how accurate my perception is. So, sometimes I run across uh legal, you know, law people or like I I'll find myself reading a law blog or I'll listen to a podcast and I think because of my personality, it tends to be like personality and interest and stuff. It tends to be like either they're originalists and I'm reading the vol conspiracy or I'm listening to this podcast called divided argument or it's law and economics and I'm reading like a law and economics person write a thing about how we should think about AI regulation. Yeah. Um, and I have some imagination that the originalist in my imagination the originalists and the law and economics people get on get along together and like they both teach at the University of Chicago even though it's not obvious that like they all agree on like I I've got to imagine that sometimes like originalist interpretation of the constitution and like law and economics prescriptions for how things should be must often diverge. But um and it also seems to me that like it seems like these people are the most likely to be computer Zy, right? Like yeah, for one, I'm computer Z and I run into them, so probably they're computer Z as well. Like Volo Conspiracy. Um like I think Eugene Volock is like uh did some programming stuff before I went into into law. Will Bode is like playing around with LLMs on the regular. Um, yeah, like like am I right to to think that this is like kind of a cluster? And if I am right, like how worrying is it that there's this like one kind of person and they're the only people like really thinking about AI in the legal profession? So, um I do think there is a cluster, but I'm not like the explanation could just be coincidence. M so um so I think if you think back to uh kind of political conservatism conservativism of the like 70s through 90s uh you know there's like this like Reagan fusion of of of sort of kind of economic um like free markety type thinking um and then uh a certain kind of way of thinking about about social values and uh and in that environment both law and economics and originalism ended up coded conservative and I think it's it was probably sociologically true also that like the people who did both those things were um you know polit politically conservative and um and and so yes so so some of the most important people who did that kind of stuff clustered at you know a particular set of of law schools because whatever some combination of those law schools are more willing to hire conservatives or they had hired some people who were open to these methodologies and those people hired some more people. Uh and and so and then and then to some extent there's like a kind of persistence of um of uh of of of yeah of of overlap in in those in those two cultures. Um, I will say I think that's breaking up to a significant degree now. Um, yeah, like for sure when I read Volo Conspiracy, I guess there's one contributor who seems pretty proTrump, but the rest of them like don't seem very on board with that. Yeah. And and you know, so there's like this whole range of things. There are there are progressive originalists now who I think are not particularly likely to also be into law and economics. Um, law and economics in some ways has been like I don't know if it's exactly to say it's right to say it's a victim of its own success, but but even when I think about like like economics departments, you know, um, academic economics departments, they're they're basically just doing like high quality empirical social science which doesn't have that much of a political veilance anymore. And and I think to some extent law and economics is like that now too. It's like um you know the political veilance is is wide and so there just like plenty of people who have kind of a law and economics bent who just don't think conservative or conservatism is very interesting and and thus because they code originalism is conservative. They don't think it's very interesting. Um so I think the uh I think to some extent those kinds that kind of um yeah like cultural overlap is uh is breaking up. I will also say if your main original the main originalist you know is is Will Bode. Uh he's he's like a teacher of mine. He's fantastic. I love Will Bode. I think he's pretty weird as a I think he's unusual as an originalist in that um a his like his his like originalist theory is is pretty different from like the Scalia Scalia theory but then also like there's uh there's one episode of his podcast with Dan Eps which I assume is yeah you me yeah you mentioned um divided argument. There's one episode where they have a guy on who is talking about his book on international relations and at some point some point Will suggests that like the main reason he's maybe I don't know if it's the main reason he's an originalist but but one good reason one could be original an originalist is that law is basically um uh just serving as a shelling point in in like a in like a 300 million person coordination game. And the most obvious shelling point for constitutional law is the document we wrote down 250 years ago. And that's a really good reason to be an originalist, but it's a super different reason than than you you might have if you were a different kind of originalist. Yeah. Anyway, I I I do get the sense that he's like unusual as an originalist. Um I I guess most originalists aren't like think that Donald Trump is actually the president, for instance, etc. I I guess I don't know if he thinks that Donald Trump is or is not the president. Um Wait, what's the argument that Donald Trump is not currently the president? Well, I Okay. Okay. So, I'm really bad at law, right? Uh because I've never studied it, but so will we'll vote. He has this big article about how like the 14th Amendment says that uh you're not allowed to have um Donald Trump be president. And so you might I would have naively thought that if the 14th amendment bars someone from being eligible to be the president and then there's a presidential election and that person wins, maybe that person isn't really the president. Now to be fair, I haven't heard him make that additional step. Um I don't really know. Yeah, I'm I'm sure I'm sure that Will has like a very um very well-considered view about exactly this question, but I'm not sure what it is. But yes, you're right. you're you're you're you're high liver point where yes, most originalists don't don't buy the I think very originalist and and probably correct um ineligibility argument um where whereas he does yeah it'd be interesting to find out uh so so so getting getting back a bit to things that are that have some chance of interesting people other than me um um so so it sounds like you think that Um, so I I was worried that maybe there was this cluster of originalists and um and law and economics people and they were the only pe law people thinking about computer stuff and and maybe that was bad because they were kind of homogeneous and it and it sounds like you think a the originalists and the law and economics people are not so like in tandem anymore and b they're you know they've the political veilance of these things is spread out such that it's not even that homogeneous anymore. Is that roughly what you're saying? At least as compared with the past like I don't know probably other law professors would dis disagree with me. I do think there's an extent to which both these things do still code conservative although I think there are you know like prominent examples um to the contrary that that everybody knows. I also don't think it's exactly true that um that those are the only people thinking about like law and AI or law and computer stuff. Um there is you know a whole mountain of like law review um articles on questions like uh whether AI will uh be biased or along racial or gender um or other uh suspect lines. And as a sociological fact those people are basically not interested at all in in existential risk. I'm not totally sure how to explain that. I think I think you could like tell a story about um about uh kind of like mood affiliation visav big tech and like if you kind of you know if you're kind of if your prior is kind of that like um in the past social media sort of promised to be you know this very powerful tool for like liberating you know autocracies in the Middle East and you know like helping people live more fulfilling lives with with their uh with their their friends and loved ones. And what actually happened was it was kind of like just like a soul sucking drag on on the world. And the basic thing that that um that that it did was like like destroy people's privacy and extract value from from like poor and otherwise disadvantaged people. uh that you will be kind of uh you know disincined to to believe in AGI. Yeah. And I think maybe that's the sociological um explanation for what's going on is it's a cluster of people who kind of have th that first set of views and and and they're they're pretty untrusting of of claims from industry about what technology is going to do. Yeah. And I think you you have a similar sort of thing in the field of AI where um people you have people who are sort of worried that AI will be biased or you know be deployed in ways that are bad for um uh you know minoritized groups or whatever and they tend to um I I think the actual thing is that people who are worried about AIX risk and these sorts of people often tend to find each other irritating. Yeah. There's big cultural differences. Al although I I I really hasten to say that I think um this divide has like I I think you're starting to see more people in the intersection now um than you used to. Um and I which to be clear I think is good. Yeah. Like I I I lament this this division. I don't think it's good for I don't think it's good for people who care about ex- risk or influential people um to to to think we're crazy or not worth engaging with. I I think I think the other way around too. I think that people who are really worried about um AI discrimination or you know privacy harms from AI whatever could benefit a lot from engaging with um people who are interested in X-risk because in in many ways the the you know in many ways the concerns um like overlap um from a yeah technical and legal perspective. So, so on that consiliatory note, um, uh, I've I've used a ton of your time. Thanks, thanks very much for being here. Um, before, uh, we wrap up. Um, if people listen to this episode and they're interested in following your work, uh, what should they do? Uh, yeah, so there's there's three places they could look depending on on what they're most interested in. So um uh you can find uh my academic work and actually most of my writing at uh my personal website which is uh peter n um n is in nancy cib s a l i b.com. Uh if you want uh more digestible pieces of writing than 88 pages of law reviewview, I'm also um uh an editor at LawFare and you can find all of my um shorter form works or not all but all of my shorter form lawfare uh works uh there. Uh many of which sort of touch on or summarize um my my longer academic work. Uh and then the last thing I mentioned at some point during our talk that uh I really think that like the field of of of law and AI safety is um you know neglected and potentially high impact. Um I think it's a really good uh idea to try and build a community of legal scholars who are interested in working on on these questions who who you know want to build out all these parts of the research agenda that I motion uh gestured at today or who think like I don't know what I've talked about today is totally wrong and there's like much better way to think about um uh law as uh as we approach AGI and so um I co-run done something called the center for law and AI risk. Uh and we are working to build that community in legal academia specifically. We think um that there is uh a lot of potential upside to having this kind of uh blue sky thinking like laying the intellectual foundations for um for how law should govern um AI to to reduce existential and catastrophic risk. So uh please do um go to claire-aii.org or and uh join us for a future event. All right. Well, well, Peter, uh thanks very much for joining me for the podcast. Uh thank you for having me. It's been a a real treat. This episode is edited by Kate Brunaut and Amber Onace helped with transcription. The opening and closing themes are by Jack Garrett. This episode was recorded at Far Labs. Financial support for the episode was provided by the Long-Term Future Fund along with patrons such as Alexi Malafv. You can become a patron yourself at patreon.com/exrpodcast or give a one-off donation at kofi.com/exrpodcast. That's kofi.com/exrpodcast. Finally, if you have any feedback about the podcast, you can fill out a super short survey at axp.fyi. Just two questions. [Laughter] [Music] [Music] [Music]