The Case for a Global Ban on Superintelligence (with Andrea Miotti)

Sam Alman even before founding OpenAI said the development of superhuman machine intelligence is the greatest threat to the existence of humanity. A mod of entropic said there's 25% chance of a catastrophic outcome for essentially humanity wiped out. Elon Musk has been extremely vocal for a decade now by talking about 20% chance of annihilation. Yet despite all of these risks, AI companies have been fighting tooth and nail to just not be regulated at all. If we build systems that are smarter than us across the board and we cannot control them, we are screwed. There is no gain from getting to super intelligence. The only actor gaining is the super intelligence itself. The fundamental win condition is that there is deep buy in about understanding how big the risks are, understanding the super intelligence under current conditions is a terrible idea for humanity. If enough people have this, we have won even without any specific law because those people will make the right decisions collectively. Welcome to the Future of Life Institute podcast. My name is Scott Stalker and I'm here with Andrea. Andrea, welcome to the podcast. >> Thank you for having me. >> Amazing. Tell us a bit about yourself. >> Yeah, absolutely. And you know, thank you for having me again on the podcast. It was great to be here last time. So, I'm Andrea Miati. I'm the founder and CEO of Control AI. Control AI is a nonprofit working in the UK, the US, and starting to work in a few other countries to prevent the most extreme risks from powerful AI systems. >> Okay. So, I want to start by talking about the current moment we're in here. Um, could you tell us about the landscape of uh organizations and funding uh when it comes to influencing AI policy? >> Yeah, absolutely. So I think what we're seeing you know in the past year and especially the past months is a veritable flurry of lobbying from most of the AI companies. So we you know what AI companies as you know many many many listeners of the show might know but it's always worth repeating are focusing on one clear goal to develop super intelligence that is AI that can replace and out compete all humans at all tasks. This is why top AI experts like Nobel Prize winners, top AI scientists and concerningly even CEOs of many of these AI companies warn that uh we risk human extinction from super intelligence. Yet despite all of these risks and these risks that are becoming, you know, more and more well known across the public and openly acknowledged by many of these companies, um KI companies have been fighting tooth and nail to just not be regulated at all. And in many ways they're deploying a similar playbook to the one that was tried and tested by tobacco companies the so-called tobacco lobbying playbook where you know uh in tobacco what happened is that the the notion that cigarettes and you know smoking could cause cancer was discovered pretty early on and was well known pretty early on inside many of these tobacco companies. But instead of stopping or instead of you know working with governments to chart a different path, these companies started to sweep this under the rug and especially to do essentially propaganda campaigns in the public to deter intimidate scientists that would reveal findings about what tobacco was causing which is cancer across many smokers. And we are seeing the same with AI companies like you know on one hand we have many ex AI company employees that quit and very bravely in many cases by losing millions of dollars tens of millions of dollars you know by now probably even more than that speak out about the risks. We have even the CEOs of these companies speaking out about the risks especially in the past like you know I I could find a quote from each one of them just kind of some of the most famous ones Sam Alman even before founding open AI said the development of superhuman machine intelligence is the greatest threat to the existence of humanity. So you know these are these words are very clear like he knew what the company was going to build could end humanity. uh other uh cos like a mod of entropic said there's 25% chance of a catastrophic outcome for human civilization essentially humanity wiped out um Elon Musk has been extremely vocal you know very very vocal for a decade now by talking about 20% chance of annihilation maybe more now in recent interviews is even saying you know no matter what when once we get to AI smart than humans humanity will clearly not be in control anymore so we can sit and and you know let them do what they want and things like this. But at the same time these companies are raising billions of dollars you know tens of billions of dollars and they are spending them to prevent any form of regulation. You know on one hand you have this threat that affects all of humanity that they're very well aware of. At the same time, they they lobby to to sweep this information under the rug to silence uh whistleblowers from speaking out and to make sure that no country regulates. And the best antidote to this is just speaking the truth. Uh the more people learn about this risk, the more they are concerned, the more they realize the stakes that are here and the more they want to act. And you know, the less they know, the the more the companies can keep doing what they're doing, which is to threaten all of humanity undeterred. >> The companies would say that they are in favor of certain types of regulations, right? They are they're not against any and all forms of of regulation. They just want targeted regulation or they want regulation when we have evidence that there's risk or they want regulation that is narrow in scope. What what do you say to that? So I think this is this is funny because I can embarrass the the play the tobacco playbook one to one. So I started my career as a as a lobbyist and what I can say is a bad lobbyist will say we want no regulation. A good lobbyist will say well obviously we want regulation just not exactly the one being proposed now. And every single time it will always be not the one proposed now. This is this is fundamentally just what a sophisticated lobbyist does instead of a of a naive and incompetent one. And this is exactly what we're seeing with these companies where of course you know they will have many arguments to defend their position. But time and again when faced with reality like when there is regulation being discussed they will always oppose it try to water it down try to get off the shelf and they do not propose anything alternative which you know so in the end look you know look at the hands not at the mouth it doesn't really matter what they say this is you know mostly uh PR what matters are the actions and again it's very similar to how the tobacco industry operated especially on on the last one that you mentioned about how we need more evidence. You know, on one hand you have the CEOs that very clearly know and have admitted that this technology could cause a human extinction, the development of super intelligence. And so, you know, regardless, you know, it's a it's a future event, but it is a risk that they are very well aware of and some risks are simply not worth taking, especially the ones that involve killing everybody we know, including likely uh themselves. But and but on the other hand there is this delaying tactic of saying we always need more evidence. You can always get more evidence right like you you can you can eternally uh get more evidence. Uh there is no like natural limit in reality that says like oh now the evidence is enough magically we have enough. And so for example, what tobacco used to do was to say well okay when the evidence started to become overwhelming that you know many many scientists were talking about obviously cigarettes cause cancer they started saying well um sure maybe that's the case we don't really know we need more tests we need more evaluations you know see where you've heard that word before but of course if you find the exact chemical that causes cancer we are very happy to see that regulated and to see that removed uh from a product but we haven't found it yet. So we you know unfortunately we have to keep producing these products and we should not be regulated until the exact chemical is found. And this is again just fundamentally a delaying tactic with tobacco governments and the public said a very simple word no I don't care you know we we we see with significant amounts of evidence and sign significant amount of theory and your own admissions that this causes cancer. I don't care that we have not found the exact chemical. you will be regulated. Now, if you want to keep operating, find a way to make your product not not cause cancer. And you know, in the end, a lot of large tobacco companies are still, you know, there and doing pretty well. You know, their their stocks are doing pretty well. They've diversified into other products that are, you know, less carcinogenic and and so on. So, this is just what companies always say that they want to dodge regulation. Let's not give too much credit to this and let's focus on dealing with the risk. What we're seeing at the moment is that capabilities of these the capabilities of of of these AI models are jagged meaning that they are very capable in one domain for example mathematics or coding and less capable in other domains like interacting with the physical world or say say writing uh might not be fully there and so while it's uh I think it's true to say that the companies want to create super intelligence How are they doing at that mission, right? How how should we evaluate how close we are to then actually developing and deploying super intelligent AI? >> Yeah. So, none of us can exactly predict the future otherwise we will all be billionaires. But I think they're making very significant progress and I think they know that they're making very significant progress. Um it is true that you know AI we are seeing that it doesn't exactly follow the same the same like basket of of of skills as humans and I think it's you know it would have been strange to even expect that it is a very different form of intelligence. It's a form of intelligence that we have developed in the past you know uh years and made many breakthroughs in the past years that we also fundamentally not even the creators of this technology understand how it works internally. So it's normal to see a lot of alien uh behavior but I think a lot of this is explained by what the companies are focusing on and these companies are focusing on on one approach to super intelligence which is to make AI systems that can automate AI R&D well and then to cause what some people call an intelligence explosion or you know recursive self-improvement and so these companies are allocating billions of dollars to automate that specific task. because they think this one task automating R&D which is a subset of automating software development will accelerate everything else to super intelligence very very quickly and on that we're seeing enormous progress like we are seeing you know now you know with cloud code uh people that before never wrote a single line of code can create entire apps from scratch you know people like nitpickers will always say like yes but there you know there's going to be one bug in the app like oh it doesn't exactly look as I intended like Sure, but like have you ever worked with a you know have you ever developed software yourself or have you ever like worked with a human software developer like you know we don't have telepathy like if you work with a fellow software developer you sometimes they will also make bugs and they will also not exactly design what you do and they will it will take them much more time they're human they need to sleep they need to eat they need to take breaks they get bored AI has none of that you can even run multiple AI in parallel and like pick the best app out of 10 you are developing in parallel with the same specs It's always it's always useful when having this disc discussion to zoom out a bit and if if I look back just a couple of years ago on this podcast I was discussing with people who were really in the know about AI whether AIs would able would be able to do things that were sort of formal so coding and mathematics that seemed like a like a barrier that was difficult to overcome and now this is an area in which the models are extremely strong perhaps this these are the strongest capabilities of models and so things are moving quite fast and yeah this things can change our capabilities can change quickly. Yeah, exactly. And also this this side of the capabilities can unlock a lot of the a lot of the other ones precisely and this is the clearly the bet of these companies and these companies are hiring for automating AI R&D engineers like one of the first companies to openly have roles for this like literally called you know automate AI R&D engineer was DeepMind and the other companies are following suit they're giving you know interviews talking about how they want to initiate this recursive self-improvement loop perhaps this year perhaps next here. This is one of the co-founders of Antropic to the Guardian a few a few months ago. And again, like should we believe them that it will happen exactly at that point? Maybe not. You know, I I hope not given that recursive self-improvement is an extremely dangerous path to super intelligence. But the the trend line is clear. They have tens of billions, in some cases hundreds of billions of dollars. Investments are coming in. They are deploying these investments very effectively to automate what they care about, which is software development and I R&D. and they're going as as fast as they can with no regulation and no oversight. So we we can bet against their money, but you know, we we'll be taking a very very risky bet and at some point somebody will crack this. >> Yeah. For super intelligence to be dangerous, it it must interact with the real world, it must change things in the physical world. And I think what we're seeing is that these models perform extremely well on benchmarks. they perform they they quickly saturate many benchmarks such that we have to make up new benchmarks for how smart these model for measuring how smart these models are but what we don't see at least not yet is evidence of strong evidence of economic impact so job losses GDP growth unemployment numbers that sort of things maybe interest rates what do you think accounts for this discrepancy >> I think this is much of a feature of the human economy rather than of the AI itself. So in many cases like there's and I think this is also why uh a lot of people have a a model of how things will go towards super intelligence where it's some kind of like gradual change where at first we see the AI only in toy environment then we see the AI doing very productive things in a few settings then we see them deployed across the economy all humans essentially like replaced out of jobs and then eventually we die out as a species either because the you know AI just doesn't care about us or the economy is now just run by AIS for AI. So there's, you know, no no food, no entertainment, and no no medical help for flesh humans. And while I think that's possible, I don't think that's what we're seeing. And that's um I think it's it shouldn't be the main expectation because ultimately there are a lot of rules, you know, we we forget this on on AI because there's no regulation, but a lot of rules that constrain how things can be done on the human economy. For example, I expected a lot of current systems could automate a lot of entry- levelvel jobs. But in a lot of areas, these jobs are protected by uh unions. They're protected by regulation. They're protected by cultural norms where people just want to have a human colleague instead of AI systems. And know I think some of these things are good and some of these things are things that are, you know, it is valuable to maintain human culture even in the face of AI. some of them are you know not necessary and not relevant for extential risk but fundamentally I think the what we're seeing is that the companies especially inside uh their development they have extreme powerful AI systems that could already disrupt a lot of things and we shouldn't read too much into like the adoption on whether these capabilities are not there yet I think the capabilities are there in a lot of areas and simply adoption will always lag compared to capabilities so we should be more cautious not less. We should always expect the the the frontier the the most powerful assist systems are much further than the ones that are deployed at scale in the economy. >> Tell me a little bit about how you talk about these issues with politicians and members of parliament. Perhaps we can start with how this works in the UK. What do you say to them? How do how do they react to this? because you're coming to them with with big propositions and you're you're potentially making big asks of them. So, how do they react? What what do you say and and and what do they say in return? >> Yeah. So, kind of for for background, so we started um at the end of 2024, so a little bit over one year ago, with a very focused and clear theory of change, which was most people have never heard about this problem. You know, we've had top experts, Nobel Prize winners, CEOs admitting that AI can cause human extinction. We have had multiple of them sounding the alarm about super intelligence. We had, you know, letters like the FLI Paul's letter, you know, even before that showing that many, many top experts across the field are concerned. But this just had not reached the vast majority of people and the vast majority of policy makers. It's not that they heard about this and didn't believe it or thought this was ludicrous. they just simply never heard about it. You know, we we should never o uh should never overestimate how much reach, you know, one single thing has, especially in the AI field, which is very often very much a a bubble of people that know each other. And so what we were set out to do was to we want to systematically inform every single lawmaker starting in the UK and then in other countries about the risks about where AI is going and about what the solutions are and make one simple ask of them uh support a public campaign so other people know that you know so you build common knowledge so you can show others know not only you care about this issue but you know about this issue and you know if you're a politician it's okay to look into this issue. It's not something weird. It's something that is happening that many of your colleagues know. And when we started, a lot of people told us it's, you know, including in in politics told us it's going to be impossible to get lawmakers to support a campaign that says the word extinction risk. Even though this is what Nobel Prize winners and even the ICOs have said or they will never understand super intelligence. This is too far. You know, it's talk about something else, focus on something else. In practice, now we've done over 150 meetings and one year later we have more than a 100 lawmakers, the largest political coalition of this size on the planet on this issue. Recognizing extinction risk, recognizing that super intelligence poses a national and global security threat and calling for binding rules on this on this most powerful AI systems, a targeted regulation on super intelligence. And I think the, you know, there is no secret sauce. The the the secret is to be honest, to be clear, to find a way to express all of the things that are happening with AI, which are a lot and some of them quite complex in clear, informative, and simple terms. And also just to listen, to not go in with assumptions about um about, you know, exactly what parts they will find interesting, but like see where are they coming from? Have they have they heard about AI before? You know, in some cases, they will not. Those are few cases. In most cases, they will have heard of AI, but they will have likely never used any of the most powerful models and they will just even not be aware of what the companies are building. They might be thinking these companies are just building some chat bots and the chat bots are a bit like a more interactive Google search where you can find some information and I think you know this was the front-facing product a few years ago. This is not the case now, but most people don't know. Again, you know, most people don't use AI on a daily basis. They don't know what these companies are building. And so it's about explaining clearly and straightforwardly what the companies are building, showing the facts, showing the expert concern, showing the plans of the company, showing how already we are seeing that a lot of theoretical concerns that you know we're in the field of AI since its inception essentially since Alan Turing are becoming real and happening in the real world and in testing things like AI models that refuse to be shut down. AI systems that are now so competent at hacking that they can escape their environment and hack out if they cannot solve a task. Sometimes threatening to blackmail or even threatening to kill humans in test settings. And of course this is still happening on a fairly controlled scale. The AI systems are not smart enough. But I think an very important thing is once people start seeing this they quickly connect the dots. And most people, including most politicians, do not want a word where humans are overpowered, replaced, or exterminated by super intelligence. And they very, very quickly connect the dots from like where things are going now to where they will go in the future. And they overwhelmingly support keeping humanity in control and putting regulation on this dangerous technology. >> How useful are demos when you're meeting with politicians? I I could imagine that just showing up with the latest most agogentic models, showing them, synthesizing documents, and putting together a report and, you know, doing some work that the politician might have had some of some of his or her employees do before. Uh that that might be impactful because to some extent seeing is believing and you seeing an AI use a desktop or use a laptop is is that impressive to people? I think a lot of demos rely more on like the communication is more important than the demo itself because in many cases you know ultimately because of what we discussed before right now the AI systems are at their most powerful on the domain of software development and using code and fundamentally code just doesn't look that impressive on a computer right it's just a bunch of text on a computer that somebody you know like many people won't even know what exactly it means and like they're doing extremely powerful things with that, you know, like with code you can essentially access and operate the like a large amount of the modern economy and a large amount of like physical infrastructure as well. You know, you can you can operate a computer like we've seen the the whole cloudbot and moldbook thing recently. But that just like on its own just looks unimpressive without explaining what this actually means and what this actually is doing. So I think what really matters is to just first of all simply show the facts. Now we have an overwhelming body of of evidence and a body of warnings from experts that are just undeniable. Like this is not, you know, this is not AI in the early 2000s where there were like, you know, a few people dreaming of AGI in secret. Like we have again Nobel Prize winners, even the ICOs themselves acknowledging literally that AI position risk on par with nuclear wars. Uh we have top experts sounding the alarm. we have some of these examples of where AIs are already misbehaving and also in in their experience they're also starting to see more that in general there is a great unease in the public about AI due to a lot of you know other things that are are happening you know there we we see the the deep fake scandals we see you know while we're not seeing it in in the statistics sometimes like there is definitely unease about the effect on jobs also because of the not stated stated goal of many these companies which is to just automate most tasks. So it's the the goal is quite clear you know even if the statistics are not and and so I think fundamentally the most important thing is just to break down the complex and sometimes you know jargonfilled things of the AI field and you know demos can help but they only help if you're actually explaining what they mean because code on a on a screen doesn't mean much unless you explain what it is. Mhm. And so when you talk to politicians, what are the types of questions that they're interested in? You mentioned having an open mind about what they might be interested in. And so I I imagine you get a range of questions. Uh is what type of questions are you getting? And and is there a difference between people from the left and and from the right? So actually this issue is extremely bipartisan and so well know we're not seeing and also you know recently I testified to the Canadian House of Commons and also there like the I didn't really see a difference between questions coming from the left or from the right. I see a lot of uh concern after hearing about the the the risks. I think a lot of people are already uneasy about where AI is going. you know they're starting to hear some of these warnings also from other sources you know from from TV from people like Joffrey Hinton that has been a you know great communicator about this but they want to understand okay you know what's happening what are the companies building um you know are risk as big as they seem and you know the unfortunate answer there is yes they are and sometimes they're bigger than they think they are and and then I think quickly the the crucial part is about it's less about understanding the risks I think the risks they understand very well. In many ways, the the case is obvious like if we build systems that are smarter than us across the board and we cannot control them, we are screwed. This is a you know, just a fundamentally terrible idea. And I think there is like many many very good human instincts that all point to this. You know we have a we have uh kind of two like many many years of of human history reflecting on this issue and seeing like if you have something vastly more powerful than you that you cannot control you are in trouble you should just not put yourself in that position. So that part is very clear where the where the questions often come is okay but what can we do about this and I think this is also this is also where the AI companies are shifting where they're lobbying rather than focusing on denying the risks it's becoming harder and harder for them to deny the risks and keep people in the dark they are focusing on spreading a feeling of impotence a feeling of nihilism of you know nothing can be done anyways and know we need to but in any case we need to race because of some excuse you know there will there was will always be an excuse of the day and so I think more of the questions are are I you know I understand this sounds like an extremely big risk like obviously we shouldn't do this but what can I do what can my country do what can I do in my role is it really possible to prevent this like what can we do about this I think that's a good place to be in because you know it's it's going to be hard you know I'm not going to mince words like it's going to be hard to prevent super intelligence and build a different path. But it's very important to move from the learning about the risks to actually thinking what action can be done to keep humanity in control. >> And so if you are a UK politician or a Canadian politician, maybe you're thinking, well, this seems like a big issue. This seems like it could change the world, but this is mostly about the US and China. We are not really at the cutting edge of this technology. And so what can we do and what do you say to that? I know you've you've been thinking about uh developing a strategy on this. >> Yeah. So I think it's there there are like a few a few stages. First of all, as we've seen and you know this was what our entire theory of change was predicated on. Common knowledge makes things move very fast once it gets going. Like right now the biggest bottleneck I think also in the you know in the US and likely in in China and other places is that just most people and most especially most politicians just do not know what the risks are do not know the companies are building super intelligence and do not know what the experts warn about and until that changes every like doing things will always be very hard like ultimately people need to understand that super intelligence is being built. This is the goal of the companies. It's not just chat bots. If it was just chat bots, you know, we you know, I wouldn't be doing this job. I I I love technology and I think technology is very useful. I am not, you know, I think there are some, you know, social issues with chatbots, but I'm not concerned about the end of humanity from chatbots. I am concerned like top experts and even the CEOs are about the end of humanity from AI systems that can overpower us and can replace us at at all tasks. And so the first step is for them to help build this common knowledge at home and abroad and information can move very quickly. This these cascades can happen very quickly. So this is why for example the you know our first ask in the UK is will you publicly support our campaign where they recognize the expert warnings recognize super intelligence specifically you know it's targeted it's focused as a national and global security threat and call for uh oversight and for regulation and the more this happens the more their colleagues the more people outside the more the public will see and look like wait a minute this politician is now talking about this. Wow. I'm, you know, I'm going to look it up. I'm going to Google it. Okay, it's real. This politician is talking about this. We have Nobel Prize winners warning about this. We have CEOs warning about this. Like, what the hell? Like, like it's actually happening. Like, you know, this is not just, you know, something I I dreamt up. Like, I I need to do something. And the more this happens, the more this can snowball. You know what what we're seeing now is now that we have all our 100 lawmakers, you know, just in in January, like just less than one month after we reached 100 lawmakers, we had two debates in the House of Lords about super intelligence risk. one about what are the threats to the UK itself to UK national security as the UK intelligence service MA5 warned about the the threat from autonomous AI systems to national security and one about should we have a international moratorium on developing super intelligence and this is the next step we need to have this national conversation we need to have politician discuss with this with each other across the political spectrum about what's happening what can we do about this this builds knowledge this builds the ability to have a bigger coalition and ability to actually act and then it's true as a as an individual country there are limits to what a country like the UK or Canada can do but this is much more than most people think and like often much more than some lawmakers think because one you know the UK is a major country it's a G7 country it's one of the P5 nuclear powers it's a close US ally you know one country is starting to openly talk about this issue can quickly lead to a domino effect with other countries realizing okay you know this is not you know this is not something that that is just going under our radar it's you know fellow major country is discussing this they're taking measures at home should we look into this as well this happens over and over in history with policy measures where like one country starting to champion an issue will quickly lead to a coalition of others looking into it and can quickly move from nothing is happening at all to there is there is major action happening around the world including with the the US and China and you know I think especially the UK can play a key role in being a champion of this issue discussing with the US discussing with other like-minded allies and build a a coalition that clearly says super intelligence is a national security threat to each of us it's a global security threat we will take measures to make it's not developed at home, we will take measures to make sure it's not developed abroad. We don't want have to have the enemy at home or abroad. If you want to join, you know, we can join in an diplomatic compact where we make sure we all monitor each other and make sure that this is not happening. If you don't want to join and you're still developing super intelligence, even though there is this major coalition of countries saying that this threatens our national security, we will take this very seriously. We will take this as any other major national security violation and we will respond in kind you know with pressure diplomatic pressure economic pressure in extreme cases even more and you know this has worked with its limits with nuclear nonproliferation you know the US does not tolerate uh countries that are developing their nuclear arsenal that could threaten them illegally we should get to the same stage with super intelligence and once we get there we're going to have a stable situation where we can make sure it's not developed anywhere and it doesn't threaten humanity. There's a category of objections here which uh we could call um but others will do it or but others right so if you're a UK politician maybe you think well if we are not at the frontier of AI maybe deep mind just leaves the UK for the US and then advanced AI is developed in the US instead if you're a US politician maybe you you think that if we don't push the frontier on AI in the US well then China will do That is sort of a very basic counterargument or objection here. What what do we say to that or what do you say to that? >> Yeah, I think two two things. One is what we're calling for is not to affect the majority of AI development but is to focus on super intelligence. So there's a lot of you know beneficial AI users especially specialized AI systems. You know you mentioned deep mind things like AlphaFold. This is not what is in scope here. You know Alphaold is a model trained on protein data. It's you know it's it's focused on a very narrow task which is to figure out how do proteins fold. It's very helpful for science. It can have its own risks, but these are not the same risks that we're talking about with super intelligence. And that's not affected and should not be affected with super intelligence like fundamentally there is a there is a hard point which is if you develop an AI system that is smarter than you that is you know the equivalent of as some call it a a counter of geniuses in a data center and you don't control it you are creating an adversary. It is just not in your interest to create an adversary that you cannot control. you know, no, you know, if a country saw uh dropping, you know, next to it the next day on an island with like 50 million people that are geniuses. They are immoral. We don't know exactly what they want and they are plotting to take over the economy. This is this would be like the number one national security threat for every single person in the country's military security services government the next day. you know much like much like in many ways much smaller uh terrorist networks become very quickly a number one threat to deal with. So kind of fundamentally there is no gain from there are gains on the way but there is no gain from getting to super intelligence. The only actor gaining is the super intelligence itself. But regardless, I think it's important for countries like the UK or for middle powers to absolutely take some principal stances without crippling their AI development and economy. And some of them are every country has a duty and obligation to protect its citizens. You know, the UK doesn't let people in the UK develop biological or chemical weapons even if somebody else was developing biological and chemical weapons abroad. So a clear red line of no super intelligent super intelligence development in the UK and you know social go for Canada for the US and so on and internationally where I think we will then need more measures beyond just uh ban on super intelligence we will need ways to monitor this we will need ways to monitor precursors of super intelligence if somebody is doing it surreptitiously commit to to coordinate with allies and see what is a way that we can build an agreement here that doesn't us, but also that makes sure that we can oversee and enforce this. And of course, there there is a trade-off like this will cost some development in the short term. But most security trade-offs look like this. And you know, we have done we have done the correct ones in the past with nuclear weapons. You know, the reason why we're, you know, here having this podcast is probably to a large extent because the US was very clear and didn't let the majority of countries on the planet just develop a nuclear arsenal that would have led to a completely unstable world system where one accident, one crazy ruler of a country or you know one terrorist group getting access to nukes could have led to a full-scale global nuclear war. We have a very limited number of nuclear able countries. We have limited stockpiles. We have the US actively preventing new ones from having this and this is why we in practice there are many issues with it but we have not seen a nuclear exchange since World War II which is pretty good. >> Mhm. You've talked about politicians and engaging with politicians politicians and getting them to support the cause publicly. What about the public here? What role does the public play? How important is it to have a a broader movement where people are informed on this topic and can sort of engage with it? >> Yeah, it's extremely important and I think you know much like lawmakers the public is being kept in the dark. The majority of the public just doesn't know where AI is going. There is a general unease about AI but they just don't know and have not heard that companies are developing super intelligence and this is the level of risk and kind of similarly with lawmakers and and with others the first step is to help the public understand you know let the public know and this just means repeating this information over and over reaching people where they are at reaching people where you know where they get their information from where they get their their information channels from their entertainment from and so on and help them you know let them know I think we have a fundamental duty to like let people know about what's happening because this is being done to them without them even being aware or without their consent and this is you know we have that's also what we do at Control AI uh so far you know we've reached hundreds of millions of people with with our content so far and we then find people that are interested in doing more about these risks we provide them with uh weekly newsletter and call to actions to help them act. The important thing is not just help them understand but you know just give them an a feeling of of great risk without giving them any way to do something about this. What's important is to also empower them to act and and I think the the public can play a key role because yes you know not everybody can dedicate their entire life or their entire career to this but a lot of small actions do help and there's a lot of again fud fear uncertainty and doubt from companies and kind of in some ways from the modern world at large about you are powerless as an individual. there is nothing you can do. The institutions will never do anything. Nothing ever happens. And this is just fundamentally false and it's a self-fulfilling prophecy. If you think you can do nothing, you will do nothing. And if you do nothing, nothing will happen. And then we will get into more worse and worse crisis. But vice versa, small actions like contacting your lawmaker, just letting them know, hey, I care about this issue. This is what the experts are saying. I want you to take a clear stance on super intelligence. I wanted to take a clear stance to protect humanity makes a massive difference. This is something that people can do in you know 5 minutes at Control. We've made tools that make it very simple to do this and so far people have sent over 150,000 messages asking the lawmakers to ban super intelligence in the US and more than 10,000 in the UK and you know this is only growing over knowing this and helping people take more and more engaged civic action to deal with the problem. You mentioned that there's a general sense of unease about AI in the public. Do you worry that this general sense of unease might be sort of pushing in the wrong direction if if if you're communicating to the public about risks of extin extinction caused by super intelligence. Maybe that's mixed up with other worries about AI, worries about jobs, worries about, you know, this AI slob on my feet is unappealing. And there's if if there's a general sense of unease, maybe that uh is not pointed in the right direction. Does that worry you? >> So I think it doesn't worry me, but but it just reminds me that I think it's very important to to tell people the truth. It's important to give people the tools to act and to understand the situation. So then they will do what they believe is best. And I think at the moment they there is a general unease. But again, most of them if you if you survey them like once you inform them about the facts like the extinction threat from AI that experts warn about and or even just give them like brief statements from the companies just describing what they're building, they become extremely supportive of things like banning super intelligence. So once they know about it, they will want to act and they will support it. But the main issues that we're seeing is that right now this is just not particularly salient. Again, most people just don't know about these risks and don't know about what the companies are building. This is changing. It's been definitely changing in the past year. You know, we've been working to change it. But also, I think there's just more information about this, more and more people speaking out just, you know, just yesterday, I saw that Hugh Grant said that AI poses an extential risk and this is extremely worrying. So, you know, we're just seeing this across a society. you know finally this information is is is propagating but and I think what's the kind of the the biggest challenge is just we need to give them this information on time otherwise I think a lot of this concern will either like fizzle out or get get dispersed across a myriad of issues so I am I am not worried but I do think people should just more and more inform the public about the full scale of the risks about what's happening and show them that there is a way forward rather than just building nihilism or like not giving them the tools to do something about this. >> On that note, if control AI is is successful, as successful as we could hope, uh what does the world look like in in 2030? >> Yeah. So, if we're successful in 2030, we have a ban on the development of super intelligence across all major countries. This is both enforced nationally. These countries have a law in their own country saying that you cannot do this at home and they have an international agreement among them that says we are working together to make sure this also doesn't happen abroad and with countries that are part of this agreement you know they uh share information monitor each other and they make sure that nobody is developing super intelligence and with countries outside of this dis this agreement or with rogue actors they are treated like we treat other rogue actors and so we make sure that they don't get access to the technology necessary to develop this. They are pressured to join the agreement or to just terminate their development programs. And you know, if we're actually successful, we're in a stable world where no major country is pursuing super intelligence. Also the fundamentally a very important thing is we're in a world where no matter what what's on paper, you know, no matter what's on on the on the text of the law or in a you know text of an agreement like world leaders and the public deeply understand the level of risk and deeply understand what's at stake and this deep buyin is fundamentally the the end goal of control AI. We want to help people understand what's happening so they can make their own opinion and understand the level of risk because ultimately to deal with this issue it it will not be dealt with while by like one small amendment in a law that then is forgotten. Like if if world leaders or at least if a significant percentage of them do not fundamentally understand what's happening and understand what the risks are, they will make the wrong trade-offs when they're faced with a difficult decision. there will be many hard decisions to make and you know this deep buyin is needed and at the same time you know while super intelligence development is prohibited we have a lot of AI systems that are you know many ways much more advanced than the ones we have now but they that they are just they're narrow specialized used for specific domains and I think this will still be a world where society is fundamentally transformed the economy is fundamentally transformed like you know even if we press the magic button and like AI development doesn't proceed at all in any area from today, I think society will be fundamentally transformed from the things that we talked about before. Like there's a big lag in adoption. There are a lot of activities that we do right now that AI is already can do better than us. And like that's just a very different society with different trade-offs. But the the what the winning work looks like is one where nobody's developing a technology that can wipe out all of humanity. We have a lot of powerful AI systems doing a bunch of other things in narrow specialized domains. They are scanning uh pictures for cancer. They are solving problems even more ambitious than protein folding. But they're firmly under human control. They are not uh they're not so general and so smart that they can overpower people. And we are using them as tools to advance humanity's growth and prosperity rather than as a new species that takes over the planet and needs to find a place for us. Is there say say we fall short of a full global prohibition on on super intelligence is there a world in which we model through with more limited regulation like say we we have banned recursive self-improvement within the leading companies or maybe we have transparency regulations or mandatory evaluations of the models pre-eployment something like that it do we have a fallback option here or are we sort of going straight for for the big win >> I think we need to go straight for the big win. I think one important thing is there are uh there are some things that are fundamentally important and some things that are you know different ways to implement them. I think there are many ways to implement this restriction, this prohibition of super intelligence. But fundamentally, if we get to a point where we have AI systems that are vastly more capable than us, vastly more capable than all of humanity combined, they are have replaced people across the economy. They make decisions. They are autonomous. They run around utilizing physical and digital infrastructure. This is a point of no return. We are in a world where even if we want to go back and even if we want to retain human control, we cannot. You know, we're facing essentially a a major powerful force that now controls our economy and we just cannot afford to get to a point of no return. There is no way out from there. So like that's the non-negotiable red line. How we get there that's you know in the end the goal is just to to get there and to make sure the humanity stays in control. So the ways to implement it are you know up for discussion. You know we have some ideas but they matter less than the end point and same with the this deep buy in the the fundamental win condition is that there is deep buy in about understanding how big the risks are understanding the super intelligence under current conditions is a terrible idea for humanity. If enough people have this, we have won even without any specific law because those people will make the the right decisions collectively they will and know when they don't others will explain to them why that's the wrong choice. Vice versa, if we have just one law that you know passes accidentally that does say you know the all of the correct words but people don't understand the risks will will still be in big trouble because enforcement will be weak and leaders will not know what to do about this. >> Yeah, it's interesting how it what matters in the real world is not only what what's on paper. It's very much about what people know and what they think about and and how they act in the world. And so sort of information is also it has any any written text has to be accompanied by an informed an informed sort of bureaucracy for it to matter. I think >> exactly and an informed public you know the public checks check checks the bureaucracy and make sure things are on track. >> Yeah. Yeah. How much how much does timing matter here? So maybe we expect super intelligence in say 2 to 5 years but what if it's more like uh 10 to 20 years which actions are robust to shifts in in in in timing uh of of super intelligence. I think this is why another reason why I like this theory of change is that it's very robust to any timing like fundamentally this these are the steps that human civilization needs to take to deal with any major risk. There is no meaningful and stable way there there might be ways to like delay a risk but there is no meaningful and stable way to deal with such a level of risk that affects all of human civilization without going through these steps. going through the steps where a large fraction of the public understand understands the risk, cares about it, asks their governments to intervene, scrutinizes companies, scrutinizes governments and, you know, demands intervention. A large fraction of decision makers know about this, understand that this is a threat to them. You know, even before being a threat to their country, it's a threat to their person. It's a threat to their their families. You know, super intelligence taking over. I think sometimes we we we we stay very much on the abstract and like that's also what the what the companies do but it's it's very concrete you know if human extinction means I die and you know other people die too that are exist now today and you know most you know fundamentally most people do not want this and if they understand this deeply they will fight back and fight for human control. So this uh so fundamentally the theory of change is the timelines don't matter too much. You know maybe there are some situations where we just don't have enough time for the public and governments to wake up and you know those are words where there is not much to do but obviously we need to do as much as we can and as fast as we can and if we have the luxury of having 20 years we will still have to go through this entire process. we will still have to go through a process where the public becomes informed, where decision makers are informed and they can act. And so the steps are exactly the same, what matters is only like how quickly this needs to happen for things to change. And while you know I would like for us to have 20 years but this is definitely not the prediction of the majority of of experts in the field and like you know nowadays there's even the joke that like so-called long timelines are you know not next year or the year after even from some of the biggest skeptics. Yeah. Uh we talked about recursive self-improvement a couple of times here and this is one of the factors that could really speed things up and so I think it's important to sort of get a grasp on where that project is and we're speaking in in in early 2026 and it seems like people are it seems like coding models are getting to a point where they're speeding up the research and development internally at the companies. This is hard to get sort of actual numbers on. So this is this is more about listening to what people who work at the companies are saying and playing with the models yourself and and seeing the benchmarks how how the models are performing on on coding benchmarks. So where do where do you think we are on the path to recursive self self-improvement? Is that something that's that's actually beginning to happen now? >> Yeah. So I what I think definitely the companies are trying extremely hard to get there. U again it's it's very it's very clear they know hire roles specifically to get AIS good enough to automate AI R&D. They advertise this very openly. they, you know, openly talk about a moment where we can uh give up control to the AI that improves itself and then hope that it will be benevolent or, you know, will keep us as as pets or whatever the the the cope of the moment is about uh human relevance after AI gets there. Um, as you're saying, it's it's difficult to know exactly where they are internally. You know, what we can see externally is that these AI systems are becoming very competent at software development, which also has a another side to it. This is why we are seeing these systems becoming very competent at hacking. You know, hacking ultimately is is writing a malicious software and using it to do attacks and as as long as these models become more powerful at developing software autonomously, they will also get more powerful at hacking. Um so hard you know hard to tell exactly where we are. What definitely is happening is the companies are investing billions of dollars into making this happen. Uh they might you know they might stumble on a breakthrough. They will accelerate things. It might take longer than they expect, but again, right now without regulation, we're just facing this very dangerous gamble that of somehow hoping that these companies fail at their stated goal where they're investing billions of dollars and literally the the smartest people uh some of the smartest people on the planet on this task. So I think this is why it's very important to have you know a very clear red line of no development of super intelligence and b to restrict and monitor precursors like recursive self-improvement. You know even if somebody is denying that they're developing super intelligence. This is not the case now but it might be the case once we have regulation that prohibits this. We need to look at things like well are you attempting to get AIs to just fully autonomously improve themselves. you know, if you're doing that, obviously that's a way to get to super intelligence without saying you're doing that. And that's not okay. You know, much like we have, you know, with things like certain drugs, we monitor the drugs and we monitor the precursor chemicals, kind of the chemicals that you can use to make those drugs. You know, recent example in the in the US that is very salient is restrictions on fentinel precursors, you know, restricted drug and restricted chemicals that are used for that. We need to monitor these precursor capabilities for super intelligence. the kind of things that we will we would see before somebody can get to super intelligence and the kind of things that can accelerate drastically beyond human control the development of super intelligence. >> We could talk about a spectrum all the way from autocomplete in code. So very simple autocomp completions all the way to fully automated end to end researcher development and development internally at the leading companies. is that how do we how do we determine what is a precursor on that spectrum? Because it seems like okay, autocomplete is just a faster way to do what you were already trying to do. But then it's it's sort of not easy to see where there's a there's a there's a red line being crossed here into uh now you're speeding up the development of of uh frontier models by by using this process. How do we how do we determine this where what what what's a precursor? >> Yeah. So the so I agree with you that for example you know just coding assistance would not fall in this category. Um for me the thing is in you know the the power of the law is that the law is not code law doesn't have to be written in exactly you know the exact specification that can be uh gained. You know this is the case also with existing legislation things are like you know chemical and biological weapons. The definition is quite broad and then it is on the burden of proof is on on companies especially when they are checked to show we are not doing the the bad thing. You know most laws don't exactly specify an you know an exact chemical or an exact thing. They specify a broad class of things and activities. For example in this case I think clearly it would fall in the category uh fully automated end to end. You know, you could use words like substantially uh you know, AIS are substantially involved in the to the design and development of other AIS, substantially involved in the design and development of AI chips. And these are in practice things that and a judge can determine like you know judges are pretty good at at doing at doing this kind of stuff like do it all the time with much murkier legal cases elsewhere like software is not a special category. It's just a more new category and you know most you know or in the same way where we have um anti-money laundering regulation on banks like you could make similar arguments where like well but what you know what exactly is the line for moneyaundering is there an exact amount or like you know is it an exact profile and generally it's a it's a broader assessment first of all it's a normative principle of you should not intentionally do it you know you should not intentionally try to make AI fully automate AI or R&D you know if you're just using copil pilot that's fine like you know it should be fine if you're starting to develop AIS and you like you let them run autonomous autonomously as agents and you check once a week that's already starting to to to cross the line but fundamentally once we do have these rules in place the incentives change and it's in the it's in the interest of the employee of this company to be protected like not even risk falling a foul of the law like right now companies are being extremely reckless because there's no consequence for recklessness with consequences. You know, even if if a company tries to push them to do this, many employees will think like, why should I risk this? You know, why should I risk this doing this very dangerous thing? I will be cautious and, you know, stick to copilot and not to, you know, trying to do experiments like fully automated, you know, agentled AI R&D. >> Mhm. If you have one company or perhaps a a a number of companies in one country developing super intelligence, it it seems like that company or that country will quickly acrue a lot of power and so they will be over they will be able to overpower perhaps other countries or other companies. And so there there are two worries with this power concentration. One is that it will happen within one company such as say Google Deepmind or OpenAI on Cropic will become will become much more powerful than other companies and perhaps uh more powerful than other countries. There is also the worry that if we set up a system in which we are able to control AI development that in itself is a form of power concentration and so is it should we worry about this the the ban on super intelligence turning into a a sort of extreme form of of power concentration where no one is allowed to to develop AI in this way because we have a a an international system that prevents So all government regulation is some form of you know restriction on absolute human liberty. But this is the is the evolution that we have done with our institutions in the past you know 2,000 years or more. Like we are able to achieve greater and greater feats because we also have decided to like pull some of these individual rights in things like governments. you know like we know most people do not object to the fact that the government has the monopoly on violence and this is a very normal thing of modern society which was a normal you know 1000 years ago or more. This has made our of course there is a trade-off where you have somewhat less individual liberty. You cannot just kill your neighbors or uh but uh you know this leads to societal outcomes that are better for everyone. And I think in this case the like in the two scenarios the one where super intelligence is developed is overwhelmingly the scenario where there is enormous power concentration compared to the other one. The the important thing with with preventing the development of super intelligence is this actually reduces the amount of power concentration in the world like no entity will be allowed to build an AI system that can literally overpower the government, the military, the police or the entire planet which you know as you said is sometimes how these companies describe their systems. You know, recently Max Seagmark from FLI was was saying in a press conference that he heard some of the AI executives talking about how they want their AI systems to overpower the government and to be powerful enough to overpower the government. And this is true like you can kind of read between the lines in some uh some of the blog posts and they definitely say this more in private and this is a unacceptable to governments bacceptable to the public. This is a much more highly power concentrated world uh one where potentially one company although with super intelligence the likely situation is soon just one one swarm of AIs you know no human will really have control be like one swarm of AI that has the majority of the power in the world into can do whatever it wants. So yes, I think all kind of uh we need to be honest. Obviously all regulation will reduce some amount of freedom, but these are trade-offs that we make across many areas. You know, we don't allow private companies to build nuclear weapons. We don't allow private citizens to build nuclear weapons. Some, you know, sometimes even people that do it as a as a fun science experiment in the backyard, they go to jail because the US government makes sure that people don't just develop enriched file material in their backyard. And it's a it's trade-offs that we make to then have a prosperous society where we are secure and we maintain our you know national and global security. And yeah, in the in the world where super intelligence continues to be developed in the you know in the least bad scenario for a while we'll have likely not a government but one company essentially taking over and then quickly the AIs of the company taking over everything else and you know leaving both the humans in the company and uh most governments around the world and most people around the world completely disempowered at the mercy of these AI and this is the in some ways the worst form and the biggest form of power concentration we would have ever seen in history and we need to avoid that. >> This is a point you've made a couple of times that we if you raise the super intelligence and actually get there you will be handing over power to the AIs it will not make you powerful but internally in the companies they must disagree right they must think that they will empower themselves or maybe they will empower their country or maybe even sort of optimistically they think they'll they'll empower the entire world by developing advanced AI. So what's the what's the disagreement there or how do you explain if you're if you work at say open AI or anthropic do you think they think that they will lose power are they just mistaken so you know I I think a lot of employees are mostly uh you know doing their doing their job and it's an extremely highly paid job and it's convenient to look the other way. I think at the at the executive level like we we can see there are public statements by some of them like some of them are just fundamentally transhumanist and you know it might be kind of hard to believe but there are public statements from them one of them is one of the co-founders of entropic to the garden where he quite openly talks about u uh having this int intelligence explosion after which they will hand over control you know he's not talking about him personally having control he's talking about handing control to the AI. that this you know I I don't think this is a you know I don't know if this is the majority of executives in the companies but there are some prominent ones that essentially just believe we should give the keys over the planet to the eyes and let them do what they want and then either hope they will be benevolent or in some extreme cases the case for example of you know Richard Saturn and I know probably unfortunately non insignificant fraction of people at these companies even if they're not benevolent these these eyes these people consider them as next stage of evolution of intelligence or some other post posthumanist ideas of of humanity is just a transient phase in history. You know, I completely reject these views. I am human. I care about being human. I care about fellow humans and I don't want us to go extinct. But I think some of these people genuinely think AI is taking over is not a bad idea. I think some other ones as you're saying might be deluded or power hungry or like willing to take this enormous risk on behalf of everybody else with the sliver of hope that they individually will be able to keep control. But again, I I think the the views are quite inconsistent and in the end what matters are actions and not words. Like a lot of these people are people that that have recognized like you know independent experts recognize that this poses an extinction risk. they are still willing to go ahead, you know, maybe because of a momentary flash of glory before the AIS take over from humans. Maybe because they generally care more about AIS than humans. I think we have seen this sadly more and more and it's aided by you know enormous exposure to AI systems and in some cases it might even be psychosis uh and other cases where they believe they believe like yes if others do it they will not have it but if I do it I might keep control and this small chance of having total domination over the planet is worth risking to sacrifice the lives of billions of people on the planet. As a as a final question here, could you tell listeners what they can do to help control AI achieve its mission? >> Yeah, absolutely. So, first of all, you know, if you're listening and you care about this issue, contact your lawmaker. We have tools that make this very, very easy in many countries around the world for you to contact your lawmaker in just a few minutes. Go to campaign.controlai.com and you will find everything there. Write to them if you have time. Call them. It's very effective to call them. they get much fewer calls than emails or forms and especially in the US they will always note down and they will really take seriously what you tell them on a call and they would keep it into account. Also we are expanding um what you know we've talked about with with gas in this episode is we have this theory of change that has worked very well. I think now we have a recipe for making this change happen at scale and so we are scaling up the team. We're scaling up the team here in the UK. We're scaling up the team in the US with some jobs going live soon. So keep an eye out for those roles on contrai.com and you know all other platforms where you can find job postings. We're hiring across a variety of roles. >> Fantastic. Thanks for chatting with me. >> Thank you so much. Great to be on again.