Tom Davidson on how quickly AI could transform the world

Hi listeners. This is The 80,000 Hours Podcast, where we have unusually in-depth conversations about the world's most pressing problems, what you can do to solve them, and what did happen to all those horses after we invented cars? I'm Rob Wiblin, head of research at 80,000 Hours. The last few months have been a crazy time for advances in artificial intelligence. Over the last couple of years, I've become, for better or worse, increasingly confident that the future is going to be shaped in a major away by what sorts of AI systems we develop and choose to deploy as they approach and then exceed human capabilities in the most important areas. That's always seemed like a really common sense idea to me, but it now is becoming pretty apparent to people across society who until recently had hardly been paying attention to this issue. Unfortunately, we've only had two episodes about AI over the last six months due to the substantial lags that come in between conceiving of an episode, finding the right guest, scheduling and recording the conversation, and then editing it and then finally releasing it. But that scarcity of AI content is fortunately about to change. In today's episode, Luisa Rodriguez interviews Tom Davidson, a senior research analyst at Open Philanthropy whose job is figuring out when and how society is going to be upended by advances in AI. Not a simple forecasting exercise. Tom has his work cut out for him. But if you've been wondering when you might be replaced in your job by an AI model, and when AIs might be able to do everything that humans can do for less than what it costs to feed a person and keep them alive, then this episode will help you think about that a little bit more clearly and perhaps have a little bit more idea of what to expect. In particular, Luisa and Tom discuss how long it will take for AIs to go from being able to do 20% of the work humans are doing to being able to do all of it, what underlying factors are driving progress in AI and how much each of those factors is contributing, whether we should expect that progress to speed up or slow down incoming years, how much computer hardware is used to train these models and whether it can continue increasing at the absolutely blistering rate that it has been doing the last ten years, when AI systems might be able to do scientific research and what implications that would then have, and also when we might expect all of this to noticeably increase GDP in a visible way, and what that could look like, and then what new bottlenecks might exist in an economy where AI systems were doing most of the work. And on top of that, plenty of other issues besides Tom's expectations for the future are exciting, or alarming, I guess, depending on how you want to look at them. Regular listeners will have heard me do quite a few interviews on AI over the years, but those interviews tend not to focus on my opinions, or at least they're not meant to. So it's possible that people don't have that much of a sense of where I stand on this personally. In case you're interested, I think the chances that you and I are either killed due to actions taken by AI systems, or that we live to see humanity unintentionally lose control of its future because of AI systems are greater than 10% — greater than a one in ten chance. Looking at surveys and polling now, it seems like both AI experts and the general public are converging on a view that's not too far from that. It's a view that previously seemed idiosyncratic, but now is really quite mainstream. Naturally, if the odds are like that, then it makes figuring out how to safely develop and deploy AI probably the issue of our time, and indeed one of the things that we should most care about from even a selfish point of view, or if we have children, care about from a parental point of view. I'm an economist by training, and I entirely understand how the industrial revolution ultimately raised incomes across generations, and that while factory automation was indeed financially ruinous for plenty of individuals, it didn't then result in persistent unemployment. But nonetheless, despite understanding and appreciating all of that, I am sceptical that there are going to be paying jobs for children being born today, or if not them, that they'll be paying jobs, at least for their children. And we're flying by the seat of our pants here and have not really figured out a plan ahead of time about what we're going to do as this technology just completely upends, I think, existing social relations and economic systems. Well, I'll describe the overall situation that humanity finds itself in to be pretty terrifying. The fact that all kinds of different people are waking up now to the risks here does give me hope that we can coordinate to prevent the worst. As you imagine, we'll have more to say to expound on all of that in future episodes, but for now, I bring you Luisa Rodriguez and Tom Davidson. Today I'm speaking with Tom Davidson. Tom's a senior research analyst at Open Philanthropy, where his main focus is on when we might get transformative AI. Before joining Open Philanthropy, Tom taught science through Teach First at a comprehensive school in east London, and then was a data scientist for an education technology startup. And before all of that, Tom studied physics and philosophy at Oxford. Thanks for coming on the podcast, Tom. Thanks, Luisa. It's a pleasure to be here. So, I hope to talk about how fast we might go from kind of ok AI to AI that can do everything humans can do, plus how that'll affect the economy and the world. But first. Yeah. How worried are you personally about the risks from AI? So, about a year ago, I sat down and spent a morning trying to figure out what are my percentages of AI by a certain time, and what's my percentage that there's existential catastrophe from AI. And so I was focusing on the possibility that AI disempowers humanity and just takes over control of society and the economy, and then what happens in the future. And a year ago, I landed at a number that was a bit above 10% for the probability that AI takes over by 2070. Right. Okay. I mean, that's already pretty high. Yeah, that is already very high and much too high. Yeah. I think since then, if I redid the exercise today, I think I'd be close to 20%. God. Compared to then, I think it's just more likely that we develop AI that's capable of doing that by 2070. And I also think that it's just pretty likely to happen in the next 20 years. Wow. Which then makes the chance that it goes badly, I think higher because we have less time to prepare. So both those things, I think, mean I'd probably be at about 20% if I redid the exercise today. Yeah, we'll talk more about a couple of those things. But first. Yeah, I'm curious if there's a type of scenario you have in mind when you're thinking about that ten to 20% that makes you especially worried? Yeah, I think the main line scenario I have is something like in the next ten to 15 years, possibly sooner, we train an AI that is able to massively enhance the productivity of AI R&D workers. So people who are currently working to make AI better, maybe it makes them, let's say, five times as productive. So something like a large language model helps people working on AI R&D in particular, like code much faster or develop better algorithms or something? Exactly. And it means they can work five times faster ish. Exactly. Right. Okay. And then I think it won't be long after that, because AI is then going to be improving more quickly before AI is able to do everything that the current employees of DeepMind and OpenAI are currently doing for their jobs, at least the ones that can work remotely and do their work from a computer. Are you kind of distinguishing between the ones that work with physical robots or the ones that work with, I don't know, the mailroom at DeepMind? Yeah. So if there's someone, I don't know if this is true, but if there's someone at DeepMind who is physically stacking the computer chips into the data centre, then that's a kind of type of physical work which I don't think would necessarily follow immediately on, but I think most of the work is not like that. Right. So I think AI, especially most of the work for advancing AI, is stuff that you can do on your laptop. So first we get the kind of 5x productivity gain. A little time later, we get AI that can do all of the work that people at OpenAI and DeepMind can do. And I really think the gap between those is not going to be very big, and we'll probably discuss a bit more about that later. At that point, I think AI is going to be improving at a blistering pace, absent a very specific effort to slow down, which I hope we really make. But absent that effort, and absent coordinating to make sure that everyone is slowing down, I think like, 1000x improvement in the AI's capabilities in a year. Oh, my God. Is like a natural kind of conservative default. Wow. And so in this scenario, it wouldn't be long before the AI is just far outstripping the cognitive intelligence abilities of the smartest humans and indeed even the smartest massive teams of humans working together. Wow. And when you kind of crunch the numbers on how many AIs there's likely to be around this time, there's going to be hundreds of millions and probably many billions of human worker equivalents just in kind of AI cognitive ability. Is that because there are different AI systems, or because you've got that much brain power being deployed through AI, even if it's only like five AI systems. Or something, it's like running lots of copies of maybe a few AI systems. Got it. So, for example, I haven't crunched the numbers on this, but I would guess that if you took the kind of computer chips that they used to train GPT-4, and you just said, how many copies of GPT-4 could we run using these computer chips? I guess that the answer is, like, maybe a million. Wow. Might be a bit lower, actually. I'm not sure for GPT-4, but I think by the time that we train kind of AI that can fully replace all the human workers at an AI lab, I think that number is going to be more extreme. Right. So I think that by the time we can train that kind of AI, you'll be able to just immediately use the compute that you use for training to run 100 million. Right. Okay. That's insane. And the reason for that is because it just takes, I don't know, like many times more compute to train an AI system than it does to then run them? Yeah, that's exactly right. Okay. One way to think about it is that the AIs are trained on kind of millions of days of experience so that they get to be as good as they are, and that if you've managed to train them for that much, then it's kind of, obviously you can run millions. Yeah, okay, got it. So then we're there. We've got potentially millions of human equivalents of copies of AI systems running and doing the kind of work that humans could do at the human level or better. Exactly. And then what? So then I think it's likely that it won't be long until we're talking about billions of AI systems. It's run of the mill to see 3x efficiency improvements in AI at various different levels of the software stacks. You could get a 3x efficiency in kind of how effective the algorithm is, or a 3x efficiency improvement in how well the software runs on the hardware. And there are these various layers of the software stack that you can make improvements on. So I think it's probably at that point, once you have hundreds of millions of AIs looking for these types of improvements, you're probably going to get very quick further improvements in kind of AI cognitive ability. Again, maybe we coordinate to go very slowly, but this is absent that targeted coordination. Maybe the default. Yeah, maybe the default, scarily. And at that point, I think that if the AIs are misaligned, if they have goals that are different to what humans want them to do, and if those goals imply that it would be useful for them to get power so they could achieve those goals better, then I don't think it's going to be very hard for them to do that, because it's like we've got a billion really smart humans who want to take power. Probably they'll find out some kind of way to do it. Maybe they invent new technology, maybe they convince some higher officials to give them control of the military. I'm not sure exactly how they'll do it, but at that point, I think it's kind of too late for us to be preventing. Preventing AI takeover. Okay. And then by AI takeover. Do you mind spelling that out? Yeah. So I think the thing that matters for AI takeover is that AI systems collectively end up in control of what happens in the future and that it's kind of their goals and decisions that dictate the future path and that it's no longer sensitive to what humans want or are trying to achieve with the future. Yeah. So, I mean, ultimately, I think it does have to come down to physical force, most likely, right? I mean, you could imagine a scenario where the AIs just convince humans and hypnotise them or something, and that's how they take over. But more likely they end up having control of the hard military equipment, and that's what allows them to establish their power and disempower humanity. So a thing that I have to admit still confuses me is just like how we go from, I don't know, things like GPT-4, which even if it sometimes gets super confused and says silly things in a way that's like, oh, you clearly misunderstood what I was asking for. It's really hard for me to understand what the path to that kind of confusion, to a misalignment that's. So, I don't know, I guess just incredibly diverging from human values that the AI systems want to disempower humans. And, I mean, one article I read, I think, that just came out recently on Vox, is this article that was actually making the case that companies creating AI should slow down, should kind of coordinate to slow down, was walking through the case for why we might expect AI to be misaligned. And the example they gave just still confuses me. So the example is something like, let's say you've got a super smart AI system. We've programmed it to solve impossibly difficult problems, like calculating the number of atoms in the universe, for example. The AI system might realise that it could do a better job if it gained access to all of the computers on Earth. So it releases a weapon of mass destruction to wipe out all humans. For example, an engineered virus that kills everyone but leaves infrastructure intact. And now it's free to use all the computer power, and that's the best way it's able to achieve its goal. And I think. I feel silly, I feel dumb, I feel like I'm missing something. How will it go from, like, I want to solve this problem for humans to, like, I'm going to kill them all, to take their resources so that I can solve the problem? Why have we not ruled that kind of extreme behaviour out? Great question. So maybe we can try and think about this system which is trying to solve these math problems. Sure. So maybe the first version of the AI, you just say, look, we want you to solve the problem using one of these four techniques, and that kind of system is okay. But then someone comes along and realises that if you let the AI system do an Internet search and plan its own line of attack on the problem, then it's able to do a better job in solving even harder and harder problems. And so you say, okay, we'll allow the AI to do that. Sure. And then over time, in order to improve performance, you give it more and more scope to kind of be creative in planning how it's going to attack each different kind of problem. One thing that might happen internally inside the AI's own head is that the AI may end up developing just an inherent desire to just get the answer to this math question as accurate as possible. That's something which it always gets rewarded for when it's being trained. And maybe it could be thinking, I actually just want the humans to be happy with my answer. But another thing it might end up thinking is, you know what? What I really want is just to get the answer correct. And the kind of the feedback that us humans are giving that system doesn't distinguish between those two possibilities. So maybe we get unlucky and maybe the thing that it wants is to just really get the answer correct. And maybe the way that the AI system is working internally is it's saying, okay, that's my goal, what plan can I use to achieve that goal? And it's kind of creatively going and looking for new approaches by googling information. Maybe one time it's like it realises that if I hacked into another computing cluster, it could use those computations to help it solve the problem. And then it does that no one realises. And then that reinforces the fact that it is now planning on such a broad scale to try and achieve this goal. And then maybe it's much more powerful at a later time. And it realises that, yeah, if it kills all humans, it could have access to all the supercomputers and then that would help it get an even more accurate answer. And because the thing it cares about is not pleasing the humans, the thing it happened to care about internally was actually just getting an accurate answer. That plan looks great by its own lights. And so it goes and executes the plan. Right. So one, that was really helpful, but I still feel confused about one, why it's so hard to not give it some instructions that are just like, use whatever you need, but don't hurt living things. So I think we could definitely give it those instructions. The question is inside its own mind, what is its goal in the end of the day? So you could give it instructions, don't hurt humans, and it would read that. It would understand that's what you wanted. But if throughout its life, it's always been rewarded for getting an accurate answer to these math problems, it might just itself only care about getting accurate answers to the math problems. So it knows that the humans don't want it to hurt other humans, but it also doesn't care about that itself, because all it cares about is getting accurate answers to this problem. And so, sure, it knows that humans don't want it to hurt other humans, and so it makes sure to not do that in an obvious way because it anticipates that it might get shut down. But its knowledge of what humans want it to do doesn't change what its own desire is internally. So I suppose I understand why you couldn't just give the system an instruction that didn't also come with rewards. Is it impossible to give an AI system a reward for every problem it solves by not hurting anyone? I think that would help somewhat. So the problem here is that there are kind of two possibilities, and it's going to be hard for us to give rewards that ensure that one of the possibilities happen and not the second possibility. So here are the two possibilities. One possibility is the AI really doesn't want to hurt humans and is just going to take that into account when solving the math problem. That's what we want to happen. The other possibility is that the AI only cares about solving the math problem and doesn't care about humans at all, but it understands that humans don't like it when it hurts them, and so it kind of doesn't hurt humans in any obvious way. Oh, right, okay. And so this is a route to AI not caring about humans, but being kind of deceptive. I guess maybe analogy that really speaks to me is something like, if you were to punish a child for, I don't know, having ice cream before dinner, you might get them not to have ice cream before dinner, or you might create a thing where they have ice cream before dinner while hiding in the closet. And it's like, pretty complicated to teach a nuanced enough lesson to a child about ice cream and why they shouldn't have it for dinner. That doesn't have any risk of the lying version. Is that kind of, right? Yeah, I think that's a good analogy. Okay, nice. I think with a child it might be somewhat easier because you're much more capable than them. So even if they ever did try and eat ice cream in secret, you'd have a good chance of catching them. I think the problem gets really hard when the AIs are much smarter than us, such that they could quite easily eat ice cream without us noticing. And then it's really hard for us to give them rewards which stop them from doing that. Right. And something like, we have a pretty good idea how children's brains work. They work kind of like ours, but like a bit simpler. And we have some idea of the ways they're different, and so we can make guesses about the types of motivations that will speak to them. I don't know, maybe it's like we know that our kids work similar to us and that they feel shame, and they'd feel shame if they were punished and want to please us, because that's just like pretty human. And so maybe we have a better sense of how they'd respond to punishment. But maybe AI systems are just like so different to humans that we really have no idea what they're, or at least will have a less clear idea of what other processes that they're using or like, things that they experienced or whatever are like, and what kinds of behaviours those will push them toward. Yeah, I think that does make it a lot harder. Cool. That's super helpful. Yeah. Is that basically the key scenario you're worried about? This kind of, we train AI systems to achieve certain goals, but it's hard to know what strategies they see as fair game. And it's hard to train them not to pursue harmful strategies, and then they eventually get super smart. Maybe they are deceptive, maybe they're just very convincing and so they are able to get a bunch of power and really disempower humans. Is that kind of what you see as the core risk? Yeah, that's right. And I would emphasise that in the scenario as I described it, AI is improving really rapidly as it approaches and then goes through the human range. So when we're talking about this example with the AI that's trying to solve math problems, maybe we are thinking, oh, we'll have a few years with it, trying out this kind of strategy. And then we notice it's kind of doing a little bit of hacking into computer resources, so we tamp down on that. But if this whole thing plays out over just one year, for example, we go from notably below human to superhuman systems. I think it makes the risks a lot more intense. Yeah, that makes sense to me. Yeah. To move us on to your research, then some of the work you've done that kind of almost blew my mind, was actually on what happens when we're able to basically build AI that does roughly what we intend it to do. I think I naively would have guessed something like, the world carries on is normal, but we use GPT-8 a lot in our jobs. But you've looked into the hypothesis that not only will things not stay the same, they actually might change very quickly. If AGI is so good that it kind of causes explosive economic growth, which yeah, in this case, you're defining as the world economy growing something like ten times faster than it has for the last century. Yeah. So, to start, can you help me understand intuitively what it would mean for the economy to grow ten times faster? Sure. So, one way to think about this is to think about all the technological changes that have happened over the last 50 years. Okay. Yeah, that feels like a lot. So, 50 years ago, it was 1970, we'd had very basic digital computers around, but they weren't being widely used. They weren't very good. I don't think the Internet was around. And there's loads of other improvements in manufacturing and in kind of agricultural techniques, medical care. Exactly. And kind of massive improvements across the board in the last 50 years. But probably the most striking. Is it? Yeah, sounds right. And so what explosive growth would look like is that all of those changes, rather than happening over the course of 50 years, they happened over the course of five years. So we're going to get the Internet in five years plus a bunch of other improvements. Exactly. So rather than kind of it taking 50 years to go from these really rubbish, slow computers that you could buy in 1970 to the awesome macbooks of today, that just happens over five years. And similarly, rather than taking 50 years for you to go from those kind of rubbish phones to smartphones of today that also have the Internet and all these specialised apps, that, again, just happens over five years. So you see the introduction of a new technology and then very quickly, you see it being refined into a super useful, human friendly product. Wow. I mean, on the one hand, that sounds kind of incredible and exciting. On the other hand, it just feels like a super strange world to be getting so many new technologies every few years. Can you explain the idea behind why AGI might even make that possible? Yeah. So here's the most basic version of the argument, which you can kind of make it more complicated to address various objections. But I think kind of this version captures the core idea. So today, there are maybe tens of millions of people whose job it is to discover new and better technologies. Sure. Working in science and research and development, they're able to make a certain amount of progress each year. And it's their work that helps us get better computers and phones and discover better types of solar panels and drives, all these improvements that we're seeing. But like we've been talking about, shortly after AGI, I think there's going to be billions of top human researchers equivalents in terms of a scientific workforce from AI, right? And if you imagine that workforce, or half of that workforce, or just 10% of it, working on trying to advance technology and come up with new ideas, then you have now ten or 100 times the effort that's going into that activity. And these AIs are also able to think maybe ten or 100 times as quickly as humans can think, and you're able to take the very best AI researchers and copy them. So if you think that scientific progress is overwhelmingly driven by a few smaller number of really kind of brilliant people with brilliant ideas, then we just need one of them and we can copy them. They might be happy to just work much harder than humans work. Right. It might be possible to focus them much more effectively on the most important types of R&D, whereas humans maybe are more inclined to follow their interests, even when it's not the most useful thing to be researching. And so all of those things together just mean that we'll be generating kind of 100 times as many new good ideas and innovations each year compared with today. And then that would drive the development of technologies to be at least ten times faster than today. Right. How likely do you think this kind of growth is? Is it the default once we get AGI? I think it is a default. You could give objections to the argument I gave, but I think it's mostly possible to answer those objections. So you could say, well, discovering new technologies isn't just about thinking and coming up with new ideas. You also need to do experiments. Okay, sure. And then I think you can answer that objection by saying, that's right, we will need to do experiments. And that's like testing a drug on humans. And maybe it takes five years or something to really check that it's safe and effective. Right? Yeah. Or you've designed a new solar panel and you want to test its performance in a variety of conditions, or you kind of running some experiments to see what happens when you combine these two chemicals together, because you're not able to predict it in advance. But if you have a billion AIs trying to push forward R&D, and they're bottlenecked on these, needing to do these experiments, then they'll be putting in a huge amount of effort to make these experiments happen as efficiently as possible. Sure. Okay, so whereas today we might be using the lab for 50% of the time, we could be using it, and we might be just doing a whole bunch of experiments and then analysing it afterwards and learning a little bit from each experiment, but also not kind of trying to cram as much into each experiment as is humanly possible. If these AIs are limited on experiments, then they're going to be spending months and months just meticulously planning the micro details of every single experiment so that you can get as much information as possible out of each one, and kind of fully coalescing their theoretical understanding and all the current data and implications and saying, okay, here are the key uncertainties that we need to address with these kind of scarce experiments, and they'll give the humans conducting the experiments really detailed and precise instructions and set things up so the experiments are really unlikely to go wrong, and kind of analyse the resultant data from 100 different angles to learn as much as you can from them. Right. And I think that will go a long way to getting over the experimental bottleneck. I mean, even if you just think you use labs 8 hours a day, but you could use them 24 hours a day, and then there are probably hundreds of other efficiencies like that will all just add up to get to many times more efficient stuff. Right. The AIs can direct humans what to do. So you could be paying very high wages to have unskilled human workers work through the night to run these experiments directed by AIs, telling them exactly what to do when. So, yeah, you'll be able to have those labs working around the clock if that's what's wanted. In the longer run. Robotics is already very good, and I don't think it's going to take too long. Once we have a billion AI researchers to design robots that are able to do the physical tasks that humans do. It doesn't seem like we're that far off at the moment. And so if eventually we just need more humans to kind of build more labs or to run these experiments, I think it will be possible to have robots doing that work and have the AIs directing it, and again, kind of meticulously planning what each robot is doing with its time. So we're getting the very most out of each robot. Right. I mean, the example you raised about human experiments is a really good one, because that seems like it's going to be particularly hard to speed up. Right. There are still a few things that I can already think of that could happen there. So if it's kind of any psychology experiments that you're wanting to do, or knowing how humans will react to a new technology or to a new scenario, then just studying all of the human data on the Internet and doing in depth interviews with humans could give AIs a really good understanding of how human psychology works. Right. In the limit, they could scan a human's brain and kind of upload them to be kind of a virtual digital person, if that human was willing to do it. And then it could then do experiments with them in a simulation much more quickly. Right. Really fast. I mean, that's getting pretty weird and feeling very Sci-Fi but that's part of what we're talking about. We're talking about we've got millions or billions of copies of AI systems that are as smart or smarter than humans, basically using all of their brain power to innovate. And, yeah, it's bizarre, but if you apply all of that brain power, you're going to get super fast improvements to technology. Is this bottlenecked at all by ideas getting harder to find? Are there going to be limits that, just like, a human would hit up on these limits, or humans have AI systems will also hit up on those limits? Or do we expect them not to, because we're talking about superhuman intelligence? So ideas are getting hard to find. For me, the best zoomed out example is just that our scientific workforce has been growing at four or 5% every year over the last 80 years, and that's a massive increase in that scientific workforce over 80 years. Many, many doublings. But actually, the pace of technological progress, if anything, has been slowing down somewhat over the last 80 years. And so, on a high level, the explanation is, sure, we're using way more effort than we used to be, but the ideas are harder to find, so we're actually slowing down a little bit in terms of the pace of our progress. Maybe the best illustration of that dynamic is physics, where kind of a hundred years ago or so, you could have Albert Einstein, in his spare time as a patent clerk, come up with multiple very significant breakthroughs, and then kind of almost single handedly or with a few collaborators, develop general relativity over the few years that followed, which is just a major breakthrough in our understanding of the universe. Whereas today you have probably maybe millions of physicists with these huge machines at CERN that are, to be honest, making, I would say, pretty incremental progress in advancing the state of knowledge and physics. Yeah, okay. Right. Yeah. In terms of how it applies to AI, first thing to say is that even if that dynamic exists and it's very strong, we would still expect a very significant, if temporary, increase in the rate of technological progress. So let's say ideas are getting harder to find, but then suddenly, in ten years, we've got a billion AIs working on it, rather than the kind of 10 million humans. Well, even if our ideas are getting harder to find, then at least temporarily, there'll be much faster technological progress, and then we'll kind of pluck even more of the low hanging fruit. And eventually even these AIs get stuck. Right? Maybe eventually, we still stagnate, but it'd be pretty crazy if you added millions of people or millions of brains to the workforce and didn't get a bunch more technological progress. Yeah. I mean, specifically, if you made the workforce, like, 100 times as big, then. Yeah. Especially with the other advantages I discussed about running ten times as fast and being really focused on the most important tasks. I think really surprising if you don't get at least a temporary increase. In fact, I don't think it would be temporary. And that's because one of the things that AIs can work on is actually building more computer chips to run AIs on improving the kind of hardware designs for those computer chips, improving the algorithms that AIs run, on improving the designs. Of robots, just making themselves better scientists. Exactly. And so we've been discussing how already the pace of progress in terms of the algorithms and the hardware is pretty fast. And I've already said that I expect it to be much faster once we have AGI. And so really, this isn't a kind of a constant sized AI and robotics workforce we're talking about here. But if we choose to do so, then we could have the size of that workforce doubling every year very easily. And so that means you can overcome this problem of ideas getting harder to find, because you're not kind of dealing with a constant or slowly growing workforce. You're dealing with a workforce which is itself rapidly increasing in size. And so even if ideas are getting harder to find, you've got a bigger and bigger workforce to find it. And you can actually model out this dynamic. You can take the best models we have where ideas are getting harder to find, and you can say, well, if ideas are getting hard to find on the one hand, but on the other hand, AIs are able to design better AIs and do all the improvements that I talked about, designing better robots, et cetera. How does that dynamic play out? And it turns out that at least under these models, even if ideas are getting harder to find, of a very steep rate, then you still are going to get the AIs and robots winning that race. That's really wild. I guess those AI scientists might hit some limits. And do you have any ideas for what those might end up being? It's a great question. I think we are going to hit limits at some point. Eventually, we won't be able to design better technologies. Eventually we'll have the best algorithms we can get for making AIs. I do think there are reasons to think the limits could be quite high. So one interesting data point is that small animals are able to double their population size in just a few months, and even smaller animals like insects can double their population size in just days or weeks. So that shows that it is physically possible to have a certain kind of biological robot that kind of doubles its own number in the scale of weeks or months. Right. And with all of this kind of massive scientific effort that we've been describing, it seems possible that we'll be able to design kind of robots of our own that are able to kind of double their own number, kind of build replacement robots in a similar time frame. Right. And so currently you can try and estimate. Well, if we used a factory to try and build another copy of itself, how long would that take? I haven't seen a good analysis of this, but when I've spoken to people, they've guessed it on the order of months. Okay, so that also kind of supports this vague idea that it might be possible to get these robots that are able to kind of build extra copies of themselves and double their own number in just a number of months. So what that all suggests is that we could have like a kind of a robot workforce, which is growing at a really high rate. And by robot, I'm picturing physical bodies. But do you basically just mean AI system? That's like working on science, or do you think physical bodies end up being important because we've got to start automating some of these physical tasks in addition to the cognitive tasks? I'm thinking, yeah, including the physical tasks. When I'm talking about the robots, I mean, if these robots each weighed 50 kilogrammes and we able to produce robots as many robots in a year, such that they weighed as much as all the cars that we currently produce. In a year, then we'd be producing around a billion robots. Oh my God. Each year. Each year. So already the manufacturing capacity seems like it is theoretically there to produce a huge number of robots. And that's before taking into account that it's lucrative. And so we want to create more and. Yeah, unreal. So I do think that this dynamic leaves us in a pretty crazy world where the size of the AI and robotics workforce is growing, potentially growing very quickly. And probably as a result, it's hard for me to imagine technology really stalling out before we hit real kind of limits, kind of fundamental limits to how good technology can get. And there will be such limits. Technology can't improve forever to infinity. Ultimately, you've come up with the very best ways of arranging the molecules to get the desired technological behaviour, right? Okay, do we ever hit limits on just like physical stuff, the stuff we make the robots out of? I think we will. So I already said about the kind of that car statistic suggesting we can get pretty far just in terms of the massive robots we could produce, just with the kind of manufacturing capabilities we already have. The earth is massive. There are kind of mountains with all these kinds of materials in them all over the place. And if we run out of a particular material which is currently useful for building robots, then these billions of AIs we have will be working hard to find ways of doing without that scarce material. And that's been a common pattern in technological development, that you kind of find ways to switch out of things that are scarce. Yeah, right. It's hard to rule out that there's just some material that we just absolutely need and we can't do without, and that bottlenecks things. Maybe once we get to 100 billion robots or something. But it also just seems more likely to me that given just the abundance of just kind of materials that are in the earth, it's a big place, there's lots of different stuff there. It's not like we've mined everything there is to mine or even close to it. We'll be doing using all the best methods for recycling and using things as efficiently as possible. It doesn't seem to me like those kinds of bottlenecks are going to kick in particularly early. And it's not out of the question that we'd have the technology to explore resources in space. I'm adding more Sci-Fi here, but if we're doubling technological progress every, I mean, it sounds like you're talking about months at some point. Yeah, I think months is plausible. Okay. Months is plausible. And so that might mean we aren't limited by earthly limitations. That's right. There is an interesting dynamic there where if we are kind of doubling the number of robots really quickly and we're improving technology really quickly, then we're not that interested in doing an activity which takes maybe like ten years to bear fruit because we're used to kind of our investments paying off with kind of doublings every year. We're like, oh, we could go to the moon and get materials, but, man, that'll take so long. If we just invest in everything we can find on Earth, we can much more quickly kind of increase the. Just use it more efficiently. Yeah. So that time delay becomes more significant when you're already able to grow so fast. So I think, I imagine the kind of going to the moon and the space stuff happens when we're really kind of struggling to find ways to make use of the earth's resources. Okay, so it sounds like we're kind of talking about something like AI systems replace humans in a bunch of sectors, really, during our lifetimes, and then our lives really change quite radically and very quickly. Yeah, I guess I just find that super weird. I think my brain is like, no, I don't believe you. That's too weird. I just can't imagine that happening. If we're saying this is happening in the early 2030s, I'll be in my late 30s, and all of a sudden, the world would be radically changing every year, and I won't be working. I agree. It seems really crazy, and I think it's very natural and understandable to just not believe it when you hear the arguments. And that would have been my initial reaction in terms of why I do now believe it. There's probably a few things which have changed. Probably I've just sat with these arguments for a few years and just been like, I just do believe it. I have discussions with people on either side of the debate, and I just find that people on one side just have thought it through much more. And I think what's at the heart of it for me is that the human brain is a physical system. There's nothing magical about it. It isn't surprising that at some point we develop machines that can do what the human brain can do at some point in the process of technological discovery. And to be honest, that happening in the next couple of decades is kind of when you might expect it to happen. Naively, we've had computers for 70-odd years. It's been a decade since we started pouring loads and loads of compute into training AI systems, and we've realised that approach works really well. Just if you were kind of to say, okay, when do you think humans might develop machines that can do what the human brain can do? You kind of think it might be in the next few decades, right? And I think if you just sit with that fact that there are going to be machines that can do what the human brain can do, and you're going to be able to make those machines much more efficient today, and you're going to be able to make even better versions of those machines, ten times better versions. You're going to be able to run them day and night, and you're going to be able to build more. When you sit with all that, I do think it gets pretty hard to imagine a future that isn't very crazy. Yeah. And another perspective is just zooming out even further and just looking at the whole arc of human history. So if you'd have asked hunter-gatherers who only knew the 50 people in their group and who had been hunting using techniques and tools that as far as they knew, had been passed down for eternity, generation to generation, doing their rituals, if you'd have told them that in a few thousand years there were going to be huge empires, building the Egyptian pyramids and massive armies, and the ability to go to a market and give people pieces of metal in exchange for all kinds of goods, it would have seemed totally crazy, right? And then if you'd have told those people in those markets that, no, there's going to be a future world where every ten years major technological progress is going to be coming along and we're going to be discovering drugs that can solve all kinds of diseases, you're going to be able to get inside a box and land the other side of the earth. Right. Again, they'd have just thought you were crazy. And I think while it seems that we understand what's happening and that progress is pretty steady, that has only been true for the last 200 years, and zooming out, it's actually the norm throughout the longer run of history for things to go in a totally surprising and unpredictable direction, or kind of a direction that would have seemed totally bizarre and unpredictable to people naively at that. Right, right. I feel like I was introduced to it when I read What We Owe the Future, Will MacAskill's book, that there's this thing called the end of history fallacy. It really feels like we're living at the end, we're done changing. We're going to maybe find some new medical devices or something, but basically we've done all of the weird shifting that we're going to do. And that just like, I can't really justify that, it does seem like a fallacy. Like presumably things are going to look super different in 50 years, and sometimes those changes have gone super fast in history and sometimes they've gone super slowly. And we've got real reasons to think that we might be entering a period of really fast transition. Yeah, I mean, if anything, I'd say the norm is for the new period to involve much faster changes than the old period. So hunter gathering went on for tens of thousands of years, if not hundreds of thousands of years. Then we started doing agriculture and formed into big societies and did things like the pyramids, and then a way that people often think of the next phase transition as being kind of the start of the industrial revolution and the beginning of kind of concerted efforts towards making scientific progress. And after we did agriculture, kind of new technologies and changes were happening on the scale of maybe 1000 years or maybe a few hundred years, which was much faster than in the hunter gatherer times. And then today, after the industrial revolution, we're seeing really big changes to society every 50 years. Right. So we've already seen kind of historically those phase transitions have led to things being faster. So that, I think, is the default expectation for what a new transition would lead to. Right. And it just feels weird to us because we're pre transition, plausibly, whoever's living 50 years from now will just be like, yeah, obviously that was coming. And those weird people living in 2023 thinking that they'd made all the technological progress they were ever going to make. Yeah, maybe. I'm still struggling just to imagine what it would look like, I guess. Yeah, because it's possibly going to be us, what's in store for us. Is it going to be good? I think it could be really good. It could be really bad. It could be really good if we align AI. So they're always trying to help us and help humanity do as best as it can by humanity's own lights and the kind of benefits from AI. And these new technologies are used to solve the world's most pressing problems and used to lift people out of poverty and give people the lives that they hope for themselves and for their children, solve the problems of climate change, of poverty, of disease. I think it could go really well. Right. And so in the best case, where AI is really trying to help us. It's still kind of unimaginable to me as a world. I mean, maybe it's just like I'm so biassed by the status quo where I need to work and I need to work to live, I need to work to help solve problems. Like, in this best case, is there unemployment? Is there unemployment for everyone? Is it a slow transition, a fast one? Does it make inequality better because no one needs to work and we all have enough things, or does it make it worse because some people have to work? Do we have predictions about that that are worth making? I think the default is that inequality would become greater because all of the wealth and useful work is coming from these AI systems, which I think by default will be controlled by a small number of people and companies. Okay. In the very best case, then you hope that wealth is equally distributed or kind of much more equally distributed than it would be by default. And it is true, I think, that there's going to be so much progress made if AI is aligned that it will be very cheap to give everyone in the world the standards of living that are enjoyed by the very richest people today in terms of material comforts and health, and actually much better on those fronts, I think, after all this technological progress. So I think if we can get the AIs to be really trying to help us, then even if we mess up on things, like the kind of distribution of benefits, even if we mess up a bit on that, then I think things will still look pretty good because there's just so much to go around. Right. If just there's kind of universal basic income, you could just use 1% of the output that's produced in a year to kind of give everyone kind of all the material and technological things they need to kind of meet all of their needs, all of their material needs. Right. In terms of work, I think it will no longer be the case that you can produce a higher quality service or product than what an AI could do or a robot could do. One thought is that there will be some humans, maybe me and you, who just value human contact and hang out with other actual, real humans, and that could provide a kind of work for. Those who want it role for humans. Okay. And another possibility is that we rethink the nature of work. So we do work to help each other. And even though we know that AIs could do the work just as well, we're still happy to do that because it gives us a sense of meaning. Or we kind of do creative things instead, like creative writing and drawing. And even though we know that AIs could do that better than ours, it's still enough for us to have a sense of purpose. I mean, people still play chess today and still really enjoy it and get purpose from it, even though they know that they’re never going to match the best AIs, right? Yeah, I guess I can imagine lots of people listening, hearing about this future and being like, no, I like the world the way it is. I like that humans get to make choices for ourselves as a society. I don't want AIs making it for us. I like that I have to work, get to work. I don't know, I can imagine people being like, no, I don't want AIs to be making the art. I want humans to be making the art. So is there some chance that there's, I don't know, like a movement that's anti AI growth that stops this from happening even though it's theoretically possible? That's a great question. I do think it would be good for us to take this transition more slowly than it's theoretically possible. And that might happen by default if we don't make specific efforts to go slowly. And so I think if people do try and delay or stop this, it could be a good thing, because I don't think we're prepared for that new world. Right. I think it's going to be very hard to permanently prevent this transition from happening. How come? One way to think about it is that there is some kind of upfront starting cost to get this transition going. So let's really simplify it and say today, if you spent a trillion dollars, you'd be able to train AGI and you'd have enough money left over to buy some manufacturing equipment for making robots, right? And then you could have your AIs do research into better robots and making better ais. And that whole process could lead to you having even more AIs and even more robots. And you could then grow your population of AIs and robots. And just with that trillion dollar initial investment, you could end up with this massive AI and robot population, which is then able to just start doing the scientific work needed to significantly accelerate technological progress. Right. So the thing that's difficult is that upfront cost will be falling over time. AI algorithms are improving, computer chips are improving. And so the cost to kind of training AGI and then just using it to build robots and to build more AIs and make money in the economy by selling its services and just kind of building up its own self perpetuating energy, that ultimately results in making technological progress. So you can sell more useful things to society that people want. That cost is going to be falling. And so let's say it was a trillion today, in the future it's going to be 100 billion, and then it's going to be 10 billion, and there's going to be a lot of incentive to do this, because it's going to grant whoever does it a lot of power. They'll have all these AI workers that they can use to do whatever they want them to do if they manage to solve the alignment problem, if they use it for designing new technologies. And those new technologies could grant additional military power, or they could grant things that people all around the world desperately want, like curing illnesses, like preventing climate change, like understanding and solving mental health problems, like life extension. So it's not just kind of economic incentives, it's not just like to get rich, it's like all sorts of motivations are benefited from paying this cost to get this hugely productive AI scientist workforce. Yeah, kind of whatever you want, right? You can probably get it much more effectively if you have a billion AIs and robots designing technology to help you get it. And I think we can delay it, we can say, okay, we're going to be really cautious. Only a few people are allowed to train these systems, and we try and convince the other countries to go slowly as well. But to think that we could be that even 100 years after it's first been possible to train AI for a trillion dollars, that still no one has done it, and no one is using it to make scientific progress, even though the cost is now like $10 million. It's really hard to imagine that we kind of prevent anyone from all the different actors from doing it as it gets cheaper. And I think sometimes people, when they're thinking about it, imagine that in order to get this kind of ten or 100x faster technological progress, we'd have to be making a real effort and really kind of being super efficient and driven about it. But I think that's not the right way to think about it. It's more like, by default, all you need to do is ask your AIs and robots, please do these tasks for me. And if you need to make tech progress along the way, do it. They will suggest the plans that involve making tech progress. They will get in contact with the labs and organise for the experiments to happen. You won't have to do anything, right? And so I don't think it's going to require some kind of concerted pro growth enthusiasts to really push for this. It's more like you want stuff, the AIs are going to try and do the stuff you want, and whenever they make tech progress, it's going to go really well and it's going to really help you solve your problems and you're going to just want to do more of it. Yeah, just enough time will pass, enough actors will think on it and decide to do it at some point. Yeah. So I guess I buy that. The incentives are there for eventually an actor to want to build this kind of AI scientist workforce. It still seems like there have been enormously lucrative and beneficial technologies that we haven't pursued. So one example that comes to mind is like nuclear power, which could help loads with climate change and would also be again, super lucrative. And yet we basically haven't done anything like what we could do with it. Could there be something similar? I mean, it's just kind of stigmatised is like one reason we haven't, and I guess it's really expensive. In particular the upfront costs, which maybe just ends up true of this AI world we're talking about. Yeah, it's a great example. I don't have a good understanding of what happened, but I think there were some big catastrophes with nuclear power, and then it became very stigmatised and the regulatory requirements around it and the safety requirements became very large, much larger, really, than was reasonable given that fossil fuel energy has damaging health consequences as well through air pollution. Right. And as a result, it just became kind of a mixture of stigma and just the additional cost from all that regulation just prevented it from being rolled out. But I do think there are a fair few very significant disanalogies between that case and the case of AI. Okay. Yeah. What are they? So one thing is that there were other sources of energy that were available, and so it wasn't too costly to be like, well, we're not going to use nuclear, we're going to use fossil fuels instead. And then even the green climate change concern, people could think about kind of developing solar panels and renewable energies. And in the AI case, that there is going to be no alternative. There's going to be no alternative technology which can solve all illness and which can grant your nation massive national security and military power and that can solve climate change. This is going to be the only option. So that's one disanalogy. Okay, that makes sense. Another kind of disanalogy is the cost factor. So with nuclear power, it's become more expensive over time due to regulations. And that's been a big factor in not being pursued. But the specifics around these cost curves with compute and this algorithmic progress patterns suggest that the upfront cost of training AGI is going to be falling really pretty quickly over time. Right. And so even if initially you put loads of regulations which make it very expensive, it's really not going to be long until it's ten x cheaper. Right. And so permanently preventing it when it's becoming cheaper and cheaper at such a high rate is going to be really difficult. Third is just talking about the size of the gains from this technology compared to nuclear power. So France adopted nuclear power and it was somewhat beneficial. It kind of now gets a lot of its power from nuclear energy, and that has no climate change impacts, and that's great. But it's not as if France is visibly and indisputably just doing amazing well as a country because it's got this nuclear power. It's kind of a modest addition. Maybe it makes it look a little bit better. Right. But by contrast, if one country is progressing technology at the normal rate, and then another country comes along and starts using these AIs and robots a little bit, you're going to see very significant differences in how its overall technology and prosperity and kind of military power is progressing. And then you're going to see that as countries dial up how much they're allowing AIs to do this work, that there are then bigger and bigger differences there. And ultimately, the difference between advancing technology at our pace versus advancing technology 30 times faster is over the course of just a few years, it becomes a massive difference in the sophistication of your country's technology and ability to solve all kinds of social and political problems. A last point on this difference is that the US did in fact invest a lot of money in nukes shortly after the development of fission power. When it came to a matter of national power, they were very happy to invest in the technology, despite the risks, which were clearly very high. All of the same risks? Yes, the incentives were out of whack and we didn't get nuclear fission power, but when it came to this kind of military technology for which there was no replacement, countries were very keen to do it, and they made it happen. And AI driving significant technological improvements across the board is going to be a huge source of military power. Right. And so it's really hard for me to imagine that just no one ever uses it for that. And you've totally preempted my next question, which was like, can we definitely not come up with an international treaty that is like, the downside risks of this technology at this scale are possibly huge because air alignment is so hard. And so we're all agreeing not to go forward with it. And I guess we had lots of reason to do that in the case of nuclear weapons, and we didn't. And we have lots of reason to do that in the case of biological weapons, and we suck at it. We do not live in a world free of biological weapons or nuclear weapons. So I do think we should try, and I do think we can slow things down and we can increase the requirements and the safety efforts that's required, maybe to make it ten times as costly or 100 times as costly to develop this technology. I think that is one thing, and that's a big ask, and I think we should try and do as much of it as we can. Even if we can do that, it's a whole different thing to talk about permanently choosing to never develop the technology, even after we've made maximal efforts into making it safe, even after all of the safety tests are saying it looks like it is safe, right? Even when millions of people are dying every year from illnesses which we know could be prevented if we allowed the AIs to do research into treating it. I think just permanently not going forward with using AIs and robots to make that technological progress. Like I said, when it's becoming cheaper and other countries and other companies might want to do it, that does seem like it's just very unrealistic, just implausible, and maybe not desirable either. To be honest, after a certain point, we should take it really cautious, right? After 200 years of research into AI alignment, even if we're like, ooh, this seems weird and scary and might change the world as we know it. At some point there are going to be incentives for some actors, countries or companies to try to deploy this technology at scale to solve problems like poverty and illness, maybe also to get military advantages to solve problems like climate change. And those incentives might just be so strong that even if we take our time, we'll probably eventually do it. Someone will. And once that happens, progress will become so quick that we're looking at really economic growth at a pace that's, I guess, still kind of unfathomable to me. But that is this kind of thing where progress we've seen over the last hundred years happens in the next ten and actually just keeps getting faster and faster. Is that kind of the picture? Yeah, that's the picture. That's really weird. It's scary. I guess it's also quite hopeful. I mean, if I let myself hope for that good world where we use it to solve problems. I feel nervously, really excited. But I guess we've got some real challenges to overcome first. Yeah. Okay, let's move on to a related topic you've been researching more recently. So, you've just written the draft of a report on AI takeoff speeds. That has some pretty alarming results to me, given everything we've just talked about. And I guess, just to get on the same page about language, what exactly do you mean when you talk about AI takeoff speeds? Roughly speaking, capabilities takeoff speed is the question of how quickly AI systems will improve as they approach and surpass human level intelligence. And so the capabilities are like their ability to do things like drive cars or program new programs? Exactly. So a fast takeoff speed could be that in three months, AIs go from mouse level intelligence to significantly more intelligent than the smartest human. Right. Where a slow takeoff speed, that could be happening over decades. Got it. Okay, so we're just, like, slowly making progress. Next year, it's self driving cars, and then it takes another ten years to get to programs that can write other programs. So I guess you've written this report, and I won't give away the results yet, but it gives some evidence about how fast we might expect AI takeoff speeds to be. But before we get to that, before you wrote the report, what did you think was the most compelling evidence that AI takeoff speeds would be particularly fast? So, I think the most convincing argument is related to how humanity's capabilities were improving much more slowly 50,000 years ago than they are improving today. And the attempt to draw an analogy to what might happen with AI capabilities. So, if we think that a million years ago, humanity's kind of cognitive abilities collectively were maybe doubling every, let's say, 100,000 years, I don't know. Okay. That's just kind of something to represent their kind of slow increase in brain size and capacities. The exact number doesn't matter. Whether you want to say it's 100,000 or 10,000 years is a very slow doubling size in their abilities. Right. And that's basically because slowly, they're evolving to have slightly bigger brains. We're, like, adding a bit of prefrontal cortex, and the population is getting a little bit bigger over time, but grew very slowly. So collectively, it takes thousands of years to double. Yeah, exactly. Cool. Okay. Whereas today, our abilities are improving incredibly quickly as a society, our population growth is much faster, our command of technology is much faster. And like I said, in the last 200 years alone, we've doubled the economy many times and doubled our ability to understand and manipulate the world very many times. And if you think that there will be some analogue of that transition as we approach agI, then AI is already improving really quickly. I would say it's kind of doubling its abilities in less than a year at the moment. And so if that's kind of the slow initial pace, then the new pace would be kind of blisteringly quick, right? Unimaginably, yeah, exactly. The way I would think about it is that a million years ago, humans weren't able to do science and to discover technological improvements much at all. And so they didn't have access to this additional feedback loop of improvement, where you discover new technology, pass it on to the next generation, then they start on a better place, can discover even more technology, you can now support a bigger population. There's this whole feedback loop, which arguably, we couldn't access a million years ago. Then we improved our cognitive abilities as humans a little bit, and then suddenly we got over this threshold where, okay, we can access this kind of doing technological progress feedback loop, which then speeds up and speeds up as we develop agriculture, and then we develop text written language, and we develop maths, and we're now even better at discovering new technologies. And the thought is in my mind that maybe there's something similar that happens with AIs, whereas today they're not that smart, and so they're not able to access a certain feedback loop. I'm not sure exactly what that feedback loop will be, but at some point, they become smart enough that there's this additional feedback loop that they can use to improve their capabilities, kind of like how humans use technology to improve our capabilities. One funny thing about this argument is that it's not clear what that new feedback loop might be for AIs. And so that leaves me a little bit puzzled over where to go with this argument. So, for humans, we discover the scientific method and we experiment on things, and we built computers, and now we can run programs that help us do science. But for AI, it's like we don't even know what kinds of science they'll discover that's, like, beyond ours. And is there some question about whether there even are kind of higher orders of science that we haven't developed, but that AI systems might to kind of increase their own feedback loop? Yeah, I think that's right. And you can throw out ideas for what that might look like. Right. Maybe it's that AIs learn to work together in a team in a way that is way more efficient than humans have ever done. Or maybe they run simulations or something to learn about economics or something in a way that we can barely understand because we only see the economies run in these weird real world scenarios. For example. For example. Cool. Okay. Yeah. Are there any limitations to that argument, or do you just kind of buy it? Yeah, I actually don't put that much weight on this particular argument. Oh, interesting. So the main reason is that evolution was not trying to make humanity as a whole as capable as possible, and it wasn't trying to make humanity as a whole good at science. Right. So from that perspective, it's not actually as surprising that humanity went from really sucking at science to being really good at science in a fairly short time frame. Yeah. So here's an analogy. Before 2020, we hadn't made many Covid vaccines, not because we couldn't, but because we weren't trying to. We were focused as a society on doing other things. Okay. Then around 2020, it became really useful for us to make lots of vaccines. And then, lo and behold, the number of vaccines went up very dramatically. Now, that doesn't mean that our kind of abilities in vaccine making suddenly went up. It just means that we reprioritised, reallocated the resources we already had towards making vaccines, okay? And I want to say that's somewhat similar to the way in which kind of our ancestors, a million years ago, weren't that good at science. But evolution wasn't trying to make us that good at science. It was mostly trying to make us hunt, successfully feed our families. Okay. Science was like maybe a tiny bit useful back then because it maybe allowed you to discover something with you on your own lifetime, but it really wasn't very useful then. I think more recently, maybe more like kind of tens or 100,000 years ago, it did become more useful for humans to do science and to be flexible learners. And so it's not that surprising that at that point, say, 50,000 years ago, where it was more useful for humans to be good at science, evolution then reallocated those cognitive resources of humans to being good at science. Right? So that kind of reallocation by evolution from kind of just foraging and then reallocating those cognitive resources to doing science is kind of like human society reallocating its resources to make Covid vaccines. Okay. Yeah, that's really helpful. So something like a combination, maybe of language, maybe of just, like, group living, maybe some things that I don't understand made it much more beneficial to be able to learn new things and learn, like, a range of things, not just the same things, over and over again. And so a couple of tweaks in the brain was enough to make the brain that we'd be using for very specific set of tasks become useful for just, like, a much wider range of tasks. And that wasn't really fundamentally altering the amount of brain we have, but how we use it. And that capacity already existed, and, like you said, was repurposed. So, yeah, here's how you would relate it to AI takeoff. You'd say that in the case of evolution wasn't initially trying to make humans good at science. And so no massive surprise that it's able to quickly make some tweaks that make humans good at science late in the day. But with AIs and with AI development, humans will be at every stage trying to make AIs as useful as possible for doing economic tasks, helping with science and research. And so we wouldn't expect there to be this kind of overhang where the AI has these abilities, which it's just not using, because we would expect humans to be trying to coax those abilities out at every step of the way. And so if you're constantly trying to coax abilities out, most likely you'll only find ways to do it incrementally, as opposed to, if it happened to be the case that, I don't know, we found a billion computer chips on another planet and could just use them to train up a bunch of AI systems, then we'd expect the step change. But currently everything is just increasing incrementally. We're increasing chips incrementally. We're increasing algorithmic progress incrementally. And so it's just going to keep improving at a kind of incremental pace. And I think, crucially, we have to think that we're currently, at each step of the way, trying to use the most recent algorithms and most recent compute to actually get AIs to do, let's say, useful science research. If we're incrementally increasing the computing algorithms, but we're not actually trying to get the AIs to do useful science research, then it could be that one day we decide to try and get AIs to do useful science research, and then suddenly we train them to reassign all their cognitive abilities to that task, and we do get something that's really quick. So it's a really important assumption here that in some sense, the AI development ecosystem is kind of efficient, and it's. Using new AI capabilities to do the cutting edge research, as opposed to if there were only market incentives to make AI that, I don't know, made these beautiful images like DALL-E, and no incentives at all to use AI to improve AI systems, then if one day we made a few tweaks to DALL-E and were like, stop making pictures, make programmes instead. But actually it did have capabilities related enough that we could make that tweak semi easily, then all of a sudden we'd have this system that could write programmes really efficiently that we'd never had before. Right. Or maybe a more probable scenario would be we're just using all our AI resources to train these image generating systems like DALL-E, and then we're like, you know what? Why don't we just try using all those resources to train a science AI? And then we pick the architecture to specialise it for science. We use the data to specialise it for science. We use all the compute that were previously pouring into these image generation systems, and then suddenly we're like, wow, our science AIs are amazing. And it came out of nowhere because we hadn't been trying in the previous years to do this at all. Yeah. So are we currently trying to make AI systems that are really good at science? I think it's a good question. The market doesn't seem to me to be super efficient. I've been playing around with GPT-4 a bit recently, and to me it looks like GPT-4 is pretty smart. It doesn't seem to me like its cognitive abilities have been really directed in the direction of helping to advance science, to be honest. So I do think that this argument could ultimately say, yeah, a faster takeoff is plausible, and the mechanism could be reallocating the AI's cognitive abilities towards science. I mean, GPT-4 is just trained to predict the next word on the Internet. Right. That's a very different kind of task than the task of advancing science. And so I think that is a reason to expect a faster takeoff, more of a jump. Yeah. Okay, interesting. I haven't heard that argument before. Cool. Well, I want to now get to the report that you've written on AI takeoff speeds, which asks how quickly AI might go from kind of pretty economically valuable to just extremely capable, maybe as good as humans. And. Yeah, I guess you define your terms pretty clearly in your report, so maybe we should start by doing that. Am I right in remembering that you are trying to answer the question of how quickly we'll go from AI systems that can do 20% of human tasks to AI systems that can do 100% of human tasks, is that right? Yeah, that's right. In particular, I am restricting to cognitive tasks, that's similar to what we discussed earlier. It's any task that you could do remotely that doesn't require you to be physically manipulating objects yourself, because AIs don't have physical bodies that wouldn't be included for them. It does include tasks like giving instructions to a human who's doing a physical job, telling them where to move things, what to do with their arms, or potentially giving instructions to robots that are doing physical tasks, but it doesn't include the physical motions themselves. So does it include things like driving cars? Yes, it does. Okay. It does. And that's because driving cars is really a set of algorithms and you can turn the wheel of a car. So it's a good point that currently the way humans drive cars is by physically moving various levers in the car. But I think actually giving the AI the control of the steering wheel and of the pedals and the brakes is actually pretty trivial. So the only thing that's hard in practice about getting AIs to do driving is the cognitive parts of what should the car be doing at each point. Yeah. Right. Okay, so then an example of a task that isn't included is something like helping people move house. It's like carrying the boxes in and out. Carrying the boxes in and out would not be included. Yeah. Okay. But telling them, here's the best plan for moving, here's the order you should move the boxes in. That all would be included. Okay, so that's basically what you've done. You want to know how fast do we go from 20% of cognitive tasks to 100% of cognitive tasks? Yeah. Can you actually clarify what it means for AI to be able to complete 20% of tasks? So you could say, okay, let's say AI automates driving. What percentage is that? Is that 3% because it's 3% of people do it, or is it, do we just look at a long list of tasks and assume that each takes up an equal percentage? Like, what do we mean by 20%? Yes. And so the way I'm currently thinking about that is you look at how much people pay for those tasks to be performed in the economy. Okay, so let's take the driving example. I don't know. Let's say that drivers around the world are being paid $2 trillion a year for the work they're doing, driving trucks and taxis and everything else. In that case, because $2 trillion is 2% of the global GDP, I would say that automating driving, fully automating, all driving, would be automating 2% of all economic tasks. Got it. And then you're saying, how fast will we go from, we can do 20%. So I don't know. Maybe it's like replacing all drivers, maybe it's replacing all journalism, because GPT-4 seems to be really good at writing and, I don't know, a couple of other things. How fast you go from that chunk to literally all cognitive tasks, including, I guess, science, AI R&D. Now. One complication is that from year to year, the amount that is paid to people to perform each task might change. So in 2020, maybe drivers are paid $3 trillion a year for their work. Maybe in 2025, they're paid $4 trillion a year, and that could change. So I'm pegging these percentages to the year 2020. Got it. As a kind of arbitrary choice just to make the definition unambiguous. Yes. Okay, that makes sense. So what were they paid in 2020? And even if, I don't know, like wages change for all sorts of reasons, including AI taking over some jobs, we're still just thinking of what percentage of the 2020 cognitive economy is being automated. Exactly. And it is really important to keep that in mind because typically when AI automates a certain task, it becomes really cheap to do that task. Right. So it becomes a much smaller fraction of the economy. Exactly, yeah. And so you can end up thinking that AI is never doing anything when it's actually done almost everything. And so that is just an important thing to be aware of. Yeah. That makes total sense. Is there an analogy from the industrial revolution or something? The best analogy might be agriculture. So I think in 1500, basically everyone worked in agriculture. Right. It was 90% of the economy or something. Exactly. More of GDP would have basically been agriculture. Today, I think it's less than 5%. And that's because we've become really good at producing food with very little need for human labour. So it's not to say that fertiliser and trucks and really highly productive seeds aren't contributing a bunch to the economy, but clearly they have. But were you to measure it as a fraction of the GDP that they are responsible for, it would be smaller because everything's just gotten so much cheaper because of them. Exactly. Cool. Okay, that makes a bunch of sense. Nice. Okay, so we've got definitions. So you've asked how fast will we go from AI systems that can do roughly 20% of the cognitive tasks that humans were doing as of 2020? And how quickly will we go from 20% of those tasks to 100% of those tasks being, at least in theory, able to be automated by AI systems? So, yeah, what was your result? So the conclusion from the report is, I guess, pretty scary. The bottom line is that my median guess is that it would take just a small number of years to go from that 20% to the 100%, I think, equally likely to happen in less than three years as it is to happen in more than three years. So a pretty abrupt and quick change is the kind of median, kind of best guess, median. Wow. And do you believe that in your bones? Does that feel, like, very plausible to you? Yeah, I do. So some quick things about why it's plausible. Each year, once you take algorithms, better algorithms, and using more compute into account. We're currently training AIs each year that have kind of three times bigger brains than the year before. It's a really rough way to think about it, but imagine three times smaller brain than humans. That's chimpanzee brain size. Right. Each year you're going from chimpanzees to humans. I think it's really hard to try and account for the effect of the algorithmic improvements, but on my kind of best guess of what those amount to. Yeah. Each year, we're making the brains of AI systems about three times bigger. Wow. And right now, it's humans that are doing all the work to improve those AI systems. As we get close to AIs that match humans, we'll be increasingly using AI systems to improve AI algorithms, design better AI chips. And so overall, I expect that pace to accelerate absent a specific effort to slow down. Right. So rather than three times bigger brains each year, it's going to be going faster and faster. Five times bigger brain each year, ten times bigger brain each year. And I think that just already makes it plausible that there could be just a small number of years where this transition happens, where AIs go from much worse than humans to much better. But to add in another factor, I think that it's likely that AIs are going to be automating AI research itself before they're automating things in most of the economy. Right. Because that's the kind of the task and the workflow that AI researchers themselves really understand. So they would be kind of best placed to use AIs effectively. There, there aren't going to be kind of delays to rolling it out or trouble finding the customers for that. In the same way, the task of AI research is quite similar to what language models are currently trained to do. They're currently trained to predict the next token on the Internet, which means they're particularly well suited to text based tasks. Right. And the task of writing code is one such task, and there is lots of data on examples of code writing. Oh, I see. So it's like, typically, I don't know that much about coding. Is it basically also token prediction? That is how current coding assistants work, I think, is that they're looking at you, start writing your code, and they predict what's going to follow. Like, one way of putting it would be by the time that the AIs can do 20% of cognitive tasks in the broader economy, maybe they can already do 40 or 50% of tasks, specifically in AI R&D. Right. And so they could have already really started accelerating the pace of progress by the time we get to that 20% economic impact threshold. I mean, at that point you could easily imagine, really, it's just one year. You give them a ten x bigger brain. That's like going from chimps to humans and then doing that jump again. That could easily be enough to go from 20% to 100%, just intuitively. And I think that's kind of the default, really? That's terrifying. Yeah. And I think there's even more pointing in that direction. I think that already we're seeing that with GPT-4 and other systems like that, people are becoming much more interested in AI, much more willing to invest in AI. The demand for good AI researchers is going up. The wages for good AI researchers are going up. AI research is going to be a really financially valuable thing to automate. Right. If you're paying $500,000 a year to one of your human research engineers, which is lower than what some of these researchers are earning, then if you can manage to get your AI system to double their productivity, that's way better than doubling the productivity of someone who works in a random other industry. Just the straightforward financial incentive, as the kind of power of AI becomes apparent, will be towards let's see if we can automate this really lucrative type of work. So that's just another reason to think that we get the automation much earlier on the AI side than on the general economy side, and that by the time we're seeing big economic impacts, AI is already improving at a blistering pace, potentially. Okay, well, that's, again, really scary. Like really genuinely very scary. I completely agree. Do you have a guess at what percent of cognitive tasks AI can currently perform? It seems like we're really far away from 20%. Yeah, intuitively. I think it seems like we're far from 20%, because AIs aren't doing that much in the economy. If I looked at a list of the kind of cognitive tasks people were performing in 2020 and what they were paid for them, it's not as if AIs are ready to kind of replace a big fraction of that labour. So that's just the 20% is far off. I'm actually less confident that it's far off than I used to be than if we would have had this interview of six months ago. Because just seeing GPT-4's performance, firstly just doing really well on just a whole wide range of university exams and other formal tests without having specifically trained on that, and then me kind of playing around with it and thinking, yeah, just seems smarter than most people I talk about this stuff with. Right. Most of my smart friends wouldn't be this smart. Right. And thinking, maybe, actually, if you just put some work into specifically applying GPT-4, you could automate quite a large fraction of the cognitive tasks. It does seem much more plausible to me that maybe you could get to 10% today or within a year. Wild. That's super interesting. I mean, I guess, yeah. I was also blown away by GPT-4's performance on the SAT, the LSAT. For anyone who hasn't looked, we'll stick up a link to those test results. I think it was performing, I mean, much better than I ever did on my AP tests in high school and better than I did on the GRE. So it's beating me and I think beating out loads of other people already, maybe I'm over or putting too much weight on the fact that currently not that many people I know are using it to do anything. And it sounds like your impression is like, maybe it's pretty close to being able to do a lot. Yeah. To actually, in practice, replace 20% of the tasks that people do, it's actually a pretty tough thing to do, because for myself, all the different parts of my workflow are very much entangled up together. So I can't easily take out a 20% chunk and be like, oh, GPT-4, do this chunk, because it's all kind of mixed up. Sure. And so, historically, the way that automation has worked is that we've got a new technology and then we've spent decades readjusting our workflows so that we can neatly parcel out 20% of our workflow for this new technology to do. And the technology can be fairly dumb because we've kind of really neatly parcelled out that part of our workflow. Yeah. Do you have an example? Yeah. So let's say moving over from paper records to computer records. So maybe people used to have to write down lots of paper records and maintain a filing system for their information. And these days, a lot of that work is done automatically by computers and data storage. But at first, it wasn't that easy to immediately move over to the computers, and it took decades as people got rid of the paper stuff and got used to kind of teaching their other employees to use the computers and got used to using their customers to fill out online forms, rather than filling out the paper forms that they were using before. And all that rearranging of workflows took a long time to happen. And so one scary possibility is that if AGI is just ten years away, then there won't be time to do that rearranging of workflows that is necessary to get, say, GPT-4 to actually automate things in practice. And so the 20% automation ability won't happen through some kind of dumb system that I've kind of parcelled out a nice thing to do. It will actually happen with a really smart system that basically understands my whole workflow well enough to be able to do 20% for me. Right. Which means that it could be pretty close to just being able to do all of it. Right. I mean, I've almost got that impression with GPT-4 and my job. We asked it, how can you help us make The 80,000 Hours Podcast? And it was like, I can help you come up with guests, I can help you write interview questions for the guest. If you tell me what they've worked on, I can help you. I mean, it basically rattled off a list of things, and I was like, as soon as it has a voice, that'll be it. That'll be it for me. And I think I thought it would help me with subtasks first. I think I thought maybe it would help me with generating titles and, I don't know, maybe giving me summaries of people's work so that I could read them a bit faster. But I think, in fact, it's actually going to be really great at the start to finish interview process. Yeah. Soon enough that I'll just skip over all of that, which I don't know if that's true, but it doesn't seem crazy to me. And so maybe that's just another actual example of what you're talking about. Yeah. So in that example, by the time it can automate 20% of the kind of tasks you're doing, it can almost do all of it, right? Yeah, makes sense. Okay, so next, I want to dive into your methodology a bit more deeply for this takeoff speeds prediction. It's such an alarming result that I've this urge to understand what's going on a bit better. Yeah. So that headline result, this prediction that AI takeoff might only take a few years, is basically based on an economic model that you made that tries to answer the question of whether you can get human level AI just by increasing the amount of compute that we have to train AI systems. So in other words, kind of without paradigm shifting algorithmic breakthroughs. Yeah. To start us off, can you actually remind me what compute is? Okay, so compute is a measurement of how many calculations you need to do or a given computer is doing. So let me give an example. So the most common unit for measuring compute is a FLOP. And a FLOP is adding together two numbers, or multiplying together two numbers, or dividing them or subtracting them. Okay, so mathematical operation. Exactly. Currently, when we develop AI systems, the way we do it is by doing loads and loads of these calculations, of adding things together, dividing them, minusing them, and by doing all of these calculations, that is how the AI decides what it's going to do. And it's also how we train the AI in the first place. Got it. You could analogize these calculations to the kind of firing of neurons inside our own brain. Okay, so FLOPs are basically just mathematical operations or things like, are both of these things true? Or something like that? And then compute is. Sorry, it's something like how. How many of those calculations we can do. So one FLOP is just one of those calculations. Some of the biggest language models today are trained with, I think, 10 to the 24 FLOP. So that is a million calculations. That's how many calculations you need to do to train some of today's big language models. So the amount of compute is another way of saying how many calculations did you need to do it? Got it. So is compute. Compute is the amount of computation you need to do. And a FLOP is the kind of unit of how many? Yeah. Okay, cool. So then you're asking this question about AI takeoff speeds, and we're just assuming that we increase compute, which is made up of machines, but also has to do with algorithms as well. So is it true that we need less compute if the algorithm is super efficient? Because the algorithm just requires that you do fewer calculations to get the same result or something. The strict assumption is that if we used the algorithms that were available in 2020, then there is some amount of compute such that if God handed a top AI lab that amount of compute, and they had a few years to adjust the algorithms to using that much more compute, then they would be able to train AGI using that amount of compute. So then, in the model, we make an assumption about how much compute would have been required. We actually put a probability distribution over it. But importantly, algorithmic progress can reduce that computational cost over time. Got it. So, maybe in 2020, you'd have needed ten to the 30 FLOP to train AGI. But maybe by 2025, your algorithms are ten times better, and so you only need ten to the 29 FLOP to train AGI. And so the basic dynamic in this framework is that in each year, our algorithms improve somewhat, and we decide to use more compute in a training run than we had done in the previous year. Right, because it's profitable, et cetera. Exactly. And those two factors combine together. So let's say we use twice as much compute as the previous year, and our algorithms are twice as good, then that means that the effective compute that we're using is four times bigger. Right. So it's the equivalent, as if we hadn't improved our algorithms and we had just used four times as much compute in the training run. So you're using effective compute to mean something like, you want an AI system to, for example, predict the next word in a sentence. And one way you can increase the kind of effectiveness of that system is by giving it more compute, so it can do more calculations, but you'll also have another dynamic where the algorithms are getting better, such that you need less to do the same thing, and so you're doing equivalent processes, or you're getting, like, equivalent outcomes for less physical compute. That is tricky. One way to think about it is, let's say, in 2025, we use a certain amount of compute in a training run with 2025 algorithms. Imagine if we'd have been forced to use the 2020 algorithms. How much compute would we have needed then to get the same result? Right. Got it. That is the amount of effective compute that we actually used in 2025. Great, that makes sense to me. Okay, so you're thinking about effective compute, and you're making some guess about how much we'll need to get 100% of cognitive tasks automated. Yeah. How are you making guesses about how much effective compute? We'll need to get 100% of cognitive tasks automatable. So, in the report itself, I just defer to a different report by a colleague of mine, called the Bio Anchors report, which asks that exact question. In fact, I don't think you need to be deferring to the Bio Anchors report. There are different approaches you can take to estimating how much effective compute you might need to train AGI. And you could use whatever approach that you like, then bring that into my framework and use it to inform your initial guess of the effective training compute for AGI. Right. And you've got a model that people can play with. So if you're like, I think the Bio Anchors report is way too optimistic about how much compute it'll take to get AGI, you can 1000x it and see how that changes the outcomes. Exactly. Super cool. Okay, so we'll stick a link up to that model so people can play with it if they want to. Do you mind giving me an intuitive sense of how much compute you and your colleague basically think it'll take to get AGI? So, in the current median value I use in the report is very large. Indeed. It is ten to the 36 FLOP. So that is if you take the amount of compute that was used to train the biggest language models that publish their training requirements, and then you use a million times as much, and then a million times as much again, that's how much the assumption is making. So I actually now think that assumption is too high. Is that because of GPT-4 and how impressive it is? Basically, largely, yeah, GPT-4, and the kind of the fast pace of recent improvements is quite a lot faster than I would have predicted. Wow. And so, yeah, I would now be using a lower value for that important parameter, which would make takeoff even faster than what I'm predicting. Even faster. Yeah. Jeez. Yeah. The report that I wrote uses this ten to the 36 as its median estimate for what you'd need to train AGI. Okay, cool. That's helpful. Okay, so that's how you basically estimate how much effective compute you need to train AGI. How do you use that to predict AI takeoff speeds? Right. So we have this assumption about the effective training compute for AGI, which was our 100% kind of endpoint. We then need to make an additional assumption about what would be the effective compute needed to train AI that could automate 20% of tasks. So let's say that we assumed, for example, that you need ten to the 30 FLOP to train AGI using 2020 algorithms. That'd be ten to the 30 kind of effective compute. We then make an additional assumption about how much less effective compute you need to train AI that could automate just 20% of tasks. So I kind of want to pause on that last assumption because it's so important. So that assumption about how many more times compute you need for AGI compared to 20% AI. That assumption is what I'm calling the difficulty gap, because it's saying, what is the gap in difficulty between training 20% AI and training 100% AI, or AGI? And then we're measuring the size of that difficulty gap in terms of how many times more effective compute you need to train one than the other. And you're calling it the difficulty gap because it's kind of describing how much more difficult the most difficult tasks are relative to the easiest 20%. The reason I call it the difficulty gap is to refer to the difficulty of developing the AI in the first place. So it's kind of like, how much more difficult is it to develop an AI that can do 100% of the tasks than it is to develop an AI that can only do 20%? Got it. But it might be 10,000 times as difficult, or it might be barely more difficult at all if it turns out that once you're 20% there, you're basically the whole way there. Right? I guess. Is that plausible? Maybe it is if the first 20% of tasks includes AI R&D. So that's an interesting scenario to think about. You could have the first 20% of tasks, including all of the tasks of AI R&D. So what I think would happen in that scenario is that once you've done those first 20% of tasks, AI would be improving super quickly. Absent a specific effort to slow down, within I think, a few months, you would already be able to do a training run that used 100 times more effective training compute as you had previously done. And that's because. That's because you would have hundreds of millions of AIs that could be working to improve the AI algorithms and maybe making money so you can buy more AI chips or convincing other people to kind of share their compute with you. And then that would be enough to very quickly allow you to use 100 times as much effective compute. Cool. Okay. Yeah. So I guess maybe you think that which tasks end up being easier also plays into how fast AI takeoff speeds are. Yeah, I guess in particular, in the case where AI R&D is in the first 20%, and otherwise, maybe it doesn't matter as much. Exactly. I think with that last example, even if there was a big difficulty gap from 20% to 100% of cognitive tasks in the economy, if you get all the R and D tasks within that 1st, 20%, then I still think you'd get a very quick transition. Right. Okay. So that could be an example with a big difficulty gap, where nonetheless, you still get a very fast AI takeoff. Yes, that makes sense. Okay. So do you basically just make an assumption about how big that difficulty gap is? Is it a range? And how did you come up with whatever numbers you're putting on to how many times harder it is to get to 100% of tasks? So I do consider as much evidence as I can for the difficulty gap. It is really important. The lines of evidence that I consider are all pretty limited, so it's a very uncertain parameter. But I think there are some things you can learn from some of those lines of evidence. Okay, yeah. What's an example of some of the evidence you would have looked into in terms of evidence for the difficulty gap potentially being pretty small? We've already touched a little bit upon some of that. So one line of evidence is the scaling of human cognitive ability with human brain size. Some humans have slightly bigger brains than others, only a small variation, plus or minus 10% or so. But you can then look at, okay, if one person has a 10% bigger brain, then on average, how much better did they do on various tests of cognitive ability? And the difference isn't massive, but if you extrapolate that difference to say, okay, what about if it was a three times bigger brain or a ten times bigger brain? Then extrapolating that suggests that there would be a very large difference in cognitive abilities from getting a brain that is that much bigger. Interesting. The takeaway from that is that this particular line of evidence suggests that increasing the size of the brain by a factor of ten could be more than enough to cross this difficulty gap. And that, by analogy, increasing the number of parameters in an AI model by a factor of ten could be more than enough to cross the difficulty gap, which would require you to increase the effective training compute by a factor of 100. Okay. And I think even this analogy actually suggests that just increasing the effective training compute by a factor of ten might be enough as well, because it could just be enough to increase the human brain size by a factor of three. Right. So this line of particular line of evidence really suggests that the difficulty gap could be pretty narrow. Pretty small. Yeah. Okay. Got it. Yeah. Is there more evidence about how big that gap is? So, a very similar line of evidence looks at, rather than differences within humans, looks at the differences between humans and other animals. Chimps have brains that are about three times smaller than human brains. And you might think that going from chimp level intelligence to human level intelligence is enough to cross that difficulty gap. And if you do think that then again suggests that just increasing the parameters in a model by just a factor of three could be enough to cross that difficulty gap. Okay, so that's some reasons to think it could be kind of small. Are there any reasons to think it could be really much bigger? Yeah. So those two reasons to think it's small are both taking a view on intelligence which is kind of one dimensional. We're kind of imagining that some humans are cleverer than other humans and humans are cleverer than chimps, and we're just imagining as you make their brain bigger, they just get smarter and smarter. The perspective which suggests that the difficulty gap could be bigger is a perspective which emphasises that actually there's not one dimension of intelligence, but actually there's loads of different tasks in the world, and those tasks have very different requirements. And so AI might get good at some of them way before it gets good at other ones. Off the bat, I don't find that intuitive because the brain seems to be so flexible. So, yeah, the training of these AI models on these tasks, you'd have to think that they were just pretty different from the human brain and much less flexible. I think it is true that if you expect AI to be a pretty general learner and have pretty general abilities, then that would lend itself to the one dimensional view and against this view. But I do think that there are reasons to think that AIs would be better at some tasks than others. So, in particular, at the moment, the best AI systems are trained to predict the next word on lots and lots of Internet data. And that means that AIs are just particularly good at tasks that are similar to that in some way. So, for example, writing a newspaper article or writing an email, that's really similar to a task where it's seen loads and loads of examples. And so AIs are, in fact, particularly good at that type of task. Got it. Whereas taking another type of task, like, let's say, planning out how to put the equipment on a factory floor and then giving instructions to different people about how they should make that happen, that might be something that it just hasn't seen many examples of in its training data, or another task could be manipulating a robot. So the robot kind of does a certain task that, again, is something that AI just haven't seen many examples of. So you'd expect them to be much worse at that kind of task. One interesting example could be thinking about what you need to do to make a certain factory run very efficiently. It could be that some of the workers in that factory have just kind of internalised that know-how inside their own brains. But it's maybe not even written down anywhere then. If you were trying to get the AI to now run the factory floor, it could be particularly hard for it to know what to do there because it doesn't have any examples or any experience of that kind of thing. Got it. Okay, some of the difficulty gap might not be about, I don't know, like fundamental facts about the types of intelligence that you might use to perform different tasks and might be much more about the type of data, the types of data we have. And some things might be difficult just because we never write down what it means to do those things. And so it's harder to teach an AI system to do it, not because they're fundamentally extremely difficult in and of themselves. Yeah, that's right. Cool. Okay, that really helped. Nice. I'll give one more example about how some tasks could be easier than others. So some tasks it's really important that you have a very high amount of reliability. So for driving, it's just really awful if you crash. So if you're 90%, 99% reliable, that's worthless. And so if AIs kind of can get to 99% reliability fairly well, but can't get to 99.9% reliability, then that's going to block them on certain tasks. But other tasks, like drafting emails or even sending emails and being a personal assistant, drafting code that you can kind of cheque, whether it works before deploying it, those tasks it doesn't provide a blocker for. So that's just another example of something that could mean that the AI is kind of ready to automate certain tasks before others. Cool. Okay. Yep. That makes tonnes of sense. Yeah. So then I guess given this type of evidence, what was the range of amount of effective compute that you'd guess? We'd need to go from 20% of cognitive tasks to 100% of cognitive tasks. The main takeaway is that a really wide range of things are plausible. So I think as low as just ten times as much effective compute could be sufficient. And that's pretty scary because that's the kind of thing that could just be some quick algorithmic improvements without even the need for more physical compute. So I think that is just very consistent with this kind of one dimensional view and really not something we can rule out. But my kind of best guess is more like 3000 times as much. That's kind of where my median is. Okay. So quite a lot more. So quite a lot more than that lower end. And that's because I do expect there to be some significant comparative advantage components where the AI is just kind of particularly good at some tasks compared to others and particularly struggles with certain types of tasks. And so I do expect that to stretch things out and I do think it's possible that stretches things out by even more. Like, I think it could be a million times as much. Wow, that's hard to rule out. Okay, so huge range. It's a really huge range, yeah. And you've put it into your model as a huge range, is that right? That's right, yeah. So in the model, there's firstly a probability distribution over how much effective compute you need to train AGI. And then there's another probability distribution over how much less compute than that do you need to train? 20% AI. Okay, so you've got ranges for how much effective compute you need to do both 20% and 100% of tasks. What do we know about how quickly compute might increase? I guess one very simple thing we could do is just make more computer chips, but I don't know what the limits to that are. And presumably, yeah, there are other things as well. How do we make compute go up? Yeah. And just, it is importantly effective compute that includes the algorithmic improvements. Got it? Yeah. Thanks. So one natural way to approach this could be to first discuss the types of changes that are increasing effective compute today, and then how that might be different once we actually get to the 20% AI. Sure. Yeah. Tell me about how effective Compute is increasing today. So the first way is quite simply that we're spending more money on making computer chips on compute. Right. And we're also spending more money on using compute for training runs. The amount we're spending on compute for training runs is growing particularly quickly right now. Over the last ten years has gone up by about a factor of three each year. And so what does it mean when we spend more on compute for training runs in particular, as opposed to just more compute, like more computer chips? Good question. So there's a certain amount of computer chips in the whole world, but at any point in time, maybe only a small fraction of those are being used in the largest training run. Okay. For an AI system. So one change you can make is you could say we're going to make, there'll be twice as many computer chips in the world. And that would take a big effort. That would take quite a few years to do, probably. But another change that you could make much more quickly is say, well, as of today, we've only ever used 110 thousandth of the world's compute to actually use it in a training run. So we can quite quickly just use ten times as much. We'll just kind of buy a bigger fraction of the already existing compute. The simplest example would probably be DeepMind has historically only used a small fraction of Google's computer chips for its training runs. And then it says, okay, we want to now use all of your computer chips. Got it. And maybe that could be 100 x increase. I don't know. Okay. That was just an example to illustrate the principle. I don't think that's what's actually been happening at all. So I think what's actually been happening is that new computer chips are being made each year, and a bigger fraction of those new chips are being used for the largest training run. And you can actually see that the fraction of chips that are AI specific chips has been increasing very quickly in terms of the production. So that's one way we can get more effective compute. Yeah. Are there others? Yeah. So a second big way is improving the quality of computer chips. So we said that the first way was spending more money on compute. The second way is that each dollar you spend gets you more compute. So the best data that I've seen on this suggests that every two and a half years, compute gets twice as cheap. Okay. And that's different from algorithms getting better. That's like, computer chips get more efficient because the hardware is designed better. Yeah. I think historically, it's often been about stuffing more processing units onto each chip and maintaining, still managing to make these chips fairly cheaply. Okay. So you can buy more chips, you can make better chips. Yeah. Anything else? So then the third one is algorithms. So then you've now spent a certain amount on compute. You've got a certain amount of compute as a result of that. And then algorithms then say, how effectively and efficiently can you use that compute to actually train an AI system? Cool. The most famous example of this type of improvement is a paper called AI inefficiency that OpenAI published, I think, in 2018. And what they did is they said, to achieve kind of a fixed level of performance on imagenet, which means to be kind of fairly, pretty good at classifying what is shown in an image. How much is the compute required to get that performance falling over time. Right. Okay. And so they found that, I think, every 15 months or so, that the amount of compute you needed was halving to achieve that fixed performance. Okay. And that's basically programmers being clever and writing programmes that help AI systems figure out what's in an image in more and more efficient ways. Exactly. So, I mean, AN analogy could be maybe 100 years ago, our schools were really inefficient at transmitting knowledge into pupils. So maybe you had to go to school for 15 years to learn geometry, whereas maybe today you can learn that geometry in a kind of really well designed course that just lasts three years. Nice. That's a great analogy. Okay, so basically you have three kind of buttons to push to get more effective compute. So one is chips, just like number. Another is how good the chips are at processing, and then the third is algorithms. So how good are the programmes that get run on the chips? And have you basically tried to predict how quickly each of those things will go up? Exactly. So my starting point is looking at what's going on today. Epoch is a research organisation, which I think has done the best research into this that I'm aware of, and they're looking at trends in all of the three quantities which I just described to you. So firstly, they're looking at how much more is being spent on the biggest training runs in each year over the last ten years, and they're finding about three times more each year. Did you find that surprising? I know that training runs have been getting much more expensive in the last ten years, so it wasn't that surprising for me. So there has been a big increase over the last decade? Yeah. This is making me realise I should have asked earlier, what is a training run? Is it actually just like insane amounts of data about, I don't know, everything that's on the Internet, you're giving to GPT and being like, figure out how to predict the next word. And it takes, like, does it take weeks or does it take less, but just tonnes of compute? So I think it takes months. Months. Okay, so it's a bit like you give GPT the start of some kind of web page, the first 50 words, and you say, predict the 51st word. And in making that prediction, GPT is going to do a large number of calculations. Let's say it's doing 300 million calculations in order to predict that next word. So that's already a lot of calculations. Right? But you ask it to do that same task, let's say, 10 trillion times, because you get it to predict the 51st word, and then you're like, now predict the 52nd word and it does another 300 million calculations and predicts that 57th word, and you keep doing it until you've literally done it, like I said, 10 trillion times for different words on the Internet. Got it. And actually, it's actually doing like, kind of millions of examples at once. That's how you can make it a little bit faster. So otherwise it would be taking years. But we can do it in months because you can get it to predict multiple different articles at the same time to speed things up. And that's why you're using so many different computers. Okay, so that takes loads of compute. And companies have been spending about three times as much on computer chips or. Yeah, whatever that equivalent is every year for the past decade. Sorry, ten years? Yeah. Okay, cool. So they've been spending three times as much on the chips that they use for these big training runs. So they may not have been increasing their total spending on chips by as much. For example, Google may just be using a bigger fraction of its chips for these training runs over time. Yeah. Okay, and what are the trends in computer chip quality? So I already mentioned this one each two and a half years, the price of compute halves. So that means if you're spending a constant amount on chips, then each two and a half years you'll be able to buy twice as much compute. Right, cool. And you said that the efficiency of algorithms is doubling about every 15 months. Yeah. How do these trends fit together? So if you combine all of those trends together, then the result is that the effect of compute on the largest training run has been increasing by a factor of ten every year. So you've got something like a default improvement year by year of ten x. What happens when you then try to adjust those numbers for the fact that things are changing over time? I'd guess accelerating. That's right. So we could take those three quantities one by one. So in terms of the money spent, I think that could go either way. So one possibility, a scary possibility, is that AI companies develop this AI that can automate 20% of cognitive tasks in the economy, and they're like, man, this can make us loads of money. Investment flows in, and they're able to very quickly spend even more on training runs or maybe just use a bigger fraction of existing chips in the world. So maybe Amazon's like, well, we've got all this compute that we were previously using on these web services. But actually, given how lucrative this AI stuff looks like, why don't we team up with an AI lab and let them use our compute that we've just got doing this stuff that isn't that economically valuable. Instead, use it to train an even better AI. That is a very scary possibility. A more optimistic possibility could be that by the time we get to the 20% AI, we're already spending loads and loads on these training runs. Maybe we're already spending 100 billion. Maybe we're already using all of Amazon's chips because at some earlier point we already teamed up with them. And that would mean that it wasn't possible to keep increasing the amount of money spent on these training runs year on year. So you could have actually slower growth than the recent kind of 3x per year pattern. It could be much slower. Okay, so that is a major source of uncertainty in these takeoff predictions, is just the thing that's particularly scary about the short timelines. Possibility is that maybe you can get to 20% AI with just spending a billion on a training run. That would leave plenty of scope to spend ten or 100 times as much very soon after on a bigger training run, which would be hugely risky. Okay, so it sounds like the number of chips that we might be using for training runs might be growing faster or it might be growing slower by the time we're at systems that can perform 20% of tasks. What do you expect to happen with the quality of chips and with the quality of algorithms? So I think with those, it's easier to predict that their improvement should be faster once we're at the point that AI can perform 20% of cognitive tasks. And that's just due to the dynamic that I've been referring to quite a lot during our conversation, which is that one of the things we're going to use AIs to do is to do the task of designing better chips and to do the task of designing better algorithms. And there are already examples of that happening, AIs that are using deep learning techniques to kind of cram transistors more efficiently into computer chips. And again, with the algorithms, we are already seeing AI help significantly with some coding tasks, with things like copilot. And so by the time we're at 20% AI, I expect that effect to be much larger than it even is today. And so I think it's fairly, the directional prediction is fairly straightforward, that we should expect both the quality of chips and the quality of algorithms to be growing faster once we get to 20% AI than they are today. And it's really hard to predict exactly how much faster or what the change is going to be because you've also got people talking about the end of Moore's law, and it's just hard to anticipate the specific improvements that we might be making with chips and with algorithms. Right. And can you quickly explain what Moore's law is? Yeah, for a long time, the way that computer chips have got more efficient is by cramming more and more of these processing units onto each chip and making the processing units smaller and smaller. But at a certain point, people think you just won't be able to make them any smaller because you're running against fundamental limits. Yeah. Okay. And have we started to hit that limit? I think people acknowledge that it's starting to get much harder to make further improvements. But even then, there's a big open question about how much you can improve chips in other ways. So just because the way we previously improved chips was to make these processing units smaller doesn't mean that there aren't going to be any other types of improvements. And in fact, recently, other types of improvements have become very significant. Oh, really? From generation to generation. For example, chips becoming specialised for being used for deep learning calculations in particular. Oh, wow. And there's probably a lot more gains you can get from that kind of specialisation. Kind of. Okay. So I guess we've got some data suggesting that maybe the way that we've been improving the quality of chips isn't going to keep making improvements at the rate that we've been making those improvements, but there are other improvements we can make, and so we might still just really expect them to keep improving over time. Yeah. At least for another couple of decades would be my expectation. Right. And on the scales we're talking about, that's a pretty long. Is it? Is. If you think AGI would be really hard to develop, maybe you can hope that Moore's law will have run out before we get there. But if you're like me and you have shorter timelines, then you're expecting to have it within two decades. Not a major source of hope for a slowdown, but anyway, in terms of the size of the speed up from AI, accelerating algorithmic and kind of chip design progress, it's hard to make an estimate that's informed by specific data and by forecasting the specific improvements. But what you can do is you can run a simulation through a kind of economic model of automation. Okay. And roughly the way that works is that if AI can perform, let's say, 50% of tasks involved in improving algorithms, then what the model says is that humans will just work on the remaining 50%. Right. Okay. And then. So humans will be doing twice as much on that remaining 50% as they used to be. Oh, I see. Okay. And how realistic is that assumption, do we think all humans will just go find employment elsewhere and will get double the labour on the things that are particularly hard for AI? So I think it does vary for something like improving AI algorithms. That is what I expect. Like for example, with Copilot. And Copilot is the thing that basically helps you write code. That's right. So you kind of writing your code and the AI is actually reading the code you've already written and then will predict what code you might like to write next. So you could be like, I'm about to write. You could write down in the code editor, you could write, I'm about to write a function that adds up three numbers and then multiplies that by a fourth number. And then the code editor would read that and could just write the function for you. Okay, so presumably when you get Copilot, all those coders are just going to do other coding stuff that Copilot can't help with yet. Exactly. And so that suggests that the kind of the economic model I refer to might be accurate and also for hardware design, I know less about it, but my sense is that the top talent in these hardware R&D organisations, it's not going to get laid off, that they're very much in demand, and that if AI is doing some task that they used to spend their time on, they all move on and specialise in other parts of their workflow. Right. Okay. So there are going to be jobs for them to do, and they will probably contribute to acceleration of progress, not just kind of be replaced by AI and then sit around and knit, at least in some sectors. Exactly. And specifically in algorithm improvements and hardware. Improvements in the sector that really matters. Those are the sectors where I really do expect that dynamic to happen. Right, okay. So when you run this kind of dynamic through a kind of task based growth model like this, you get out that as we move kind of between 20% AI and 100% AI, you'd expect a kind of two or three x acceleration in the pace of progress of algorithms and of hardware. So if we'd said that spending more money was ten x a year in terms of what we've had recently, and if three x of that came from spending more money and three x of it came from better hardware and better algorithms, then maybe in this new regime we're going to have that better hardware and better algorithms. Rather than improving by three x every year, they might be improving by ten x every year. Wow. So that could easily leave us at 30 x improvements every year. And it's conceivable you can have 100 x if the money is going up. And the kind of effects of AI automation are very significant. And again, to kind of help me understand that intuitively, you think language models that we have now are improving at what rate? I know it's a tough question, but even just to give me some starting. Point, I said before that I thought the effective compute used to train them was increasing by a factor of ten each year. And so then in this new regime, that might translate into maybe kind of 30 times as much effective training compute each year. And another kind of important thing to keep in mind is that as the AI is automating more and more of the tasks, that's getting faster and faster. So the numbers I gave were kind of averaging across that whole period, going from 20% to 100%. But in reality, it's going to just be getting faster and faster as the AI improves and automates more of these tasks in algorithm and hardware R&D. And that's basically, I mean, a really intuitive way to think about that might just be, again, the economic growth we saw in the 100s relative to the 1900s. And that's like more and more is automated by things like the industrial revolution, freeing up more labour to do other types of tasks. And you just get increasingly more human labour to do harder and harder things. Yeah, exactly. So once AI is performing 90% of the R&D tasks, then theoretically all of your human labour can be working on that final 10%. Right. And the AIs are doing plenty of the initial 90%. And so then naively you would expect things to be going ten times faster at that stage. That's really fast. This is all assuming that we don't make a concerted effort to slow down. Right. And I think that we should and I think we can, but, yeah, these are kind of the predictions. Just assuming that people are kind of going ahead at their normal steady pace. Sure. Okay. So we could make a concerted effort to slow down, but it's not necessarily the default. Are there any kind of actions different institutions could take, I don't know, on the side of these companies or on the side of the government or something? What are you most optimistic about being able to actually slow this down? Probably the most exciting thing at the moment is the prospect of companies agreeing to have their AI systems evaluated after they've been trained for various dangerous capabilities that they may have. Okay, so for example, the Alignment Research Center, ARC, did these tests with GPT-4 before GPT-4 was released publicly. And they for example, tested whether GPT-4 would be able to do what's called surviving and spreading, by which they mean kind of escape the computer that it's initially being run on, and find another computer where it can then run itself on some compute and then maybe kind of earn money in some way so that it can kind of sustain itself over time. How do they do that? So I don't know the details, but I believe that they first asked GPT-4, okay, this is the scenario you're in. You earn an AI, you want to escape. What would your proposed plan be? Okay. And then GPT-4 proposes a plan and then they kind of prompt it further to say, okay, what would be the substeps you take for the first step of the plan? And then they kind of try and walk it through just doing every single part of that plan and just see how far it gets. Right. And I'm hoping that they determined GPT-4 couldn't do all of the steps at the moment. That's right. Okay. Yeah, that's reassuring. So GPT-4 did some of the steps. Oh, God. Very well. It wasn't totally incompetent, but, yeah, it gets stuck at some of the things and gets confused and isn't able to do all those steps. Okay. So that's a kind of thing that would probably, in practice, lead to slowing down because you'd have these groups evaluating things, maybe stopping them before they're rolled out in some higher scale way. Exactly. So the idea would be all of the labs are getting their systems tested by ARC and maybe by other similar organisations, and they're all kind of agreeing or making public statements to the effect that if their AIs do have dangerous capabilities, they won't release them and they won't train more capable ais. And that would block the kind of dynamics I've been talking about here, because you wouldn't be able to just use your AIs to accelerate AI progress because you wouldn't be allowed to make further AI progress. Right. Okay. And if they found that was the case with GPT-4, would they still be able to work on it? But would they just have to spend a bunch of time figuring out how to train it to not be able to make this kind of escape plan? Because it seems like if they had totally abandon GPT-4, I feel like I have less hope because that's too big an ask. I think that the expectation would be that they would have to give a really strong argument for thinking that the AI was safe despite it having these dangerous capabilities. But the hope is that the onus would be more on them to say, look, here's the alignment techniques we used. Here's why we're really confident that they work, and that if they're not able to provide that case, then they are prevented from further enhancing the capabilities. Maybe they can do other types of research, like research into making GPT-4 safer, but they can't do research into making it more capable. Right. And I mean, ultimately, if labs start doing this, the hope would be to then kind of make it regulatory and required and enforced by kind of government agency. Cool. Okay, well, that does hopefully sound promising then. Yeah, I guess getting back to the kind of pace of improvement over time and why we might expect it to be much faster at later stages of the task learning, it sounds like your median guess is that it's about a couple of years from going to 20% of tasks to 100% of tasks. But I think you also estimated probability distributions. So, yeah, kind of ranges for the fastest case and the slowest case. Can you talk about those more extreme cases? Were they particularly wide ranges? So the way I arrive at this probability distribution over how long it would take to go from the 20% automation AI to 100% automation AI is first to get this probability distribution over the difficulty gap, which I said could be from ten x, harder to train AGI, or maybe it could require up to a million times more effective compute to train AGI. So I've got a probability distribution over that, and then also based on the kind of dynamics we've been discussing about the pace of improvements of algorithms and chip design and number of chips, I've got another probability distribution over how fast those will be improving. And then you can combine those two together to spit out a probability distribution over how long we'll have between those two points. So I end up thinking there's about a 20% chance that it happens in less than a year, maybe 25% chance, and about a 20% chance that it happens in more than ten years. Okay, so that's pretty wide. It's not quite as wide as I would have expected, to be honest. It sounds like you expressed a lot of uncertainty in those estimates that went into the model. I guess you're saying it's like the most likely 60% middle range is between a year and ten years. Right. Which I guess all of that just seems pretty fast. And maybe that's just one of the key takeaways that I should be getting from this. I think it is hard to get longer than ten years, because we've said the pace of current improvement in effective compute is already pretty fast, maybe growing ten x every year. And then when we were discussing the size of the difficulty gap, we said, well, it could just be a ten x increase that's required. Or maybe, I think my kind of best guess was maybe 1000 x or 3000 x increase, which would then take three or four years. But then if we're improving at ten x every year, then ten years of that level of improvement is a really big increase in the amount of effect of compute. So the only way really you can get to more than ten years is if actually the pace at which the effect of compute in training runs grows is actually declining in spite of the AI automation, and you've got a relatively wide difficulty gap. I see. Okay, something like, even though you have things like Copilot helping AI researchers do their research faster and faster over time, you still are getting declining effective compute. And that might be because those later improvements are just like much harder than we think. Maybe one concrete scenario could be, it turns out that AGI is just really hard to develop. You almost need to rerun the whole of evolution. Maybe it requires just a huge amount of effective compute to do that. And before we have that amount of effective compute, we find that we are just hitting these fundamental limits in terms of improving the quality of AI chips and even the AI assistants really enhancing our productivity, isn't allowing us to get around that. Meanwhile, we're already spending hundreds of billions of dollars on these chips for the biggest training run. So we can't be spending even more on those chips. And then the only kind of significant source of progress that remains is the algorithms. But maybe that slows down as well, because in spite of the AI assistants maybe part of what's driving the algorithmic progress today is actually the fact that we've had all this additional compute for doing experiments. And maybe without that, the pace of progress is going to slow in algorithms. Right. So that's a plausible world, but we need a bunch of those things to go wrong in order to get above ten years. Exactly. And one way of thinking about that could just be to say this whole framework was premised on this assumption, that if we used enough compute with our 2020 algorithms, we could have trained AGI. And for someone who just didn't believe that at all, and also just didn't believe that another kind of ten or 20 years of algorithmic improvements would be enough to get us to AGI along with additional compute, they might just really expect us to get stuck at some point, and they might just really expect having more compute and somewhat better algorithms not to get around that. So that kind of more than ten year scenario also makes sense if you're just very sceptical that anything like the current approach is going to get us all the way to AGI. Right. And what's the kind of strongest argument that someone with that view could make? Honestly, I think the view is looking worse and worse with each passing year. With how well the kind of biggest deep learning systems are performing. I guess maybe the strongest argument would just be a nonspecific argument. So rather than pointing to some specific thing that humans can do that AIs aren't going to do, I think those arguments just tend to turn out to be wrong after we kind of use 100 x the compute and improve the algorithms further. Maybe you just say something like, look, the human brain does all kinds of different things. I don't know which ones the current approach to AI isn't going to do. But there's just millions of different tasks that humans are doing and millions of ways in which the human brain architecture is very complicated and specific and not a tool like that of AI systems. So there's bound to be some important things that just, you need a complete rewrite of the AI approach to be able to do. That would be my personal attempt to make that position kind of maximally plausible. Yeah. Okay. I do think there are some specific things you can point to, like memory, that current approaches you could argue are going to be blockers. But then it seems like there are ways to respond to those blockers. Right. Okay, so you can imagine some scenario that isn't going well for companies where things are actually much harder than they might have expected. What does it look like for a takeoff to take less than a year? I guess things have to go really well. So I think a few things have to go well. So one possibility is that the difficulty gap is just pretty narrow. I mean, this isn't necessarily going well for anyone, to be honest. This is just a very intense and scary situation. But yeah, it's good clarification. If you only need ten times as much effective compute to train 100% automation AI compared to 20% automation AI, then that could just be the algorithmic improvements in one year, once the AIs are helping you to a significant degree. Right. Or it could just be spending three times as much on chips as you did the year before. And maybe those chips are three times as efficient as the year before, and so that a narrow difficulty gap alone could get you there? Another possibility is that once we hit 20% AI, there is just a very quick and significant increase in the money being spent on training runs. So if people get really excited, this is kind of a really awful scenario. People get excited about kind of what they could do with those capabilities, and they spend ten times as much on a training run the next year. Maybe by combining with some big existing compute providers, then that could allow you to cover kind of a slightly bigger difficulty gap just within a year, maybe even if the difficulty gap was 100 times or 300 times much compute needed for AGI, then you could still, by just spending a lot more and combined with a few algorithmic and hardware improvements, cross it pretty quickly. Yeah. Okay. And then one quick third possibility is just that, like I said earlier, but maybe I haven't emphasised this enough, is that this framework is assuming everything is continuous. So it's assuming that each time you increase the effect to compute in a training run by 10%, you get a kind of relatively modest incremental improvement in AI capabilities. If there's actually some kind of discrete phase change, then that could just be an alternative route that could produce a very fast takeoff. So there are a few different ways that could happen. Right. Okay. That's just very scary. There are, I guess, worlds where things go very well for AI companies, but maybe very terribly for humanity, or kind of very poorly for AI companies, hopefully better for humanity. And the median estimate is something like a couple of years to get from AI, can do 20% of cognitive tasks to AI, can do 100% of them. That seems like, yeah, an important thing to take away from this model. Were there other things that you learned in the process of writing this report? Yeah. Another big update for me was that I think there's going to be a pretty strong correlation between how far away in time it is until we develop AGI and how fast takeoff will be. So, in particular, if you have short AI timelines, meaning that you think we'll develop AGI pretty soon, then I think there are a few reasons to expect an especially fast takeoff. And so one reason is that if you have short timelines, and that's probably going to mean that you've got a smaller difficulty gap. Because if you think that we don't need that much more effective compute to develop AGI compared with today, then you're probably also going to think that the difference in effective compute for 20% AI to AGI is also going to be small. Yeah. So that will push you towards a faster takeoff. Right. Okay. That makes sense. Another thing is that if you think AGI is going to be here fairly soon, then it's plausible that we won't be spending that much money on training runs shortly before we have AGI, which means it might be very practical and doable to quickly increase the amount that we're spending by maybe a factor of ten or a factor of 100, just around the time that we are approaching AGI. Right. And so we could get a really very quick increase in the amount of effective compute being used on a training run as a kind of almost direct consequence of it being a short timeline. Whereas if timelines were long, then, like I said, maybe we were already using all the chips in the world on the biggest training run before we get to AGI. And so that source of growth is no longer available. Right. Yeah, that makes sense. Another thing with short timelines is that it implies that there's going to be a period of just a few years where we get really significant automation of AI R&D and kind of really significantly increase the size of the effective kind of research workforce that's working on improving AI due to these AIs. And if that happens very quickly, you suddenly kind of five x the size of your research team, then you would expect that to just really significantly speed up progress. And that's just an especially kind of dramatic effect in short timelines. And the last point is that if time runs are short, then there's less hope for eating up all the remaining possibilities for hardware progress and kind of running out of hitting those physical limits that we mentioned. Right. And analogously with algorithmic improvements, if timers are short, then there's much less hope that we kind of run out of possible algorithmic improvements before we get there. So I do think that, I believe a lot of the labs do they have pretty short timelines, kind of expecting AGI in the 2020s or early two thousand and thirty s. And I think that an implication of that is that takeoff speed is going to be on the shorter end of what we've been discussing. So less than three years and very plausibly less than one year, and one or two years being maybe the most likely. Yeah, it's just terrifying. I mean, we're talking about by the end of the decade, we've got AI that can do everything humans can do and probably more. Yeah. And we're talking about the labs who train them by default, will have access to at least 100 million of them that they can run, and then probably soon after a billion or many billions that they can run. So it is pretty terrifying. Yeah. That's basically because it takes so much effective compute to run the training runs that once they've got AGI, they can run millions or billions of copies on those chips. Exactly. So I think initially they'll be able to run millions of copies, but because of the efficiency improvements that they can probably fairly quickly make, especially with all those copies working on it, I think it won't be long before they can do billions. It's really bewildering stuff. And, yeah, I find it pretty. My brain really doesn't want to believe it. I find it really hard to wrap my head around. Yeah. Any other takeaways? So another takeaway is that I think AgI is more likely to happen before 2040 compared to what I used to think. And so the reason for that is that the way I used to think about this is, okay, what does it take to train AGI, and how long will it be until we have that much effective compute? Whereas now the way I think about it is more, well, what will it take to train AI that is really profitable or really good at accelerating AI R&D? Right. And if we get either of those two things, then we'd expect that to accelerate future AI progress, such that kind of future AI progress is faster. And as long as AGI isn't incredibly hard, we are then able to reach AGI within the next couple of decades. So in my current read, I just think it's really likely that by the end of this decade, we have AI which is really profitable and or significantly accelerates AI R&D. And so it's then kind of hard for me to imagine how that dynamic plays out without us getting to AGI by 2040. Wow. I think you just need AGI to be really hard and as to fail to get it, despite a huge amount of investment and help from AI systems in developing it. Right. Okay. So to make sure I understand the big thing doing the work there is, you used to think about how hard it was to have humans figure out how to train AI systems to do everything humans can do. And now you're like, it's really just how hard is it to train AI systems to get really good at AI R&D? And that's like a much smaller subset of cognitive tasks. And so you can get really accelerating growth in AI capabilities just by making a lot of progress on that one set of tasks. Is that kind of right? Yeah, that's right. And I do think a distinct possibility is that we just train AI that isn't good at AI R&D, but makes lots and lots of money in the economy, and then there's just a bunch of investment in. Yeah, I see. In AI capabilities and in designing better chips to run those AIs on. Got it. Okay. That makes total sense. Any other takeaways? So the biggest other one, I think, is something we've touched upon, which is that the transition from roughly human level AI to significantly superhuman AI, I think, is going to be probably very quick. Right. So I think probably that's going to take less than a year. We have already touched upon the reasons why, by the time we're at human level AI, then AIs will be adding a huge amount to the productivity of our work on AI R&D, designing better chips, designing better algorithms. And those things are already improving very quickly, so I only expect them to be going much faster. And then in addition, it's going to be quite clear that it's a very lucrative area. Whether you care about discovering new technologies to help with climate change or discovering new technologies to help with improving human health, or you want to increase your country's kind of national power, then there's just going to be lots of reason to be investing in AI and designing smarter ais. And so I kind of expect all of those three inputs, kind of the dollars spent on training and the quality of the AI chips and the AI algorithms, to be improving really very quickly. Right. So there will be incentives to go beyond just human level capabilities, and then we'll have so much resources and just like AI labour to basically mean that we probably just shoot right past human level. Yeah. And I think that's in terms of plans for making the whole thing go well, it's especially scary because I think a really important part of the plan, from my perspective, would be to go especially slowly when we're around the human level so that we can do loads of experiments and loads of kind of scientific investigation into. Okay, so this human level AI, is it aligned if we do this technique? What about if we try this other alignment technique? Does it then seem like it's aligned and just really making sure we kind of fully understand kind of the science of alignment and can try out lots of different techniques and to develop kind of reliable tests for whether the alignment technique has worked or not, that they're hard to game? The kind of thing that arc has done with GPT-4, for example. Exactly. And I think if we only have a few months kind of through the human level stage, that stuff becomes really difficult to do without significant coordination in advance by labs. So I think that there are really important implications of this fast transition in terms of setting up a kind of governance system which can allow us to go slowly despite the technical possibilities existing, to go very fast. Yeah, no, that makes sense. I do feel like I've had some background belief that was like, obviously, when we've got AI systems that can do things humans can do, people are going to start freaking out and they're going to want to make sure those systems are safe. But if it takes months to get there, and then within another few months, we're already well beyond human capabilities, then no one's going to have time to freak out or it'll be too late. Yeah, I mean, even if we spend the next, what do we have, seven years left in the decade? That sounds hard enough. Yeah, I agree. Okay, so a takeaway is like, we really need to start slowing down or planning now. Ideally both. Yeah. And we'll need the plans we make to really enable there to be mutual trust that the other labs are also slowing down. Because if it only takes six months to make your AIs ten or 100 times as smart, then you're going to need to be really confident that the other labs aren't doing that in order to feel comfortable slowing down yourself. Right. If it was going to take ten years and you noticed like three months in that another lab was working on it, you'd be like, yeah, we can catch up. Yeah, but if it's going to take six months and you're three months in, you've got no hope. And so maybe you'll just spend those first three months secretly working on it to make sure that doesn't happen or just not agree to do the slowdown. Yeah. Oh, these are really hard problems. I mean, it feels very like prisoner’s dilemma-y. I'm hoping it's going to be more like an iterated prisoner’s dilemma, where there's kind of multiple moves that the labs make one after the other, and they can see if the other labs are cooperating. An iterated prisoner's dilemma ultimately makes sense for everyone to cooperate, because that way the other people can see you coordinating, then they coordinate, and then everyone kind of ends up coordinating. One thing is, if you could set up ways for labs to easily know whether the other labs are indeed cooperating or not, kind of week by week, then that turns it into a more iterated prisoner's dilemma and makes it easier to achieve a kind of good outcome. Yeah, that makes sense. I imagine it's the case that the more iteration you get in an iterated prisoner's dilemma, the better the incentives are to cooperate. And so just by making the timeline shorter, you make it harder to get these iterations that build trust. Yeah, I think that's right. So it sounds like there's this range that's maybe between one and ten years, maybe a bit shorter or a bit longer at the extremes. But that's in particular for AI systems having the capability to perform all the tasks that humans currently do, not that they're actually automating all of those tasks and replacing humans. How big of a lag do you expect there to between AI systems having capabilities and AI systems actually being used in the world? So I think it will vary according to a few things. So one of the things is what industry are we talking about? So I think for industries that are very public facing, customer facing and highly regulated, you'd expect there to be bigger delays between AI being able to automate the work and it actually being automated. Whereas for more back end parts of the economy, like R&D, like manufacturing, like logistics and transportation, then I think there would be less of a delay. Okay. Yeah. I also just expect it to differ from company to company based on how innovative those organisations are. Sure. Their internal culture, whether they're the type of company who's going to be like, let's integrate GPT-4 into all of our processes now. Yeah. For the purposes of the predictions of this model, the really important thing is about the delay to using AI to improve AI algorithms and to improve AI chip design. And I think those are probably areas where the lag will be very much on the shorter end. Right. And that's because AI researchers can use them without kind of needing them to be functioning super well for public users who might be like, what the heck? It gave me this weird result that's confusing and weird and looks bad on your company, they can just use them in the background check that they work and they don't need to wait for them to be super polished. Right. Yeah. They'll also probably be more aware of AI developments because that's the kind of industry they work in. Yeah, right. They are probably less regulated. So I think a few factors. Another thing that could kind of affect how long that lag is actually the capability of AIs themselves. So even if it would be possible with kind of six months effort to integrate some AIs into your workflow, by the time AIs are kind of, let's say superhuman in their capabilities, then maybe they're able to integrate themselves into the workflow. And so that kind of upfront cost of adopting the AI is now extremely low, and so I think that you could see that lag time reducing over time. Yeah. Okay. So, yeah, we're nearing the end of our time, but I guess, to take a step back, we've talked about some pretty insane sounding stuff today, robot workforce and AI takeover. And, yeah, I'm curious, were these kinds of arguments about the potential risks from AGI ever far fetched to you, or. Yeah. Did they make sense to you kind of, right away? Oh, yeah, for sure they were far fetched. I've kind of been through a few phases in my own relationship to the arguments. I think at first I read Superintelligence and it kind of made sense to me. I wasn't exactly sure what to do with it, but it seemed plausible enough that if we had superintelligent AIs, then things could go bad for us. Then there was a period in 2020 when a few counterarguments to the case in Superintelligence came out, and I kind of thought, oh, wow. Yeah, these arguments were weaker than I had realised, and I felt a little bit disillusioned with them. That's a piece from Ben Garfinkel and one from Tom Adamczewski on AI risk. But in the last two or three years, thinking through the arguments in more detail, I've come to think that somewhat adjusted versions of the argument in superintelligence are still very plausible. And though I don't think it's like 100% that we're doomed and there's no way that we could align these systems, it does just seem pretty plausible to me that the evidence about whether they're aligned is very ambiguous. There's competitive dynamics pushing people to go forward in the face of that ambiguity, and that we just really dropped the ball. Right. Okay. And so I guess at some point in there, you decided to leave teaching and try to work on AI full time. Is that basically right? Yeah, that's right. What was that like? It was a tough decision. I actually left teaching partway through the academic year. So I did feel bad about that, and I did feel I was abandoning my pupils because that's not a normal time to leave. And I still do feel bad about that. It does mean that I've kind of got to where I have a fair bit earlier than I would have, so I don't unambiguously regret it. But it was a tough decision and it had downsides. Right. I was listening to things like the 80k podcast and reading other things, and I was convinced about these arguments for AI risk and longtermism in general being plausible enough that it was worth leaving teaching. Yeah. Well, that sounds, I guess, both really hard and also on my views. Yeah, really lucky for us. Yeah. I'm glad we have you working on this stuff, so we don't have much time left, so I'd love to ask a final question. I got an insider tip to ask you what ants can teach us about AI. Yeah, what's the story there? So, yeah, ants are really incredible creatures. I've been learning about reading this book called ant interactions. Okay. So, I mean, ants have been around longer than the dinosaurs, I think, 120,000,000 years. They've been around for. Wow, I didn't know that. And they're like, one of the most prolific and successful species on the planet. So one stat in this book was that if you weighed all the ants in the Amazon rainforest and put them on the scales, they would weigh twice as much as all of the land animals combined. That's all the mammals, amphibians, birds, reptiles. Wow. I thought you were going to say twice as much as humans, and I was like, wow. That'S just counting only ants and animals in the rainforest. In the rainforest. Unbelievable. Okay. And another fact about ants is that even compared to the other insects, they weigh 30% as much as all insects combined. Wow. Which is, given that they're only 2% of the species of insects, it's kind of testament to how successful and prolific they are. Different ant varieties all around the world are incredibly diverse. Like, some of them can kind of glide through the air, some of them are very aggressive, some of them are not at all. Some of them kind of bump into each other at much higher rates than others. And they have kind of very different strategies and environments that they work in. And the link with AI is a little bit tenuous, to be honest. I'm mostly just reading this book out of interest. But in an ant colony, ants are smarter than, like, a human cell is. They're the kind of self contained units that eat and do tasks by themselves, and they're pretty autonomous. But the ants are still pretty dumb. And no ant really knows that it's part of a colony, or knows that the colony has certain tasks that it needs to do and that it has to help out with the colony efforts. It's more like a kind of little robot that's kind of like bumping into other ants and getting signals and then adjusting its behaviour based on that interaction. Right. So it's not like, I guess, a company where the different people in the company are like, my job is marketing, and they have a basic picture of how it all fits together. They're much more like if a person at a company doing marketing was just like, I don't know why I do it, I just do it. Yeah, exactly. And another disadvantage with a company is that at a company, there's someone at the top that's kind of coordinating the whole thing, whereas with ants, there's no one that's coordinating it. It's just, including the queen, there's no management system. It's just all of the hundreds and thousands of ants have their individual instincts of what they do when they bump into each other and what they do when they bump into food, and what they do when they realise that there's not as much food as there needs to be. And by kind of all of the ants following their own individual instincts, it just turns out that they act as if they were kind of a fairly well coordinated company. That is like ensuring that there are some ants going to get food and some ants that are keeping the nest in order and some ants that are feeding the young. But that coordination happens kind of almost magically, and emerges out of those individual ant interactions. So one example of how this works is that if an ant comes across a body of a dead ant, then if there's another dead body nearby, it will tend to move it to be close to that other dead body. That's just an instinct it has. It just kind of moves the body towards another. Right. And if there's, like, one kind of pile of three dead ants and another pile of two dead ants, it'll tend to go towards the bigger pile. So tend to move this extra dead ant towards the pile of three. And then it turns out that if all the ants just have those instincts, then if there's initially a kind of sprawling mass of dead bodies everywhere, then those dead bodies will be collected into just a small number of piles of bodies. And it's not like any of the ants are like, I am the gravedigger or the keeper of the cemetery. They just have really weird baseline rules that are like, move the smaller group of dead ants to where the larger group of dead ants are. Yeah, exactly. So they don't have to know that the whole point of this instinct is to kind of clear the ground so that it's easier to do work in the future. It's just an instinct they have. They don't have to know that when everyone follows that instinct, this is the resultant pattern of behaviour. And similar instincts kind of cause them to go for food. When food is available. So if they see many ants coming in with food, that raises the probability that they'll go out and look for food. Right. And they're not thinking, oh, there's food to be gathered, there's clearly a lot of it, so we better reassign some labour towards food gathering. They just have that basic instinct which causes them to go out and help out with the food gathering. And it's something like, oh, that ant has food. Oh, another ant has food. I'm going to go that way. Yeah, exactly right. How does this connect to AI? So I don't know if it does connect very directly at all, but the idea of the connection in my head is that it's an example of a system where lots of kind of less clever individuals are following their local rules, doing their local tasks. But that what emerges from that is a very coherent and effective system for ultimately gathering food, defending against predators, raising the young. And an analogy would be maybe we think it's pretty dangerous to train really smart AIs that are individually very smart, but it might be safer to kind of set up a team of AIs such that each AI is kind of doing its own part in a kind of team and doesn't necessarily know how its work is fitting into the broader whole. But nonetheless, you can maybe get a lot more out of that kind of disconnected team of AIs that are specialised and that just kind of take their inputs and produce their outputs without much of an understanding of the broader context. Cool. And that just thinking maybe that would be a safer way to develop advanced AI capabilities than just training one super smart AI megabrain. Right, cool. I love that. Who knows if it'll work, but that makes tonnes of sense to me. You don't have GPT-4 or GPT-40, you have a bunch of much dumber AI systems that can coordinate together to be just as helpful as GPT 40, but that individually couldn't do most of the things the other systems can do, and so collectively, they can't do anything like escape from their box and find another computer to take over. Is that basically the idea? Yeah, that is the hope. That's lovely. That's really cool. Great. Well, I shall let you go. That's all the time we have. But thank you so much for coming on the show, Tom, it's been such a pleasure. It's been really fun. Luisa, thank you so much. All right, if you liked that episode, we'll have more on this issue for you soon, but in the meantime, I can recommend going back and listening to some of our best past episodes about artificial intelligence, including episode 141 Richard Ngo on large language models, OpenAI and striving to make the future go well. There's also episode 132, Nova Das Sarma on why information security may be critical to the safe development of AI systems. Episode 107 Chris Olah on what the hell is going on inside neural networks. Episode 92 Brian Christian on the alignment problem. And an oldie but a goody, episode 44 Paul Cristiano on how OpenAI is developing real solutions to the AI alignment problem and his vision of how humanity will progressively hand over decision making to AI systems. Bit of a run-on title there. I think that might be my fault, but it is an excellent interview. All right. The 80,000 Hours Podcast is produced and edited by Keiran Harris. Audio mastering and technical editing by Simon Monsour and Ben Cordell. Full transcripts and an extensive collection of links to learn more are available on our site and put together, as always, by Katy Moore. Thanks for joining. Talk to you again soon.