Facing Superintelligence (with Ben Goertzel)

When I introduced the term AGI, the AI market was real, but AGI was beyond the pale. Everyone was laughing at us like, "If this means adjusted gross income, no one will ever pay attention to you." Invention often happens actually by brickage and cobbling together to make something happen. And the most elegant simple AAM's razor way often comes later. While I'm not an optimist that LLMs can lead to human level AGI because I think they don't have creativity in in a fundamental sense, I I am an optimist that LLMs could take over like 95% of human jobs. I just think that 95% of human jobs can be done without fundamental creativity or or inventiveness. Human value systems are complex, self-contradictory, incoherent, heterogeneous, always changing. They're evolving. The AGI's value system, I think, can be more coherent than human values, but will also be evolving, and you'd like there to be give and take and mutual information between the two. Welcome to the Future of Life Institute podcast. My name is Gus Docker, and I'm here with Ben Girtzel, who is the CEO of Singularity Net. Ben, welcome to the podcast. Good to be here. Thanks for having me. You have been in the AI game for a long time. When did you get your PhD in AI again? Uh my PhD was in ' 89 which was in mathematics rather than AI but I was doing I was in fact doing AI research as a grad student before I got my my PhD. And I probably wrote my first lines of attempted AI code in 79 or 80 like I mean well well before before college even I mean as you know but many people don't AI has been around for some time right I mean I'm pretty old the AI field is is is even older than me. What first convinced you that we could see artificial general intelligence in your lifetime? It was probably 1973 or so. I found a book in the town library in Hadenfield, New Jersey, where we had moved recently. And it was called the Prometheus Project written by Princeton physicist Gerald Fineberg. And he laid out a fairly coherent argument that within the next few decades we would get machines smarter than people, strong nanotech to build build machines out of molecules and then be able to fix the human body so we didn't age and die. And he he said this could be used for many purposes. Could be used for consciousness expansion, could be used for mindless consumerism. He thought that should be decided democratically. and he was trying to get the UN to undertake some global education and voting process to decide, you know, to what end to put all these amazing new technologies that were uh were were in the wing. So I'd seen these ideas in science fiction before that. But then I was I guess age seven or something eight. I I seeing this scientist lay out these ideas in a in a non-fictional form was quite interesting. Of course, not having the web, then there was no way to contact other crazy people who also took this stuff this stuff seriously. But they argument arguments seemed quite credible to me. And then in the late 70s when personal computers you could program at home started to become a thing. I mean that seemed very much in the line with uh with this same vision that that Fineberg had laid out, right? I mean his his book was basically the singularity is near published in 1968 right and as I as I later found out Valentin Church had published a book in Russia called the phenomenon of science late 1960s as well laying out basically the same ideas. So I mean if you were if you were paying attention these concepts were always there and honestly they already seemed sensible just given the advance of computers already at that at that point right that doesn't seem obvious to me given how primitive computers were in the early 70s it doesn't seem obvious that that you would be able to project from there that we would get something smarter than than humans. I mean Jay Good wrote his paper on the intelligence explosion in 65 which is the the year before I was born. And if you look back at the work of Makullikum Pitts and Turring, all these guys in the 40s and and 50s, it was already quite clear from a at least math and physics and biology view. Like it was it was clear, you know, brains can be viewed as computing devices and we're building general purpose computers and they're getting faster and faster and can do more and more step by step. like we could beat the world champion of checkers in the late 1960s, right? You didn't get chess till the 90s, but I mean you had already in the early '7s, you had the early versions of what would become Mathematica and Maple. You had computer algebra. I mean, there there's there's a lot of stuff and historically actually in the late60s there was a lot of AI over optimism, right? I mean, there was a lot of people at that time saying, "Wow, look at all this amazing stuff computers can do. So yeah, obviously they're going to overtake human humans pretty soon. So I think actually a lot of a lot of people were seeing it that way back then because that happened to be like before the the series of AI winters and and and summers set in. I think when I when I was a little kid first reading about this stuff, AI was still in like its first first blush of of of excitement before people realized how how how difficult it was in in in in some ways. But yeah, it wasn't obvious like it is now, right? I mean, now of course everything has changed since chat GPT and and average people are like, well, yeah, you mean the machine isn't smarter than people already? Are you sure? like there's there's no shot value in the concept of of superhuman intelligence at at all now. Certainly wasn't like that in the early 70s. I mean, you you had to you had to pay attention and and and do your research and and and background reading and then and then then think with an open mind. But on the other hand, it's not like I had to invent the idea on my own either. It's like a small town library. is a paperback book by a Princeton physicist laying out the whole argument for super AGI and singularity, right? I mean, the the ideas are there and from sources that seemed reasonably mainstream and and and and credible, right? It just the way culture goes, it takes a long time a long time for things to get from that stage to really becoming becoming dominant. What feels different about the AI moment we're in right now? You could count that moment starting from say 2012 where deep learning begins to really work. You could count it from 2022 when chat GBT becomes something that everyone knows about. Does something feel different about this wave of AI from a technical point of view? Clearly the impact of scale has been has been the main factor. And while I don't really think LLMs or standard deep neural nets are the route to AGI, I I do think the factor of increasing scale of compute and data which has allowed the LLM revolution to happen. I mean I think that is the same primary underlying factor that will let we'll let AGI happen. Right? So I mean the the AI fields had a lot of good ideas for a long time. I mean we were doing they were doing automated theorem proving in the late 60s when when I was a little baby right John Holland invented evolutionary learning genetic algorithms in in in the 70s there were multi-layer perceptrons in late 60s early 70s which are would now be called deep neural nets right I mean this stuff has been around a while but when I was teaching deep neural networks at University of Western Australia in the mid 90s when I was an academic earlier in in my career before I went to industry. I mean, we were doing multi-layer perceptrons with recurrent back propagation and it took like three hours on a fast sun workstation to train a network with like 35 neurons, right? So, it it wasn't actually that the ideas were way way off or something. It was mostly that without greater scale of compute and then greater scale of data as a secondary point really. I mean without without without that you couldn't refine and and tweak and adjust the ideas until they until they actually actually work. So you were just left with the initial conceptual version of things. So I mean scale makes a huge difference. It means that you can run experiments in minutes that would have taken like years earlier in my career literally. Right. So I mean that's that's quite quite big. I mean I think I think we're also now as of the last six months or so we're at the point where AI technology is aggressively accelerating the speed of creation of of AI technology, right? Like you you can't use LLMs to write AGI yet. I mean they're bad at doing complex original thinking, but I mean you you can use them to generate unit tests. You can use them to take your rough notes and turn them into a structured paper for your for for your colleagues. You can use them to write scripts, right? So I mean already like I'm concretely seeing on a technical level stuff that would have taken me five days takes takes one day or something, right? So I think I think we're already at the point where AI is accelerating the creation of of of AI and that's I mean that came because of scale but it's now it's now a separate thing on on on its own and I think I think that cultural and kind of attitudinal aspect is important and is very different now also right like when I when I when I introduced the term AGI in 2003, four, five. I mean, we introduced it as the title of a book we were editing and that that book was finally published 2005, I think, from Spring or just edited a bunch of papers called artificial general intelligence. But then you really couldn't do like a workshop or a session on human level thinking machines at a normal AI conference be it be it a neural net conference or or or a triple AIK whatever like it was it was uh you talked about it in the bar afterwards when you were talking about reincarnation and backwards time travel or something right so it was like it was way out there and that made a big difference. I mean, it meant that only really dedicated or really crazy people were going to work on AGI because it was kind of career suicide, right? Like when I when I got my PhD in ' 89, AI as a whole was career suicide. And I mean, I did my PhD in numerical analysis for that reason in math. Then when I wanted to switch to computer science, I did computer graphics because there were no jobs in AI and graphics was also ma ma math heavy. Right? Now by by the early odds when I introduced the term AGI, the AI market was real like it was okay. There were industry jobs in AI, there were university jobs in AI, but AGI was still still beyond the pale, right? And I remember when we put that term out there, everyone was laughing at us like if this means adjusted gross income, no one will ever pay attention to you. Been sort of watching like when when will AGI as artificial general intelligence overtake adjusted adjusted gross gross income in the in the goo Google search ranking, right? And it's it's a depends on how you count now, right? Not not now now they're actually they're they're in competition but yeah at that time at that time you couldn't talk about AGI like even like an academic seminar in a top university or something. So that the fact that the attitude has shifted so much now makes a big difference. I mean it means it's feasible to get funding to do AGI projects. I mean, it's still it's still very hard because getting funding for anything is very hard, but it's it's no longer very very very very very hard. It's only very hard, right? And I mean, you can much more easily get like bright young students who care about their career to plunge into AGI whereas pre previously just didn't seem like a practical thing to do. It's it's made it's probably made the human constitution of the field less interesting because I I I think when you when you had to fight and be a crazed maverick to pursue AGI you had a lot of interesting characters who were thinking all day for decades about how to make thinking machines and the the first AGI conferences I organized in 2006 8 n and so on were sort of like that now now it's it's It's a morally acceptable thing to do and you can you can make money at it. But the the change in attitude is a is important along with along with the the the technical aspects I think. Yeah. So something seems to have happened since chat GBT went mainstream where it is now basically mainstream to talk about AGI and perhaps even super intelligence. almost all people only take seriously what they can put their hands on and and see in front of them, right? And I mean that includes political decision makers and see CEOs and so forth. Like there's there's not that many people who will take more seriously something they can project and imagine than something they something they see in front of them, right? So yeah, once you had chat GBT there, I mean sounds like it's intelligent. It can it can do a lot of stuff that has the vibe vibe of intelligence and that definitely qualitatively convinced everyone like holy AGI might really be near. It was interesting. I found almost everyone after they spent some number of hours playing without a lens, they could also see these are not AGI and they could intuitively understand intuitively understand why even without a technical background. But but I mean still that that's like you want to convince people you can fly to the moon. Well, if you send a rocket up really high so far you can't see it, then it comes back down, more people are going to believe you could possibly go go all the way to the moon, right? I mean that that's that's pretty simple to understand. I think we have shot ourselves in the foot and I'm thinking of society at large here by making AGI a topic that's that that couldn't be discussed for a long time. What do you think we've lost from not having a a longer conversation and and a conversation in kind of prestigious venues about AGI for longer? I'm not all that sure that the conversations in prestigious venues are are are are usually that that that that valuable. Anyway, I feel like our species generally deals with things at the last minute and after the fact rather than in foresight. And when when people are trying to figure something out in foresight, it becomes mostly a projection of their own ego or their own imagination on on the thing. Anyway, so I mean I think by not taking AGI seriously, we're getting it later than we could have otherwise. I mean I think we could have built human level AGI some years ago. We could have built we could have built it on massively parallel hardware which was kind of became less of a focus of the the field a long time ago. So I mean certainly if if we had a rational world government in 1970 right then that government had said let's develop safe beneficial AGI as a priority of our species which is what Fineberg was promoting in Prometheus project. I mean I think we would have an AGI well well before now. So I mean resource just wasn't put on it in terms of thinking through the sort of ethical, social, political, human implications of of of AGI. I mean, we haven't done a good job of thinking through things like disarmament or combating world hunger or I mean a lot of other much more basic stuff like we're we're still struggling with like trans athletes in in sports competitions or something, right? So sort of I I I guess if this idea had been taken more seriously in the human population at large, you would have had more diverse creativity and more different ways of thinking popping up regard regarding regarding the regarding the theme. But I'm I'm not currently all that positive on our our social and political institutions uh being able to think through very very hard issues in a in in a in in a useful way. Like we're we're still blowing each other up all around the world and 60% of kids in Ethiopia die of not don't die but they're they're their brain stunted due to malnutrition. Right. So I mean as as I've seen through our AI office in in that country. So I mean we're even the issues that are out there like I've been hearing about world hunger and disarmament since I was a baby, right? We we we seem to suck at dealing with those relatively very simple things, right? Like it shouldn't be that hard to stop blowing each other up over territorial disputes and to like send send food to little kids. But but we We're not even good at that, right? So, I don't know how much brilliance we would have brought to bear on on the social issues about AGI even if they'd been sort of more more in the forefront. You mentioned that neural nets, it's an idea that's been out there for a while and we we didn't have the scale of data and compute for for these neural nets to actually work in a convincing way. I guess that took GPUs developed for originally for gaming, but also data being being created and and published on the internet for all the ingredients to come together to for for neural nets to to work. If you look back uh through the history of AI as a field, do you think there are other hidden gems that might where the theory is sound but we don't have the scale or the implementation for it to work yet? A high percentage of the historical AI paradigms probably actually will work when when you when you when you scale them up enough. And you can see that when you dig into the details. So if you look at genetic algorithms and genetic programming. So use of algorithms model modeling evolution by natural selection to to to learn things. I mean there was a whole body of work by David Goldberg at University of Illinois and others. He had a book on the the design of innovation and another book on competent evolutionary learning or something. I mean competent genetic algorithms. So all this work from the 80s and 90s was about using evolutionary algorithms to solve problems but then doing kind of back of the envelope estimates of how much resource you should need for these algorithms. So like if you're if you're trying to maximize a certain fitness function using a genetic algorithm, what's your optimal population size beyond which you're getting diminishing returns? And those optimal population sizes when I calculated them back in the 90s, they were always like orders of magnitude bigger than what we could do on computers. And so I'm just like, well, okay, life isn't optimal. We're not going to do the optimal population size here. We'll just do what we can. And then genetic algorithms are good at solving some problems but they just take forever or don't work well for solving other problems. So I mean in that case I mean you have decades old theory giving reasonably strong reasons to believe it can work much better when you when you scales much bigger. Now if you look at logic based AI which goes back to the 60s I mean again due to lack of data really logic based AI got it started with this sort of stupid methodology of people hand coding common sense knowledge like you would you would type like a human would literally type a logical formula like grass is green lawnmowers are used to mow grass in p logic form maybe explain what the per what the purpose what what was the vision here. Yeah. So the idea with logic based AI was that I mean what makes humans so different than apes and bunnies and so forth is largely our ability to do advanced logical reasoning in ma math and and and science and and and philosophy and and whatnot. And the idea the idea was sort of that this capability emerges almost as a kind of virtual machine on on top of the lower level like machinery for for for seeing it and and moving. So maybe you can just implement that module in a way that doesn't depend that much on the underlying neural substrate. And I mean in that direction a calculator does arithmetic very well. Computer algebra systems do do algebra very well. and they don't try to emulate exactly how the human brain does that. You've just abstracted some some rules and and and procedures, right? And then the the issue you run up against is twofold. I mean, one is okay, but where does all the knowledge come from? Because humans, even when we're doing logical reasoning, in the end, we're getting the knowledge from from seeing and and and acting and and and that then you still have problems of scale because the human brain is doing logical reasoning when it's doing it on a tremendous amount of of of of knowledge, right? So there was an attempt to work around that problem in the AI field still going on till now in some areas but it was it was a major thing in the in the 90s anyway and in the late 80s when I got my PhD which is part of why I ended up doing a PhD in in math rather than AI because this whole paradigm made no sense to me but I mean there basically the idea was just type in the type in the formal knowledge, right? Like if people eat steak, type in a predicate argument relation eats parenthesy people, state close parenthesy, right? And just just type in common sense knowledge about the world and then your logical reasoning system will will reason based on that. And I that never seemed sensible to me for the reasons that are obvious to everyone now. Like the amount of knowledge is just too much. It's fuzzy. It's probabistic. It's it's it's it's messy, right? But on the other hand, there's an argument that logic based AI was inappropriately tred and feathered because of its historical association with the hand coding of knowledge because you you can take a logic you can take a logic system and you can connect it to a camera and a microphone, right? and you can connect it to to to an actuator like the the actual formal mechanism of using logical inference as a core engine of an AGI. I mean that's not tied it's not tied to that old idea of typing in handcoded knowledge, right? And now now a couple things are different. One thing is you can use LLMs to translate natural language into logic formulas, right? So the late the late last six months worth of top LLMs, you can have them take an English sentence and output a bunch of high order predicate logic or or dependent type logic what whatever you want. So you can get you can get a humongous corpus of logic expressions to be into your logic system without having people type them in. them and you can you can also write converters from sensory data into logic expression form and then then you have a scaling problem right then you're says okay well great I have literally trillions of higher order predicate logic expressions how do I do reasoning based based on this right and so that's something that the historical logic based AI field just couldn't explore right so I think in in some ways the AI field wasted a lot of time trying to accommodate for limited compute resources. So in logic based AI because you couldn't do reasoning on the trillion premises, you would spend a lot of time trying to craft the best 500 premises. But it turns out that was that was just a time timewasting way to do things. And we did a lot of tricks to get good results out of genetic algorithms with a population size of a couple thousand. Turns out that's entirely useless now. And it's it's actually much easier you have them to it's much easier to get to do big things if you good things if you just jack up the population size, right? And it requires less thinking and and and less work. I mean, it's sort of like all the work that went into making chess or go playing engines before we got a machine learning based approach. there were there were very complex rule-based approaches to try to try to outdo like basic alpha beta pruning for for for for playing these games. Now now that that's that's all it's all irrelevant right so yeah I think evolutionary learning logic based AI another example is hyper vectors which was big with a that was big in the 80s and 90s people were talking about highdimensional sparse vectors to model episodic memory and and and so forth and you just can do it at large enough scale. The last five years there's a huge literature on doing all sorts of memory with hypervectors, hypervector based chips and so forth. So yeah, my my view is that the use of modern scaled up compute tech and data to accelerate back propagation based deep neural nets historically is going to seem like that just happened to be the thing that got scaled up first, right? And and we're going to see a bunch of other historical AI paradigms get scaled up over the next few years. And then it's by connecting together these scaled up versions of these various historical AI paradigms. It's by connecting them together that we'll probably get to the first the first AGI. And that at that level, this is not even such a controversial point of view. I I think the question about AGI architecture on whose answer I probably differ from most deep neural net big tech people now is you could say well let's take a deep neural net like an LLM use it as the hub then add some evolutionary learning some logic engines out of long-term memory add working memory like add these things on the periphery around the central component of your AGI architecture which is an LLM. I don't think that's going to work to get to full-on human level AGI. Although I think it could work to get to something doing 95% of human jobs, which is how Sam has tried to redefine AGI. I don't think it can get to a system that can really generalize beyond its experience after the fashion of of of people, which is the meaning that I had for the term AI when I introduced it. But I think if you take something else more flexible and more introspective and make it the central component then you know an LLM can be one of the very powerful things cooperating with that central hub and feeding it knowledge and helping it syn helping it synthesize things. So I guess one one question about a architecture is do you want a monolithic or a sort of hybrid approach? like is it only LLM only logic engine or you have multiple components another is if you have multiple components is there one that's sort of more central and if so if if so which is it right and this this sort of debate I think isn't resolved within the AGI R&D community and it might be it might be you could build many different sorts of AGIS by kind of mixing and matching and combining historical AI components that have been scaled up in in different ways. Do you think these limitations are fundamental? So so that we need different approaches working together in order to get AGI. I think that what are diff what are different approaches and what aren't is almost a matter of culture or or mindset rather than than mask. So I I I published a paper or posted on arcs have a paper well there was a short version published in the AGI conference series on called patterns of cognition where I tried to show that all the core algorithms that we're using in my open cog hyperarm project which include logical reasoning some some variations of attractor neural nets evolutionary learning some concept formation I tried to show that all of these can be past as basically forms of approximate stochastic dynamic programming. So you can you can sort of take a whole bunch of AI algorithms that look really different and you can actually cast them in a common ma mathematical form. It's just not the way people are typically look look looking at this right and I mean the reason I did that exploration is I was trying to make sure the infrastructure for our new open cloud kyperon system will let all these things run fast. So if I could reduce them to a common mathematical form then there's just one thing that you have to make you have to make work fast right. So yeah, what's what's the same or different kind of depends on your perspective. Like in in mathematics, you see this all the time, right? Like in in algebra, you have homorphism between structures. In topology, you have homeomorphism. They seem different but kind of similar. Then category theory was invented. It's like, no, these are all morphisms, right? And so you you could see that what seemed to be different branches of math done by different people coming out of different historical lineages actually what they were doing were kind of trivially seen as specializations of of the of the same thing and now everyone accepts they're they're the same thing. So we I mean for sort of tribal reasons we're looking at deep neural nets and logic systems as like super super different things. But like when you're in code working with these things like so we have in open cog highromp which is my big AGI project now we have we have a network of nodes and links and the nodes and links can have symbolic types or floating point numbers associated with them. They have update rules associated with them. Now, pretty much the difference between a neural net put into this network and a probabilistic logic system put into this network, it's like what little nonlinear algebra function do you put in the node to update the numbers that are coming in and going out, right? So, I I mean it is different. It's a different it's a different way of thinking, but it's it's not like building a computer out of cells from a slime mold versus building it out of diamond nanotech or something, right? I mean I mean these are they're actually just in every case we're propagating numerical values through these node and link networks in in a machine and then we're debating about what nonlinear function do we use to translate the input numbers in into an output number but but because people are tribal and like to fight over ego and resources these start to seem like totally opposite camps with totally different ways of thinking because I I mean really a neural net is quite loosely connected with with with with the brain anyway. So I mean people like to say it's biologically inspired in a very distant historical way it is but like there's there's no backdrop in the brain. You do have asperites and ga and extracellular charge diffusion diffusion in in the brain. So like a lot a lot of the differences between the AI paradigms are not that big. I mean you have evolutionary learning but then in the 80s you had Edelman's neural Darwinism which claimed that the neural assemblies in the brain are evolving by by natural selection. Right? So I mean you could you could make a decent argument that just as sort of physics, computer science and math are converging into one thing. All these different AI paradigms are looking more and more similar as things progress. Wouldn't you expect us to build the first AGI using the simplest methods uh available? And and the we the first AGI we build will be built using the easiest way to to build an AGI. And I I would I would guess that that method is not a combination of methods, but rather something very simple that you can scale. Honestly, that doesn't that doesn't seem to be how software development usually works or math actually. Usually the first way you do something is kind of a cluge and it's just it was easiest for you to do given the materials at hand and given your point of view then. But then you like you know in in mathematics often the first proof of something is a big horrible ugly mess. Then like 50 years later you realize how how simple and elegant it it actually was. We could also argue that modern electric cars are going to be a lot simpler than the internal combustion engines and so forth. So yeah, I think I think invention often happens actually by drailage and cobbling together the that you have available to to make something happen. And the most elegant simple AAM's razor way often comes later. I mean certainly like physics is a mess. The standard model of physics is a mess. Now everyone thinks there's going to be something a lot simpler but the the early quantum mechanics was also way messier than the new quantum mechanics of Heisenberg and and Schroinger and so on right it was it was a mess of stuff adapted from classical physics. So yeah, I don't I don't also scale plays a big role here because some something like Marcus Hooters's AXE or Jurgen Schmiders's girdle girdle machine or or similar ideas I had even even before I read that. I mean there's mathematical arguments you can make a really really really simple AGI if you just had a big enough computer, right? Because in effect these really simple AGI algorithms involve brute force searches over large spaces of programs, right? So like if if before you take each action, you can search all programs up to a certain size and figure out which one if you executed it would lead you to the best action, right? Then you just run that. Then that's that's much easier than doing all the garbage we do in in modern AI systems. The problem is anything approaching a brute force search over program space is just infeasible using current hardware. know will it be infeasible using like a phento computer or autocomputer or something it's I mean you can't truly do complete enumeration over program space but you could do things much you could do things much closer to a brute force search over program space if you had massively more more hardware so it it might be that once we've gotten sufficiently powerful hardware which could end up being a few years post singularity who knows right it Maybe once you're there, you can radically simplify AI algorithms more in the direction of of girdle machines and AXE and whatnot because I think much of the complexity in modern AI is working around resource limitations and the resource limitations themselves are not simple they're particular right like so we have GPU and CPU we have cache RAM we have we have main RAM we have networks of computers with certain bandwidth So it seems like as long as your infrastructure is heterogeneous in its resource limitations, you're going to end up wanting to adapt your AGI system to be somewhat heterogeneous in its operation for efficient operation on that on that infrastructure. So then then that's like simplicity conditional on your infrastructure and your data. But simplicity conditional on your infrastructure and data is not the same as simplicity by by our own intuition like you get axe or or girdle machine, right? So yeah, I I I the the other thing I would say though is simplicity in a deep tech stack is often carefully engineered on top of a lot of complexity. Right? So like it's it's also like it seems simple to us to walk down the street because we're not aware of what's happening in our in our cerebellum and writing a Python script to train a deep neurore model seems really simple until you try to write the the CUDA code running on the NVIDIA GPU to make the matrix multiplication algorithm manage the cache RAM all the different layers of processors in inside the Nvidia GPU. Right? So what I mean we we built things so the type level that most people have to deal with looks simple but it's really like a quite complex stack working around the strengths and weaknesses of of the underlying infrastructure and this is often top of mind because trying to build an approach that isn't just deep neurallets like we're doing in hyperon we have this weighted labeled metagraph thing called the atom space which is the central sort of knowledge structure of our system. But we you got to rebuild the whole tech stack then because because these are not are not rooted in matrix multiplication most of the graph algorithms that that we're dealing with here. And so if you're if you if you're trying to make a different AI paradigm it's not just scripting different algorithm. It's repurposing pieces to build a whole different tech stack down to the down to down to the chip level. Right? So there there's a lot of complexity and this is this is why you have to view AGI as being built by like a whole huge combination of of of of industries, right? Like I mean we will we'll give a touring award to the guy who like tweaks the vac propagation algorithm to converge better on recurrent nests or something and that and and that that's that's all important but obviously if that guy was like sitting on a desert island to make that innovation it's not going to make an AGI like it's it's coming at this huge huge combination of hardware and software in in innovations which are mostly being pursued not because of AGI, but just because they're making they're making somebody money or letting somebody look more powerful than their their opponents, right? Do you think physical embodiment is going to be necessary for AGI and and if so, why? So, I I had a a funny experience with this. I was giving a talk at a non-technical futurist conference and I was talking given about people who were interested in embodiment for AI versus people took a more disembodied approach and this very newagy middle-aged lady came up to me with like a bunch of crystal jewelry and so forth. He's like, "Well, I'm so amazed someone's finally talking about disembodied AI. Like, I've been seeing these AI poltergeists in my house since since forever, right?" And I'm like, "No, that even your polar guys is not actually disembodied. Like, it's embodied in an electromagnetic disturbance that we just don't we just don't fully understand." Right? So, Pay Wang, another longtime AI researcher who was a pioneer in the Chinese AGI scene in the 80s and 90s, he had a paper once called a laptop is a body, right? I guess the the point is your your AI, I mean, it's always seeing something and doing something, right? So it's it's otherwise you as the programmer or tester could not be interacting with it with it either. Right? So the it's a question of a what sensory and motoric bandwidth are needed to get to certain kinds of AGI and b to get humanike AGI as opposed to just arbitrarily intelligent AGI that might be off in a different direction than humans. Right? to get human like AGI, you know, how much do you want to have a humanlike embodiment? That's on on the first question. I think robust embodiment is convenient, but probably not necessary. Like I I would imagine you could get a vastly superhuman AGI with a much restricted sensorium and and the motoric world than than than people have. sort of depends on what you want to do. Like if you started by making a theorem prover and a sort of a scientific research assistant that's doing symbol manipulation, then you can give it limited insight into the physical world. It probably can work fine, right? And there's a lot of camera inputs to the internet that it can use without having its own body to tool around. There's two issues with that sort of approach. One is of course it's harder for us to know what's going on because if a mind you're building is very non-human, you don't have so much intuition to go on in in in in designing and testing it then. But also a very non-human AGI like that for better or worse will probably have less of a strong understanding of what it is to be human and human values and and and culture and all that, right? So I think there's a stronger argument that if we want an AGI we can relate to on a sort of Barian I thou level like relate to on a deep level then then for that AGI to have something vaguely resembling a human embodiment is probably probably quite valuable right I mean for the same reason like I can I can empathize with other men better than with women in in some ways I can empathize with other people better than with apes or or rats right I mean Having having an embodiment like ours doesn't guarantee that it's empathic toward us or understands what we're up against as humans, but it kind of would would give it a head a head start, right? So I think what we can do certainly with a hyper type architecture and in different ways with deep norled architectures I mean you can you can take protoagi systems with different bodies and different levels of attachment to their embodiment. You can have them learn stuff and you can then network them together and even merge their knowledge bases in some ways. Right? So which is something we can't do in in the human sphere all all that well. So I mean you can take a fat system, you can take an protoagi system used to control a humanoid robot, you can take a system controlling biology lab equipment and with some work and some caveats, I mean you can have what's learned by all these systems combined together to to synergize and fuel like a sort of semicoherent overall artificial mind. So I I don't think it has to be has to be either or. I I do think though that the ease of doing things with embodiment is increasing very fast also, right? So I mean so we've I worked for years with David Hansen at Hansen Robotics. We had we made Sophia the first robot citizen. I led the software team behind that. But now I'm still working with David, but we have a different robotics project called Mind Children. And we in the last nine months or so we put together a 3 and 1/2t tall humanoid robot. It can look at you. It can talk. It pick things up. We can it's not walking. It has wheels. It rolls rolls around the room. But that the the ease of making your own robot with the properties that you want for for teaching and evaluing your your your protoagi system. Like it's it's incredibly easier than five years ago, let alone let alone 20 years ago. So it's a seems like what's happening is early stage protoagi stuff is just being tried out in a variety of humanoid robots along with other applications and then the knowledge base is just will get will get munched together somehow so that the field isn't the field isn't requiring itself to to ask the either or question. We're just doing both, which is which is what you get from having more attention and and resources into the field. Generally speaking, do you think the notion of aligning AIs with human values makes sense? And what are the best approaches here? alignment is not a term or language that comes naturally to me. But I mean I think the intention behind it is probably something fairly reasonable. The first thing I would note is people are not very well aligned with themselves let alone with the with with each other. So what's the bar for alignment with humanity needs to be thought through carefully. Like I I have got probably gotten more self-aware as as I've gotten older through meditation and various other practices. But one of the thing one becomes aware of then is how incoherent and non-unified one's own self is right. So like I I mean when I visit subsahar in Africa, I will give a decent pile of money to poor suffering people I see in the street. When I come back home to the Seattle area, I send less less money to those people. and I will go out and buy a piece of weird keyboard equipment to play music instead of sending all disposal income that I have to save kids who are starving in Africa. Yet, if I was in front of those kids and I had the chance like buy food to give this kid right in front of me versus buy a keyboard, I would probably buy food to give that kid right in front of me and not own a keyboard, right? and just learn to play a cheaper instrument. So, I mean, I can see I'm I myself am not entirely morally coherent. I mean, I don't beat myself up too much. I've given a lot of money to initiatives in Africa and spent a lot of time on it, right? But and I don't I don't feel the need to force myself to be entirely coherent either. I mean, I think most humans, probably all humans, we're more like clusters of behavior patterns than like unified, rational, coherent entities if if we really are are honest with ourselves. And that's even more so on the on the collective level, right? Like if I I mean if I go through rural Ethiopia which is a beautiful place where I love to travel but the the average people are heavily Ethiop Ethiopian Orthodox Christian right and the I mean they don't think AGI will ever have a soul even it will be be much smarter than than than us and I mean the the attitude there is rapidly homophobic right where I mean my mom is gay I was I was raised in a to totally like queer friendly ambiance. Now these are lovely people you meet in Ethiopian villages. They're just raised to believe that you know you'll you'll you'll burn in hell if you're gay, right? So I mean if you so if you look at the lack of alignment in each of our own minds if we're honest with ourselves and then the lack of alignment among different human beings you got to ask like what exactly are we thinking the AGI is supposed to align to like is it Elon Musk's value system that doesn't seem very coherent e either right like is it is it like the the weighted average of all Silicon Valley VCs and and software developers weighted by their bank account. I mean, or their IQ like it's it's not really clear what you want to align with. So, I end up thinking about it a little differently than that, but it may capture the spirit of what people are looking at with alignment. One approach is to is to talk about some minimal set of commitments. You would want the AIS aligned to something like you want the AIS to not destroy humanity, so not cause our extinction and you want the AIS to to not be in in in complete control over humanity. So, of course, there are some people that that disagree with those notions, but I think that's something that you would find quite wide agreement on among many humans. I don't think the main issue with that is that some people would disagree with them. I I think the main issue with that has been highlighted in science fiction since before I was born and was summarized very well by Eleazar Yedkowski who I I differ with on a number of things but I've known him since forever and we've agree on a lot of things too. I mean he he made the point that human values are complex right and we summarize them in natural language in ways that we culturally have a common understanding of. So it makes us think it's it's it's simple. But these things are really very very ambiguous and their interpretation as we think of it depends on a bunch of implicit cultural assumptions. And I mean this was highlighted in the science fiction book the humanoids that used to be required reading in in MIT's AI department way back when before AI was was so popular. And in this book by Jack Williamson, which I read probably 75 or something when I was a kid, right? I mean, the people create these human level intelligent humanoids which are more physically powerful than humans and they give them a mandate to serve and protect and guard men from harm, right? And everyone in my generation in the AI field read this book. And of course, they wouldn't let people use power tools. wouldn't let people use hammers. Like in the end, if you were upset about your girlfriend dumping you, they would inject you with some some euphoride because that obviously was causing you harm, right? So they they interpreted serve and protect and guard men from harm in a different way than the authors had had originally intended. Right? And one of the lessons of the failure of rule-based expert system AI where you code all the AI's knowledge by hand. One of the lessons there is like even if you decide to refine serve and protect and guard men from harm into a whole volume of logic expressions, it's still not enough. Like there's always a loophole. There's always room for for interpretation. And of course, this is why our legal system has case law, right? Because I mean I mean we we try to enumerate law in detail but then in the end judges have to use nearest neighbor matching in a very fuzzy and informal way against against a bunch of of cases which causes really annoying annoying problems as we see now in US Supreme Court right but it's it it's for the reason that Eleazar said like human values are complex and and I mean what what we mean by something like don't cause the extinction of humanity seems like it's straightforward, but it's not straightforward. Like if because some people will argue that replacing human cells with genetically engineered cells is the end of humanity. Some will argue that isn't replacing it with like robotic cells is is is the is the end of humanity. Some would argue a brain chip implant is the end of humanity. Some would argue staring at your phone all day is is is is the end of humanity. And you can try to enumerate every case, but you're basically guaranteed that the world will throw at you some case that wasn't in your enumeration of of of cases, right? For one thing, we don't have a formalization of the world and we don't know what new technologies and trends are are going to going to emerge. So the the thing is that enumerating some principles you want the AGI to follow, of course you want to do it of of of course it makes sense. But it's foolish to think that's going to give you anything resembling a guarantee. Like I think those maxims have to be on top of some more implicit resonance of of of the AI with with with human values in it. This is sort of like raising human kids, right? Which I mean I've have five kids and one granddaughter. Some you see like giving your kids some core principles they have to obey and telling them these principles over and over or even rewarding and punishing them for obeying the principles or not like this. This does not work very well, right? I mean, and I mean, if you raise your kids with the right vibe of compassion and values and you carry out activities together with them in which you're collectively pursuing activities in accordance with your values and then on top of that, you tell them some core principles that that sort of reify and abstract what they what they've got what they've gotten implicitly through the shared activity with you like that that can work reasonably well and and I mean this is what education systems have always tried to do right so that but now you might say you don't have to do that with an AI system that's just because people are perverse but I think for any humanlike AGI architecture it's going to be like that because you have this vast teeming massive self-organizing activity that's conditioned based on experience and then the rules and principles that you give it are just guiding this vast mass teaming mass of self-organizing activity and that in the end that will be true if you have a huge logic engine as well as if you have a huge neural net because I mean in any case you've got a massive amount of stuff going on that's not predictable in detail by the programmer and you and you need you need it to be making up its own stuff as it goes along, right? Like otherwise otherwise it's not going to get it's not going to get to to human level of of general intelligence. So yeah, going going back to alignment or things resembling alignment that not be more might be more meaningful or achievable. I I mean I I think you can hope for compassion. You can try to get AGI systems that are empathic and compassionate to people as well as declaratively understanding human values which LLM can can already do. You can hope for AGI systems that are compassionate and have a working practical understanding of human values. You can also think about what I would call meta goals to put into an AGI system. So you you can ask the AGI system to have as a value as a meta goal like don't change your top level goals very fast or heedlessly, right? Like I don't I don't think it's fair or workable to try to build an AGI system that keeps the initial top level goals we gave it in precise form forever. I think if you do that it will just try to work around them in in various crazy ways. Sort of like humanity has like we had a goal to reproduce. Hey we invented birth control and we hacked we hacked around all the mechanisms there. Right. So I think if you try to restrain AGI to rigidly hold the top level goals that the original programmers put in like it won't work. Self-organization will just kind of work work around that and you get a perverse system. I think it will I think it could work better to make an AGI have a top level goal to evolve its top level goals in a sort of moderated and responsible way after interacting with the others in its environment and reflecting on itself carefully. So I mean I think you can design a metag goal system in a way that decreases the odds of the AGI system like weirdly going off in a totally non-human direction. What what you want is for it to have a top level value of evolving its own value system in a way that sort of has a high mutual information with the evolution of of of human value systems. Right? is human value systems are complex, self-contradictory, incoherent, heterogeneous, always changing. They're evolving. The AGI's value system, I think, can be more coherent than human values, but will also be evolving. And you'd like there to be give and take and mutual information between the two the two evolving evolving value systems. And you can you can bake that into the value system of the a of the AGI like yeah you're going to change your values as you do be sure you're closely connecting with with with human values as as as they change in that process and if that probably is a species of of alignment. It's just not what many people are what many people are thinking about when they're talking about alignment because they're seem to be thinking more like there's some core of human values and we can get the core of the AI's values just go alongside whereas my my feeling is it's more like human values are going like that then you want the AI values to follow their own chaotic orbit like sort of coupled a bit with the chaotic orbits of of of of human values and that This perspective just makes it much harder to think about guarantees. And some people seem to want guarantees. And I I don't think we're going to have guarantees. We're going to have very fudgily probably approximately correct value systems rather than guaranteed value systems. When you look at where we are with AI progress right now, we're seeing incredible performance on a number of benchmarks. uh math performance, programming abilities. AIS are passing all kinds of tests, college levels level exams. What does that mean for impact in the real world? Because I I I see the the AI optimists being very impressed by these benchmarks whereas the AI pessimists are asking questions such as you know when will AI show up in productivity statistics or in GDP or in unemployment and yeah what what do you think of that disagreement? So I I I think the the roll out of AI tech into the practical economy is gated by human stupidity, human culture and h human ego and all the con constructs that that that we have governing governing our world. Right? So I think yeah while I'm not an optimist that LLMs can lead to human level AGI because I think they don't have creativity in in a fundamental sense I I am an optimist that LLMs with minor additions and tweaks could take over like 95% of of of human jobs. I I agree with Sam Alvin on that point. I just think that 95% of human jobs I mean as a vague handwavy figure can be done without fundamental creativity or or or or inventiveness. So if you have a system that can do really clever nearest neighbor matching against everything people have done as recorded on the internet like most of what people are doing is a repetition of something that's already been done and recorded on on on the internet. Right. So I I mean I mean I I I I I think that we could roll out deep neural net driven systems to do tremendous variety of human jobs right now. It's not happening that fast just because that's not how society and indust and industry are are are organized, right? And and then that that becomes more a socio-csychological question. I mean a a very simple example, some friends of mine had a startup company called Apprent a number of years ago and among other things they automated the McDonald's drive-thru and it it worked. like I I I use the system. It was rolled out in some McDonald's in somewhere in the Midwest. Now due to some organizational issues within McDonald's that was rolled back. Now they're planning to roll out a new system. Right? So I mean that will happen. It can be done by AI right now. It's not perfect. The people aren't perfect either. But that's just I mean that was could have been rolled out five years ago, right? I mean so that that same story all over the place like even when AI could do the job and could do it cheaper and better than than people the roll out is very slow because society is organized a certain way and there's a lot of momentum I mean law is is another thing like that right like fundamentally right now a great amount of parallegal work and drafting of contracts and so on can be done by LLMs. Lawyers and parallegals are using them in the house to do their work and then charging charging an an hourly rate for an hour for what was actually two minutes of of of going on to chat GPT or or Deepseek or something, right? But the legal profession is in no hurry to restructure to optimize itself around around the use of of large language models and there's all these protections like licenses to practice and so forth. So yeah, I I really think on the one hand Gary Marcus and other LLM pessimists are correct that some people oversell LLM and there are limits to their general intelligence. Totally. On the other hand, I think if everyone was lazy and didn't want to work and we had a political will to just give people free money, I mean, we could reorganize society. So, right now, AI would do a a tremendous majority of of of of jobs. Like, we're just not doing it. I mean, you know, look look at look at the slow roll out of automated convenience stores, right? Like Amazon had these stores where camera would just take take take a picture of of your food when you leave. Like I there's no question in my mind like this technology could work right now, right? Like it's not it's not it's not it's not that it's not that hard. But then of course people are jerks and want to steal stuff from the store. And then being policed by a Robocop has a different social vibe than being policed by by by the by by the human security guard, right? So there's all these social and psychological issues that that slow down adoption. So what what that means is that the bar is pretty high, right? Like the AGI has to be way way way better or way way way cheaper than than people. And when the margin is enough, then the social obstacles to adopt it will be will be overcome. So Caleum Chase, a friend of mine from UK who's written a bunch of books on this sort of thing. He sort of thinks the great obsolescence of human jobs will come in one huge batch because he he sort of figures like at a certain point you'll be close enough to AGI that the cost savings and the efficiency gain quality gain is just too much for people to ignore and everyone everyone will just have to immediately roll it out and then it will happen like in a big in a big wave wave all over the place. And I think that it it might happen that way, but and I think that's more because of the sort of face transition dynamics of the human social networks making up the economy rather than necessarily because of the of of the AI AI capabilities. And yeah, I can as as another example, I can look at that in in music because I'm a musician do a bunch of computer music stuff, right? So I I mean if you trained an LLM or a comparable deep neural net on all the music up to the year 1900 and see nothing after 1900 like that AI will never invent neocclassical metal grind core progressive jazz hip hop right like it's a it's not going to synthesize that for music before 1900 if you ask it to put together West African rhythm with western classical music like it'll a boach fugue to a West African beat or something which could be interesting but it's it's not the same as the deeper fusion that happened to create jazz or something right but on the other hand so there's there's missing creativity like like you're not going to have the next Jimmyi Hendricks or John Cultrain be be a deep neural net of of the current style of of deep neural net on the other hand almost none of the music industry requires that Right. So if you're if if you're looking at say make background music for my video game or my advertisement or or my movie or something or even like generate pop song to play on Spotify for people to play in the elevator like these are solved problems by AI music generation now it's just record labels don't want it the music industry doesn't want it musicians don't want it right so I mean the the roll out there is gated by what the community of humans involved wants rather than by what the what the technology already can demonstraably do. But but but that situation can't last forever, right? That situation will face pressure from from the market. Yeah. Clear clearly. So but but then but the question you ask is when and the point is when when is more about these social dynamics and then regulatory capture by groups that feel that feel threatened, right? is that that can you saw that with the medical profession for a long time like for a long time we had AI that could diagnose disease based on symptoms as well or better than a doctor. We had that from from rulebased AI even before modern neural nets but I mean the medical industry will not allow that to be rolled out. I I saw that in China 10 years ago, like in the waiting room in the hospital in Shanghai, they had a WeChat chat bot where you could just tell your symptoms to the WeChat bot and it would tell you it would tell you what was wrong with you before you went in to see the doctor and the doctor would just double check what the WeChat bot said. Right? So, China rolled that out in a number of hospitals that I saw personally 10 plus years ago. US we still don't have it, right? you just sit you're just sitting in the waiting room get getting sick from the person next to you and and filling out forms and listening to music for four hours right so I mean that and that's that's just because US medical regulatory establishment is much worse than than than China's bas basically so yeah I I think I I think Ray Kerszswwell's prognostication of human level AGI by 2029 that he put out in his book the singularities near 2005 is looking remarkably preient, right? Like we we might beat it by a couple years. We might get there 2026, 27, 28. It might be a few years slower than that. I mean, but but on on the whole on the whole that seems fairly on the mark as as predictions go and I I sort of I'm getting inclined toward Caleum Chase's idea that right around that time of the breakthrough the human level AGI is when the massive job obsolescence will occur because it seems like there's so much psychological and institutional resistance to it that it's AI is just going to take over different industries. ries in a weird erratic pattern just gated by the fact that people don't want their jobs obsoleted and that the people running companies are not that savvy about about AI in most in most verticals. Now I think however Rey was too pessimistic when he said human level AGI 2029 and super intelligence 2045 like I don't I don't think there will be a 16-year gap. I I think there will be a gap of one to three years or something. I mean I mean and probably the thing slowing down the transition will be the AGI's own conservatism about how fast it wants to responsibly self-improve because I feel like once you have a human level AGI it should be able to increase its intelligence by an order of magnitude qualitatively speaking at least just by software improvements because it's going to be a better AGI programmer than we are and then you get into hardware improvements I mean if you have an AGI robotizing factories and and dealing with the hardware. It's not going to take it more than a couple years to make like a radically superior batch of chips and and and and so forth. It shouldn't be able to it shouldn't be able to speed up by some small integer how fast we can roll out new chips. If it's a like a human level AGI built on top of current tech, they can already do do math and engineering better than people in in in some senses. So even I think even if Kalum is right, I mean we're then like what three to six years from the massive massive elimination of of human jobs which is a situation our social and political systems are not especially well prepared for particularly on a on a global level if you look in in the developing world but even even we're not well prepared in the developed world. Do you think the transition from AGI to super intelligence would be slowed down by the the physical world? So just gathering enough materials to create enough chips to train sufficiently large model to to get to higher level int of intelligence. No, no, because I think the whole direction of I think the whole direction of training larger and larger models is sort of intellectually bankrupt. And I mean, and I think LLM already has a lot more data than I do, and it's not as generally intelligent as as as as as I as I am yet. So I mean I think on the one hand yeah you need a lot of computers and you need a lot of data but but I I think you don't need as much data as modern LLMs have to make a human level AGI in in my view like that they already they already know more than than you and I do with within their weak ability to to to know things. So I my gut feel is that once we get an AGI that AGI will be able to increase its efficiency of operation tremendously just by by improving its improving its software right and and then it won't even need to roll out new hardware or get more data to become a super intelligence on the other hand of course it will be able to design new new hardware We're also there's a bit more a bit more time lag to that but just if you if you look at the software stacks that we're using like I mean we have these servers we have Q and Open CL we have Linux then we have like a Rust kernel on top of Rust we have our own AGI language meta we have this stack is utterly not the optimal way to implement the AGI that we're trying to build on top of it right like if the AGI just rebuilt everything from the from from from the ground up without having to go through all these awkward layers that are built for human understanding and are there for historical reasons really. I mean I don't I don't I don't see a big obstacle to massively massively optimizing AGI in into ASI. I think if there's a slowdown factor, it would be the AGI's own value system. Like it not like even if I could rewrite my own brain arbitrarily fast, I probably wouldn't, right? I mean, I might be more reckless than some people, but I I mean, I want to survive. I care about my mental well-being and that of my family and friends, right? summit, you would there there's an argument for sake of safety and common sense like you you make small changes and improvements, see how they pan out in the real world, make other small changes, see how they pan out in the real world, roll back if they if they if they aren't working out. So it probably the transition from AGI to ASI will be gated by responsible self modification on on the part of the AGI. The this is where we hit the big challenge that I see will happen in this transition period though. I think that you have the following sort of Sila versus Charbet issue, a rock in a hard place issue, right? The issue is if you have multiple competing efforts at AGI, for example, an AGI arms race between the US and China such as certain national leaders are currently advocating, right? So if you have that sort of situation, so let's say that multiple parties get a human level AGI around the same time, which is almost guaranteed to happen, right? I mean, even it even if you kept everything locked down, that doesn't work too well. Like someone gets poached by someone else and offered $10 million to share the trade secrets, right? I mean, we can we can see with transformers from Google to OpenAI, then back to Google and to Deep Seek and so on. You can see that you can see that you can keep things locked down a little while but not not not that long. Right? So if you have multiple competing parties building AGIS then in order to have a moderated pace of advanced super intelligence you would need agreement from all the parties controlling the AGIS about moderating the pace of development. And and you then have a really annoying arms race psychology, right? Because clearly I mean from where I stand right now it might look different once we have the AGI but it seems to me now the most likely most responsible thing to do is not go from AGI to super intelligence at the maximum possible speed. Right? probably the most responsible thing to do will be do that by baby steps and use some experimental information gathering a as you go. I mean maybe not. Maybe the AGI will just print out a very compelling argument why upgrading to super intelligence in one fell swoop is is the best thing. Maybe it'll be right. But supposing that some gradual increase from AGI to ASI is the best thing to do, that requires a lot of trust among the competing parties, right? Because U US's AGI isn't isn't going to want to go slowly if it thinks China's AGI is going is going fast. And a decentralized network building AGI isn't going to vote to go slowly if it thinks the centralized network that wants to make it illegal and put its developers in jail is going fast. Right? So th this this is potentially dangerous and and annoying, right? I mean, because we we already have the makings of an AGI arms race and but before we actually have AGI, it's mostly a matter of humans using tools to advance their own their own particular sectarian interests. But if that's transmitted into the AGIS where they're driving their own development and then what exactly happens because each AGI to the extent it either implicitly or explicitly like wants to keep surviving, right? I mean it will see that the other AGI getting to the singularity first is a threat to to its own survival and to the survival of the people it loves who created it, right? Like the the AGI might think like just to save the life of my creators who I've been trained to empathize with, I need to make myself smarter and smarter before this other AGI that's ruled by these guys who literally want to kill my creators. Right? And so this this is sort of what our geopolitical system is is tending toward now, right? I mean, and there's there's a possibility that having the first AGI be a sort of decentralized open global brain can diffuse that dynamic because you'll have a sort of decentralized open thing which is just smarter than any of the sectarian AIs and then more resources just get pitched into the open decentralized global brain and the sectarium ones can't catch up. I mean that that's a possibility but obviously there's a lot lot of different uncertainties there than any anyone could think of immediately. Ben, as a as a last question here, what what what do we do about all of this? You've sketched out a situation where it seems that we that we are racing towards AGI and there might be competition to get there and this competition could be quite dangerous. So what what do we do? Do we have different layers that or different options that we can put together to to get a good outcome here? Seems like the most plausible course to a beneficial outcome I can see is the first AGI is created by some group that wants to make it open and decentralized and doesn't care about controlling it personally and doesn't care about putting their own personal value systems in it as opposed to all the other human values. And then this first AGI has got to rapidly make a plan for secure beneficial AGI development and roll out across the world and then people have to choose to adopt that right and I mean that I don't think is unthinkable. So if you if if you look at Steve Omahru and M Max Max Tegmart from Future of Life Institute that they've written some stuff about trying to make a provably secure infrastructure for for the all all the all the technology in the world, right? And I mean I'm I I I love the idea. I I mean I' I've done research myself on how to make systems provably secure both in quantum computing and in LLMs and and and so forth. I don't I don't think that's terribly viable to roll out in the near term. And I argued this with with Steve who I've known a long time in in an interview interview we did for a couple reasons. I mean, one is it's just a lot more expensive to do things in in a in a in a secure way. So, I mean, I I posted a paper recently on secure transformer architecture that will only slow things down by a factor of two or SL so just to make a transformer that isn't so susceptible to prompt injection attacks and that's just slowing down by a factor of two to protect against one kind of attack vector. It's not pro provably secure. If you look at homamorphic encryption or something, which you need to make AI processes really secure with respect to other other people hacking in and and spying and seeing what they're doing. I mean, right now that slows you down by a factor of several hundred. I did some interesting calculations suggesting you could do homorphic encryption of complex programs with only maybe one order of magnitude slowdown on a quantum computer. So, interestingly, it might be that some of this security is easier when you're into quantum computing just because of the different way quantum computers operate. But, I mean, even so, it's just way more expensive to do things in a provably secure way. Adding on top of it the fact that we don't really know how to do it yet for most of the processes in a in in our in our global tech stack. It's a really interesting research area, but it is a research area, right? So you you would be asking the world to pause all sorts of development and instead put huge amounts of money into research fundamental provability of of safety of different parts of our tech stack at a time when the US is cutting out funding for the NSF for all sorts of basic basic research. Right? So I I mean doesn't seem plausible in that sense. And there's also the arms race dynamic. Whereas if the US chose to slow itself down by making everything provably secure, like if China or Russia didn't, the US is quickly going to decide that's stupid. But suppose your first AGI thinks provable security is important, right? And suppose that first AGI then solves these tech problems faster than humans have been able to and tells you how to make chips and operating systems and LLMs and reasoning systems and whatnot that are provably secure in infrastructure like here's okay here's provably secure embedded Linux you know version one right so if the first AGI is oriented towards benefit and and safety for humans as well as AGIS and figures out how to make the right sort of infrastructure and if the first AGI manages to roll that out, I mean then then then you could have a more beneficial transition. And if if you want to go science fictional, this can be achieved in in various ways, right? You you could say the first AGI release is a botnet that just replaces everyone's OS with a provably secure OS and then bada bing bada bing, you're done, right? I mean, that's how it would work in a in a science fiction story and could happen. You can't can't rule it out. There's also there's also a possible future where this comes out like global nuclear disarmament or like treaties on biological weaponry or something, right? Where once you really have the AGI and it's there in front of you, like it's really clear, whoa, this thing is smarter than people. Then suddenly the people running major world governments are like well okay yes we will adopt we will adopt this safety protocol and then the AGI presumably would roll out technology that allowed monitoring of whether the safety protocol was act was actually being being adopted. So it seems like that's at least a plausible avenue, right? Like the first AGI leads a process of enforcing some reasonable safety for the next stage. But the reason I think that's more plausible after the first AGI is launched is that I'm hoping the first AGI can just make security by design not be obscenely expensive and difficult, which which which it which it is. It is it is right it is right now and it seems like for humans to make it not obscenely expensive and difficult will will take a long time and the AI industry isn't going to pause for it. Right? So so to make this happen you would need the first AGI to be beneficially oriented and have a value system that makes it want to do this sort of thing. You would also need it to be really good at at tech. It seems like the really good at tech part is kind of falling into place, right? I mean, we don't have AGI yet, but already LLMs are remarkably good at doing different sorts of math and physics. I mean, they can't they can't ground their math and physics activity in in an overall context. So, I mean, there they're still they're still missing a lot, but on the whole, the direction is the first AGI will probably be really good at math, engineering, and and and physics. So it seems like the value system part is a part that has to fall into place. I don't think that's hard on a conceptual or engineering standpoint. And I'll have some papers on that that I'll present at the AGI 25 conference which we're having in in Rejec in August. We've had a conference on AGR&D every year since 2006 or so. The I think it may be hard in that the value system that open AI or Chinese government put on the first AGI system may not end up being the right one to make the first AGI properly steward the transition from AGI to to ASI. I mean my my hope is that by developing opencog hyperon as a sort of hybrid multiparadigm AGI system and rolling that out on global decentralized networks with full openness. My hope is we can get a sort of a value system that's determined by sort of informal participatory democracy among various interested parties and we can get the right value system there. But I mean there's I don't see any social guarantees here here either, right? So I mean in in in that sense our species is on a very high risk high reward trajectory by any rational reckoning. I I tend to be very optimistic about how the singularity will come out in my heart sort of based on a personal or spiritual sort of intuition about it. But if I look at the situation analytically confidence interval is very very wide and there's tremendous uncertainty on all sorts of of important points. Ben, thanks for chatting with me. It's been really interesting. Yeah. Yeah. Thanks. Thanks. Thanks for the good questions.