AI Governance with Elizabeth Seger

[Music] hello everybody in this episode I'll be speaking with Elizabeth Seager Elizabeth completed her PhD in philosophy of science at Cambridge in 2022 and is now a researcher at the center for governance of AI in Oxford where she works on AI democratization and open source AI regulation she recently led the production of a large report on the risks and benefits of model sharing which we will talk about in this episode foring to what we're discussing you can check the description of the episode and you can read the transcripts atp. net well Elizabeth welcome to the podcast awesome thanks for having me cool so we're going to be talking about a couple of papers basically about democratizing and open sourcing AI when we're talking about this like what sorts of AI should we be thinking about what sorts of AI do you have like primarily in mind so when talking about open sourcing and model sharing I'm I've been primarily interested in talking about Frontier Foundation models um so understanding Frontier Foundation models as those systems sort of on the The Cutting Edge of development right now when talking more generally about AI democratization I tend to think much more broadly I think a lot of conversation around democratization right now is about Frontier AI systems but it's important to recognize that AI is a very broad category and well so is democratization which is I'm sure something we'll talk about okay cool so in that case let's let's actually just dive into the democratization paper um so this is democratizing AI multiple meanings goals and methods by yourself a vadia Ben garfinkle Divia Sedar and Alan deur so before I start asking questions can you give us just a rundown of basically what is this paper yeah so this this paper on AI democratization was um born out of an observation around how different actors were using the term AI democratization or democratizing AI whether we're talking about Labs developing AI systems or policy makers or even people in the AI governance space and there just seem to be a lot of inconsistency around what this term AI democratization meant so the paper really started out as just an effort to explain what what does it actually mean when someone says democratizing AI I think there's there's overlap here with the discussions and debates around open sourcing I think there's a lot at least for me personally there was frustration when people would say like oh well we open source the model and therefore it's democratized it's like what what what does that even mean so so the goal of this AI democratization paper was really just to outline the different meanings of AI democratization that were in use so not necessarily saying how it should be used but just how is the term being used and we broke it up into four categories so democratization of AI use is just allowing many more people to interact with and use and benefit from the technology then there's the democratization of development which is allowing many more people to contribute to the development processes and help make the systems really sort of cater to many diverse interests and needs democratization of profits which is about basically Distributing the profits that might acre to Labs that are the primary developers and and controllers of the technology um and those pref profits could be massive and then the democratization of AI governance which is just involving more people in decision-making processes about AI about how it's distributed about how it's developed about how it's regulated and about who makes the decisions and these are all different ways that democratization has been has been discussed and then in the paper we go a step further and say okay well if these are the different kinds of democratization what are the stated goals of those kinds of democratization and then what activities help support those goals and I think the main the main idea here was to show that often times when people talk about AI democratization they're specifically talking about open sourcing or model sharing and I think one of the main points we try and make is to say that something like for example open sourcing AI systems is one form of model sharing model sharing is one aspect of democratizing the development of AI systems and democratizing the development of AI systems is one form of AI democratization so if you're going to commit to an AI democratization effort or say that like these are the goals you have in mind there's so much more involved in these processes it's not just about releasing a model but many more proactive efforts that can go into Distributing access to the models ability to use the models and to benefit from the models whether that's through use or through the profits that it produces gotcha so in in the paper you like cite a few examples basically of people like using either talking about something Akin democratization or like specifically using the term democratizing Ai and in all the examples we use it's basic Labs talking about it and it's mostly in a context of like we want to democratize AI in the sense of like everybody should use our product is kind of how I read it I'm wondering like do people other than Labs talk about democratizing AI like is this a thing broadly yeah absolutely so so in the paper I think we focused a lot of the lab terminology because I mean to be honest it was written initially as sort of a push back against the use of the term by labs to basically say use our product it's like okay what does this actually mean there's more to it guys um but no no so the term the term AI democratization or even just discussing AI democratization is much more widely engaged with so one of the co-authors Divia sidarth for example she and saffron Hong run the collective intelligence project and this is a group that focuses largely on on democratizing AI governance processes so getting lots of people involved in decision making about for example AI alignment or just about different kinds of governance decisions and and how that can align with the the needs and interests of really diverse populations so I think there are there are definitely groups that are that are sort of popping up and getting really involved in AI democratization from sort of the democratizing governance standpoint it is a term that's used by labs and not necessarily consistently to just mean use our products some of them mean like when stability AI discussed Ai democratization and the whole AI for the People by the people Mantra that was very much about model sharing and and making models accessible to lots of people to use as they saw fit and so yeah so you see this in Labs you see this with a lot of groups that are popping up to help democratize governance processes probably hear about it less in policy and governance circles though um often times like sometimes I've talked to like union leaders and stuff sometimes and when they think about AI democratization they've spoken about it sort of like how are these Technologies going to help like help our Workforce or both in the sense of like making sure that it's like helps the work Workforce with with their needs and helps make their jobs easier but necess not necessarily in a way that would threaten jobs and sort of like how do we make the integration of the systems into new environments and settings actually work for the people that are using them um and integrating different perspectives in that way so it's definitely something that I think is it's a terminology that's been spread um and part of part of why we wrote thiser is trying to understand all the different ways the terminology being used though it was largely written in response to the way it was being thrown around by labs and being seen in the media right that actually gets to something I was thinking like so in this paper you you list these like four definitions and that sort of inspired me to think like okay can I think of like any other things like I might mean by democratizing Ai and one thing that it sounded like these union leaders were talking about is something like how customizable AI is or how like to what extent can like any person like wield it specifically to do like their own thing rather than just getting like a you know one siiz fits-all deal I'm wondering like I don't know if you have any thoughts about like that at least potential use of the term democratization yeah so I think that that this is something that we've sort of filed under the democratizing use category and so one of the things that we tried to do under each of these categories is to show how there wasn't necessarily just one way to democratize use or one way to democratize development so when we talk about democratizing use one way to do that is just to make a product available like say here use use gp4 there you go it's available easy access but other ways to democratize use are for example to make the systems more easily accessible by maybe having a interface be more intuitive like the models like the API interface have that be more intuitive or to provide services to help it be more easily customizable as you mentioned to to allow people to whether they're fine tuning it to focus on a particular need or maybe there are ways through plugins for example to to integrate it with different Services into different applications or then even to have to provide support services to help people integrate your products into Downstream applications or into different work streams like these are all different things that can contribute towards democratizing the use of a system and I think customization is definitely one of those gotcha so now that we have like a a better sense of just like what we're talking about in terms of democratization a question that you address a little bit in the paper but isn't precisely the Forefront front is just like how important is democratization in this context because like like for most Technologies I don't think a ton about their democratization right like air dehumidifiers or water bottles or something like democratization I don't know maybe I'm like heartless but for me democratization is just like not near the top of the normative priorities for these things so like should should we even care that much about democratizing AI uh so short answer is yes I I think we should care about democratizing AI I mean you make a good point like we don't we don't talk about democratizing air humidifiers or water bottles um but I think you know this is this is a question that comes up when talking about AI a lot is just how is it different from other Technologies and this is a question that's been asked you know you know whether you're talking about how it's shared or or in this context how how it's democratized and specifically with regard to the discussion on democratization making something available for more people to use available for more people to benefit from or contribute to the design processes I think this is particularly important in the case of AI a because if AI promises to be as transformative a technology as we're hoping it to be um this could you know this will massively impact lives of of across the globe from different groups different nationalities different Geographic settings um and so making sure that we can have input from different groups into design processes into understanding what needs are best served that's going to be important to make sure that this technology doesn't just serve a few very well but serves many people and benefits the world as a whole I think this is also particularly important to note with regard to the democ ization of profits AI is already but but may continue to be just like an insanely financially lucrative industry and we could begin to see the acral of profits to Leading Labs on on on huge scales um you know where where we might see a particular company for example have you know be able to measure its its gross income in terms of like total percentages of total World output or something like those that is huge economic profit and and we want a way to make sure that ideally everyone benefits from this it doesn't just pile up in a few companies in a few Geographic locations Silicon Valley um how do we how do we make sure that this is well distributed and um there are some direct methods that can be employed so like in the section of the paper where we talk about democratizing profits some of the things that we discuss are like you know you could have adher adherence to like windfall Clauses this would be something where uh let's say a company has some kind of windfall profit like measured in terms of percentage of gross World output in that case there might be a a commitment to redistribute some of those funds or we might see like Taxation and redistribution schemes but there are also more indirect ways as well that you can think about Distributing profits this would be for example are there ways that we can get more people involved in democratization of development so more people involved in developing products so there's kind of overlap between democratizing profits democratizing development more people can contribute to development of products then this might challenge a natural monopoly that might otherwise form around large Labs more competition more distribution of profits more people in on the game so yeah I think I think what it comes down to is like if this technology can be as big and as transformative as it promises to be having more people involved in the development process and being able to benefit from the value that's produced by these is really important yeah I think that makes sense actually that reminds me of so so as as you mentioned there's some link between democratizing I guess the development of AI and the profits in the sense that like if more people make AI products that means more people can profit from them presumably um there's also to my mind there's sort of a Continuum between like the use of AI and the development of AI right like you know you can use a thing like in some sense when you use a thing for task you're sort of developing a way of using it for that task right there there's sort of a fuzzy line there and there's also some sort of link between developing Ai and governing AI right just in the sense that like if I develop some AI product I'm now de facto the main person governing how is made and how which is sort of what we're seeing right now like a lot of the AI governance discussion is around the fact that you know major AI labs are making huge impactful decisions about the future of AI like they're making govern decisions and so there's like a big open question of who who should actually be making these questions should it just be a few unelected Tech leaders of major companies or should this be distributed more so yeah there's there's definitely overlap between the categories yeah so I take the point of your paper to be saying like oh these are like very distinct like these are conceptually pretty distinct and like you know you can promote one sort of democratization without promoting other sorts of democratization but to the degree that like they do seem to have these links and overlaps like is it maybe a of just you know Rising tide lifts all boats and like you know bu one kind of democrati get the others free I think I think kind of maybe sort of not not really okay all all of the above um no so there definitely are is overlap between between some of the categories and I think it's either in this paper or maybe in the blog post version I think I think we say like these are not perfectly distinct categories there's going to be overlap between them I think part of the utility of of breaking it up into the categories is to point out that there are lots of different meanings there are lots of ways to achieve these different meanings and goals it's not just about like just open- sourcing a model or just making it available for people to use there's a lot more to it and so I think in some respects there is sort of a rising tide lifts all boats kind of thing um like you said you might get more people involved in development processes and if more people are developing AI systems more people profit from it that can help distribute profits but on the other hand you can also see direct conflicts between categories sometimes so for example you might and think this is something we write about in the paper is talking about democratizing governance so decisions about AI whether that's about AI how AI is developed or even how decisions are made about how models are shared these days there's there's a really heated debate around open- sourcing Frontier AI models and um let's say you you de democratize this decision-making process around whether or not certain models should be released open source and let's say the outcome of this democratized decision- Mak process process is to say okay maybe maybe some models that that pose substantially high risks should should not be released open source let this is a hypothetical but let's say that's the outcome of a of a deliberative democratic process and decision making that could be in direct conflict with the interests of democratizing development where democratizing development that is that is always furthered by providing better access to models so you could have a case in which you've democratized a a governance decision decision but the outcome of that governance decision is to impede some other form of democratization um I mean not just with development I mean I guess you could have a democratic process that says I don't know like large companies should not be taxed and have profits redistributed I mean that would be in direct conflict with an effort to democratize profits I like so there there could be conflicts um I think it's usually between the governance category and and the other Cate ories I mean I guess governance often will conflict with um with specific goals that's kind of the the purpose is to to temper decision- making and make it more I guess yeah you get more people involved in the decision- making processes um and make sure that those decisions are democratically legitimate and reflect the needs and interests of wider population of impacted people and stakeholders okay yeah so this actually brings up a question I had about the interaction between like democratizing use development and democratizing governance so like in the paper yeah as you mentioned I guess the main interaction you draw out is this tension right between like democratizing use like it's you know potentially not necessarily what a democratic process would produce on the governance side but so when I think about like since I don't know about a year ago we've had like chat gbt which in some sense like really democratized the I guess the use of large language models in the sense that it's like relatively cheap to I know maybe free to use these things and so so I don't know one effect is that like you know now it is impossible to at least have banned chat GPT a year ago or something but to my mind a big effect of this is something like increasing people's knowledge of like where AI is at what kinds of things that language models can do and essentially like I don't know if this is a term but like you can think of people as having like governance demands right like you know if I know more about a technology I might say like oh this is like totally fine or oh we need to like you know do this thing or that thing about it and if people like know more about technology they can have you know they can basically have a better sense of what they want their governance demands to be and so in that sense it seems like there's this positive at least some degree there can be a positive interaction between democratizing use and democratizing governance yeah no I think I think absolutely I mean this is sort of a classic pillar of democracy is the having the informed public right you need to you need to have the information to be able to do the good decision making about the object of your decision- making so yeah so I think I think like release of chat GPT for example really put put AI on the public stage suddenly I have my my like my mom and dad texting me asking me questions about Ai and that never would have happened a year ago and uh yeah so I think it really it put the technology on the stage it's got more people involved in it I think more people have a general sense of at least what current capabilities are able to do and of the limitations and that's all been very valuable for sort of enabling more democratic decision making processes about Ai and so there's there's definitely that that link between you know you democratize use you democratize development people understand the technology better they'll be able to make more well-informed decisions or form more well-informed opinions about about about how the technology should be regulated so I think no I think that that's not necessarily intention I guess the attention tends to go the other directions like if a governance decision perhaps would say like oh this this particular technology is too dangerous or or releasing this particular model is just you know that that surpasses an acceptable risk threshold I guess if you took a strict definition of democratizing development as meaning like always spreading the technology further always getting more people involved in a development process always making a system accessible then yeah a decision to restrict access would be in direct conflict with democratizing development um but I think this is another thing we try and say in the paper is like trying to get away from this idea that AI democratization is inherently good I think this is a problem with with actually kind of like a problem with the term democratization is especially in like Western Democratic societies often times democracy is used as a standin for all things good you know so you get company saying oh we're democratizing AI like it almost doesn't even matter what that means they've you know they've democracy washed their products now it's a it looks like a good thing and so um I think that's one that was one thing we were trying to get across in the paper is yeah AI democratization is not necessarily a good thing you know spreading a technology for more people to contribute to development process or for more people to use is only good if that's actually going to benefit people but if it poses significant risk or is going to put a lot of people In Harm's Way then that's not inherently positive so I guess one thing to say is just you know if all that's meant when people say AI democratization or democratize AI is just make the system more accessible then just say make the system more accessible because I think that's that's very clear what it means and it's not inherently good or bad I mean I don't know if we're going to successfully change everyone's terminology in the course of one podcast but um well we can try we can we can try but I think that that's maybe one key takeaway from the paper as well um if people say they're they're engaging in an effort to democratize AI that does not necessarily mean it's going to be a net beneficial effort gotcha so I guess I'd like to talk a bit about the individual senses in in particular I I guess I want to start off with um when you talk about democratizing profits so one thing I noticed is in the I think there's one or two Who quotes use this section and I think the quotes actually say something like like we want to make sure that not all the value created by AI um stays in one place or you know that the benefits of AI don't just occur don't just occre to like a small group of people and and like in in those quotes they don't actually use the term profit right like and in some ways you can imagine that like the the these could go different ways right for instance take GitHub co-pilot right it costs I guess $100 a year suppose that like everyone in the world just like pay $100 a year and it all goes to GitHub but like tons of people use GitHub co-pilot to do incredibly useful things and you know they like get a bunch of value and benefits out of it that that may or may not be um you know probably most of the benefits will be Financial but you know not not all of them will be so that seems to me like it would be a case of you know spreading the value or benefits of AI but it's not necessarily face of spreading the profit from AI yeah so no I mean it was it was so yeah the profits category is kind of a it was an odd category actually in the process of writing this paper and unlike most papers we actually wrote the blog post first and then wrote the paper as supposed to go in the other direction and in the blog post which is on G's website it actually breaks into four categories democratization of use democratization of development democratization of benefits and democratization of governance and for the version in the paper we chose the profits terminology instead of the benefits terminology because benefits was in a way like too it was too broad um because you could I think there was just I mean you've already pointed this out like there's a lot of overlap between these categories and like one reason to democratize development for example is to make sure that more people can benefit from the technology that it serves their need a reason to democratize use is to ensure that more people can access and use and and even use the systems to produce value and profit if you can integrate it with with your own businesses and with your own applications um you can develop value on it from from your side so I think we chose this democratization of profits terminology just to point out that there is a discussion around just like the massive acral of profits to large tech companies I think yeah you you do point out like some of the quotes that we use talk a little bit more broadly about acrel of value I think sometimes it's hard to measure profits just in terms of like the literal money that drops into someone's bank account okay so yeah so I think it's yeah it's hard to find exact quotes from from companies I know that there's been there's been a lot of discussion not by companies but actually this is one thing that you will hear policy makers talk about more this was part of when when Universal basic income was something that was being discussed a lot it it had its moment when people were talking about it a lot more a lot of the discussion around Universal basic income was around what was around job loss caused by automation but also around the acral of profits to to tech companies and to developers of these new technologies and okay if all the profits are going to shift from the workforce to the developers of these Technologies how are we going to redistribute it out so that's actually one context in which like democratization of profits was directly linked to the discussion around AI development yeah I think maybe some of the quotes we used in the paper do do more L gesture to just value yeah but yeah but we separated profits out just because the way we had it in the blog post was just a bit it was a bit too General too there was too much overlap we were kind of double dipping you know yeah yeah so um I think it was just to illustrate that that there is this discussion specifically about profits but I don't think I mean it's definitely not the most commonly used if if you heard someone talking about AI democratization just walking down the street you wouldn't immediately think oh they must be talking about profit redistribution like that's not where your brain goes it's just but it is it is one aspect worth noting I think is the main takeway fair enough yeah that the next one I want to talk a bit more about is um I guess democratizing AI governance so I guess related to the thing you just said it seems to me that if somebody talks about democratizing AI I think it's pretty unlikely that they like if they just say democratizing AI they mean democratizing governance I'm wondering if that matches with your experience so yes and no okay I think often times when you see in Media or you're listening to tech companies and and the the dialogue that that they're having yeah very rarely will it talk about democratizing governance I think we are starting to see a shift though for example with like open AI had this AI democratization grant program I can't I can't remember exactly what it's called right now but basically they putting out grants to research groups to study methods for basically trying to bring in input from more diverse communities to try and better understand what values systems should align to and so in that sense you're kind of talking about AI democratization in this more participatory deliberative process of bringing people into decision-making processes to Define principles or Define values for for AI alignment so open AI had that grant program I think anthropic just put out a blog post where they they had worked closely with the collective intelligence project that I mentioned earlier uh doing a similar thing for developing um their constitutional AI defining what um the constitutional principles are that AI system should be aligned to and that was more of a democratic governance process so I think we actually I think we're starting to see this terminology of ai democratization democratizing ai seep into the technical landscape a bit more and I think that's a that's a very positive thing I think it's it's kind of showing that like they're starting to to tap into and really embody the Democratic principles of democratizing AI as opposed to just meaning distribution and access it actually means reflecting and representing the the interests and values of stakeholders and impacted populations so I think we are starting to see that shift but I think I think you're probably right in that mostly again like in the walking down the street you hear someone talking about AI democratization probably just about distribution of a technology making it more accessible that's sort of the classic terminology um but I do think that in sort of the AI governance space and and even in the terminology being used by Labs uh we're starting to see the governance meaning seep in and that's exciting gotcha so getting to the question about whether this is good I think in the paper you say something like you know the notion of democratization of use benefits uh use development and profit I I think you say something like those ones aren't inherently good and they get their normative force from democratization basically in terms of the governance method so I guess first of all like that wasn't actually so obvious to me so I think like I I think sometimes democratization I I think it often carries this implicit sense of like being more related to egalitarian ISM than like democracy as a political decision making method yeah so and you know I think like you can have egalitarianism without Democratic decision- making and you can have Democratic decision- making without egalitarianism right like people can vote away minorities rights or whatever so yeah I was wondering like yeah what why do you say or okay I I guess what I actually mean is please defend your claim that uh that that's the source of like the goodness of democracy or of democratization as a word yeah so um okay so a couple things three things maybe two things we'll see how it comes out all right so to start out with I do stand by the idea that like the first three forms of democratization so development use profits not necessarily inherently good not not inherently bad just these are things that happen and if we stick by their definitions of like it just means making something more accessible whether that's access for use or development or making the profits more accessible access is not inherently good or bad again if we use the like access for development example y if you have a technology that's going to be potentially really dangerous you don't necessarily want everyone to have access to that and a democratic decision- making process might result in the conclusion that indeed not everyone should have access to it so that's the first part it's like standing by that first half the claim sure the second part around the thing that gives it the moral force or value is the Democratic decision- making process that was involved I don't know if that's exactly what we're going for in the paper I think it's there are lots of different methods that you can use to help reflect and and serve the interests of wider populations and I think you know for some technologies like let's go back to your water bottle example we don't need a a panel of people from around the globe to tell us how to distribute water bottles like it's a you know this this is a decision that that probably water bottle distributors can be like let's just sell our water bottles to as many people as possible and yeah probably fine I think it comes to like where the impact's going to be how big the impact's going to be um are there areas where it like the negative impacts of a technology might be more disproportionately felt and in those cases being able to bring those voices in to inform decision- making processes that might affect them disproportionately is really important yeah and I know we examp in theer of like different kinds ofoc processes that can be used to decision whether these are participatory processes or more deliberative de Democratic processes so like in the open source paper we just released we talk about some of these processes as ways of making more democratic decisions around AI but then also point to like one way to democratize governance decisions is to support regulation that is put forth by Democratic governance and that those governments hopefully if structured well reflect and and serve the interests of the constituents and hopefully some of those governance processes are actually formed by having maybe some sort of deliberative process that takes into account stakeholder interests and and the interests of impacted populations but it's not like the only way to govern AI is to say like okay let's have a participatory panel where we where we we take in all this insight and then use it to inform every every decision because I think you're right like you want that would just be completely impractical to have like every decision that a lab makes be informed by a participatory panel or some sort of deliberative democratic process so I think I you kind of have to range it depending on the potential impact of the question so I don't know if that I don't know if that answers your question completely but I guess it's like you know stand by the first point that like democratization not inherently good yeah the extent of the goodness I guess reflects how well the decisions reflect and serve the interests of the people that are going to be impacted and then there are lots of different ways of figuring out how to do that okay that makes sense so I guess the second thing I want to ask about that sentence is like when you say that um the democratization of use development of benefits is not inherently good like it kind of makes it sound like you think that democratization of governance is inherently good so I found this paper uh that you actually site it's called against democra in AI by uh him Reich him written in 2023 published in 2023 so he basically makes a few critiques uh firstly like he just says that AI isn't the sort of thing that needs to be Democratic like AI in itself doesn't affect a bunch of people there are just organizations that use Ai and they affect a bunch of people and you know maybe they should be democratized but AI itself doesn't have to be he says that like the the relevant institutions like the the relevant things are already Democratic democracy is Prett resource intensive people have to of you have maybe have to a of VES and things and he also has complaints that like democracy is imperfect and doesn't necessarily fix oppression so there are a few complaints about like democratizing AI floating around so and iess I guess in this case he's talking about democratizing AI governance yeah is it the case that democratizing AI governance is inherently good I don't think so again I think it comes back to this uh the ranging it question what are we talking like first of all like the resource intensive Point absolutely it can be incredibly resource intensive to do like an in-depth participatory governance process this is why you don't do it for every single question that's out there like yeah so so that's like really good point there's also the point of like you know you need to be well informed to make good decisions so maybe we don't want the general public making decisions about like all all questions around AI like I mean experts are having a hard enough time for eles trying to decide what constitutes like a highrisk system like we don't we're not even good at benchmarking this among the community of AI experts how are you going to sort of tap into more like general public Notions of what a high-risk system is or like what proper benchmarking should look like you know so there's some questions where it just doesn't make sense to democratize it so I think you have to think about like what questions are there is it like realistic to have more democratized discussions around so like maybe something like thinking about what acceptable risk thresholds are or like what kind of values to align to and I think it's also important to remember there are lots of different kinds of democratic processes when we talk about Democratic processes it's not necessarily what you might think of is like we all go to the polls and cast our ballot on you know do we want a or do we want B you know these could be more deliberative processes where you just get you know some stakeholders in a room they have discussion they try and figure out where they're what are the main issues these discussions so like in like there are there are processes in place to hold like deliberative Democratic processes that are informed like you have experts come in and try and inform the discussion so this would be a more informed deliberative process one way is you know sometimes when it comes to I don't know like with image generation systems there was a whole bunch of discussion around bias and images and you know part of this is just getting people this is again we see overlap between governance and and and other kinds of categories you know if you get more people involved using the systems and they start understanding where the biases are that is a form of Education having more familiarity and then if you can raise these complaints you know whether that's you know directly to the companies or maybe through some sort of like political intermediary I remember there being a what was it a statement released by Anna ESU which I who I think was the congressional representative in I want to say South San Jose okay that talked in quite a bit of depth about um was right after the release of stable diffusion and talking about how there was like a lot of racial bias and some of the images that were that were being produced um specifically against Asian women and like so this is something that was raised up like through a democratic process was brought to the attention of the Congressional office and then the statement was released and then you know that had KnockOn effects on like future governance decisions and decision- making by labs to to do things like put safety filters so I think I think that's one thing to keep in mind is like you can there's there's a wide range of things that you can think about in terms of like what it means to democratize a governance project process okay so like governance is just decision- making and then how do you make sure those decision- making processes reflect the interests of the people who are going to be impacted it's not just about direct participation although there is a lot of work being done to bring more direct participatory methods into even more the more small scale decision- making to make it more realistic like this complaint about it being so resource intensive that's true we also have Technologies available to help make it less resource intensive and how can we how can we tap into those to actually help like a public input to these decisions in a well-informed way yeah so it's lots of interesting work going on I don't think it's inherent like governance isn't Democratic governance isn't also isn't inherently good when it comes to AI governance I think again it depends it depends on the question if the cost of doing a full-blown Democratic process is going to far outstrip the benefit of do doing a full-blown Democratic process that'd be a situation in which is probably not a net benefit to do a democratic decision- making process yeah so that c to the to him's Resource intensive complaint he also has these complaints that like basically AI itself is not the kind of thing that needs to be democratically governed so he says something like I'll see if I can remember but like the kind of thing that needs to be democratically governed is like maybe something that like just impinges on people's lives whether they like it or not or something that's like just inherent to large scale cooperation or something like that and he basically makes this claim that like look AI doesn't do those things like AI is housed in institutions that do those things right so if you're worried about like I don't know maybe the police using facial recognition or something like you should worry about Democratic accountability of police not facial recognition is the point I take him to be making so so first he says like look we don't need to democratize AI we need democratize things that potentially use Ai and secondly it seems like he's saying like the relevant things that would be using AI they're like already Democratic so like we shouldn't sort of have conflicting like I think I think he talks about Democratic overlap or something where like there I think he really doesn't like the idea of these like different you know Democratic institutions like trying to yeah affect the same decision and maybe they conflict or something yeah so I think I don't disag agree so I think I think this this idea like the if I understood correctly this idea of like you know you don't need to democratize the AI you need to democratize the institution that's using it I think that's a completely valid point like these are not like AI itself again it's not it's not inherently bad or good it's a dual use technology it depends what you're going to do with the thing but I would say that you know trying to decide for example how law enforcement should use AI systems that is an AI governance question this is this is a question about how AI is being used about how AI is being regulated in this context not in terms of how it's being developed but in context of how it's being used what are the appropriate use cases of AI this is an AI governance question and it's not necessarily that like these are regulations that are placed on AI specifically a lot of AI regulation really it's just like already exists within different institutions it's just figuring out how to have that regulation apply to AI systems you know these this might be cases like if you have requirements or like certain health and safety standards that have to be met by by by certain technology in a work setting AI shouldn't be held to different standards it's just a matter of figuring out how to measure and make sure that those standards are met by the AI system so like these are yeah so I guess I I based on what you said I I want to say that I I completely agree okay but I would say that it still falls under the umbrella of democratizing AI governance I think that what might be happening here is just like another conflict over like what does the terminology mean where it's like we don't need to democratize AI as in we don't need to get more people directly involved in the development and decision- making about individual decisions around AI development in which case like yes I agree but but we might be talking about democratizing AI governance and different this is why these discussions get really complicated because we just end up talking past each other because we use the same word to mean five different things yeah yeah it can be rough so before we close up our discussion on democratizing AI I'm wondering I don't know I guess you thought about democratizing AI for a while do you have any favorite interventions to make AI more democratic or to make AI less Democratic in any of the relevant senses you I'll be completely honest this is sort of where my expertise drops off in terms of like the really yeah the more nitty-gritty of like what processor are available and exactly when are certain processes the best ones to be applied for this I would really look more closely at like the work that collective intelligence projects doing or my co-author of Evia who's on the paper he's really involved with these discussions and works uh with various labs to to help Implement different Democratic processes yeah I'm just going to say this is where my expertise drops off and I would I would Point towards my colleagues for more in-depth discussion on that work all right the next thing I want to talk about is basically open sourcing AI so uh there's this report open sourcing highly capable Foundation models evaluation of risks benefits and alternative methods for pursuing open source objectives it's by yourself and a bunch of other co-authors but mostly As far as I could see at the center for the governance of AI and again could you could you perhaps give an overview of what's going on in this report okay yeah so this this report's um a little harder to give an overview on because unlike the democratization paper which is like eight pages this one's about 60 Pages y but um yeah I say so quick overview first thing is that we wanted to point out that there's so there's there's a debate right now around whether or not large Foundation models especially Frontier Foundation models should be open sourced this is debate that has kicked off I mean starting with stability ai's release of stable diffusion and then there was the Llama 2 release and then the weights were leaked and now meta is kind of getting behind this open source thing there were the protests outside of meta about whether or not system should be open source and then we're also seeing a lot of activity with the development of the EU AI act and as part of the EU AI act there's um some some parties are are pushing for an exemption of regulation for groups that produce uh that develop open source models so a lot of discussion around open source um and I think the goal of this paper was to cut through what is becoming a quickly polarized debate we're finding that you have people that are you know see the benefits of open sourcing and are just like very very hardcore pro-op source and then and won't really hear any of the arguments around the potential dangers and then you have people who are very anti-open Source only want to talk about the dangers and sort of refuse to see the benefits and it just makes it really hard to have any kind of like coherent dialogue around this question yet at the same time country governments are trying to make very real policy around model sharing so how can we how can we cut through this polarized debate to just try and say Here's here's the lay of the land so what we're trying to do in this paper is really provide a a well balanced well researched overview of both the risks and benefits of op sourcing highly capable models by highly capable models we basically what we're talking about is Frontier AI systems we just don't use the term Frontier because the frontier moves but we want to be clear that there will be some systems that might be highly capable enough and pose risks that are high enough that even if they aren't on the frontier we should still be careful about open sourcing them so basically we're talking about Frontier models but Frontier keeps going so we want to be so we can get into the terminology debate a little bit more later but basically we wanted to outline what are the risks and benefits of open sourcing these increasingly highly capable models but then not just to have like yet another piece that says well here are the risks and here are the benefits but then kind of cut through by saying maybe there are also alternative ways that we can try to achieve some of the same benefits of open sourcing but at less risk so the idea here is actually quite Sim Illy to the democratization paper is to not just say like what is open sourcing and is it good or bad but to say what are the like what are the specific goals of open sourcing why do people say that open sourcing is good that it's something we should be doing that should be defended that should be preserved so why is it we want to be open source to open source AI models and then if we can be specific about why we want to open source are there other methods that we can applo to try to achieve some of these same goals at less risk or and these might be other model sharing methods like releasing a model behind API or doing a staged release or it might be other more proactive measures like let's say one one benefit of open sourcing is that it can help uh accelerate research progress both in in developing more greater AI capabilities but also promoting AI Safety Research like another way you can promote AI Safety Research is to dedicate a certain percentage of profits towards AI Safety Research or to have research collaborations and you can do these things without necessarily having to provide access for anybody to have access to the model so this is what we were trying to do with this paper is really just say like okay we're going to give first of all just a very well researched overview of both the risks and benefits in fact majority of the paper actually focuses on what are the benefits and trying to break down the benefits of open sourcing and then really doing a deep dive into alternative methods or or other things that can be done to help pursue these benefits where the risks might just be too high and I think that the the main claim that we do make in the paper with regard to like is open sourcing good or bad first of all it's a false dichotomy but um I think our our our main statement is like open sourcing is overwhelmingly good open sourcing is like open source development of software open source software underpins all of the technology we're using today this like it's it's huge it's it's hugely important it's been great for technological development for the development of safe technology that reflects lots of different user needs and interests great stuff the issue is that there might be some cuttingedge highly capable AI systems that pose risks of malicious use or the proliferation of dangerous capabilities or even proliferation of harms and vulnerabilities to Downstream applications and where these risks are extreme enough we need to take a step back and say maybe we should have some processes in place to decide whether or not open sourcing these systems is okay so I think the main thrust of the report is open open sourcing is overarching good there just may be cases in which it's not the best decision and we need to be careful in those cases so yeah I think that's that's like the main overview of the paper gotcha and yeah in case listeners were put off by the 60 something Pages there I I want listeners to know there is an executive summary pretty two-page executive summary don't worry yeah I I read the paper and I I vouch for the executive summary being a fair Fair summary of it so don't don't don't be intimidated listeners I've also been putting off writing the blog post so there will be a blog post version at some point okay I promise yeah so it's it's possible that uh that that might be done by uh the time we release this in which case we'll link the or you know maybe maybe by Christmas yeah yeah hope hopefully you'll have a Christmas gift listeners so when we're talking about so we're talking about these like highly capable found found models and basically about the the risks of open sourcing them as as I guess we've already gotten into like what is a highly capable Foundation model is like a little bit unclear maybe it be easier to say like what kinds of risks are we talking about because I think like like once we know the kinds of risks a we know you know what sorts of things we should be willing to do to avert those risks or just what we should be thinking of and B we can just say okay we're just worried about AIS that could potentially cause those risks so yeah what of risks so I think yeah in fact yeah I think framing in terms of risks is really helpful so generally here we're thinking about risks of significant significant harm significant societal or or even physical harm or even economic harm but and of course now you say oh Define significant I but basically just more catastrophic extreme significant societal risks harm so we use some examples in the paper like looking at potential for malicious use there are some more diffuse harms like if you're thinking about political influence operations so this kind of falls into the misinformation disinformation discussion how might AI systems be used to basically influence political campaigns so disinformation undermine trust in political leaders um and then in turn like if you can if you can disrupt information ecosystems and disintegrate the processes by which we exchange information or or even just disintegrate trust in key information sources enough that can also impact our ability as a society to respect to uh respond to things like like crises so I mean like kind of like pandemic is a good example of like if if it's if it's really hard to get people accurate information about for example whether or not mask wearing is effective you're not going to have really good coordinated decision- making around mask wearing yep so so I think this is one where it's just like it's a little bit more diffus it's it's harder to it's it's harder to measure so I think this is one point where it's it's quite difficult to sort of identify when the harm is happening because it's because it's so diffuse but I think like this is one potential significant harm i' I'd say maybe thinking about like maybe disrupting major political elections or something like that there's some options for malicious use that are talked about a little more frequently I think because they're more well defined and sort of easier to wrap your head around these are things like using generative AI to produce biological weapons or toxins even production of malware uh to mount cyber attacks against key critical infrastructure like imagine taking down an electrical Grid or imagine like on Election Day taking down an electoral system these could have like significant societal impact in terms of like yeah harm to society or or physical harm and I think one key thing to point out here is that we aren't necessarily seeing these capabilities already person some people may disagree but I think my opinion is that Technologies with the potential for this kind of harm do not currently exist I think it is wise to assume that they will come into existence in the not too distant future largely because we're seeing indications of these kinds of capabilities developing like we've already seen how even narrow AI systems that are used in drug Discovery you know if you if you flip that parameter that's supposed to optimize for non toxicity to optimize for toxicity now you have now you have a toxin generator and so it's like it doesn't take a huge stretch of the imagination to see how more capable systems could be used to cause quite significant societal harm so no I don't think there's currently systems that do this I think we need to be prepared for a future in which these systems do exist so it's worth thinking about the wisdom of releasing these systems now even if we might not be present with the technology at this moment okay so as a follow-up question so you're you're currently on the AI exis research podcast I'm wondering is so it seems like you're mostly thinking of things less severe than all humans dying or just permanently stunting the human trajectory I mean I wouldn't say necessarily like I think this is definitely possibility I think I think part of what's happening is like you kind of have to range how you talk about these things like I am I am concerned about X risk from Ai and I think you know we could have systems where whether it's the Cyber capability or the biological Terror capability or even the taking down political systems capability and authoritarian like bolstering authoritarian governments like these are these are things that could cause existential risk and I am personally worried about this okay but you know trying to write papers that are digestible to a much wider population who might not be totally bought in to the x- risk arguments I think there are still X risk aside there are still very genuine Arguments for not necessarily releasing a model because of the harm that it can cause even if you don't think that those are existential risks like even if it's just catastrophic harm probably still not a great idea okay so at the very least there's like there's a range going on there yeah yeah so one question I had in my mind especially when thinking about these we're thinking about AI that can cause these kinds of risks and potentially open sourcing it and part of me is thinking it seems like a lot of a lot of basically the risks of open sourcing seem to be well there's now more people who can use this AI system to do a bunch of harm and you know the harm is just so great that that's really scary and dangerous so one thing I was wondering is like if like if it's so dangerous for many people to have this AI technology is it not also dangerous for anybody at all to have this AI technology right like like it how big is this Zone where like it makes sense to make the AI but not to open source it or does the Zone even exist yeah no so me you make a very genuine point and there are those that would argue that we should just hit the big old red stop button right now you know I I think like that is that is an argument some people make I guess where we're coming from in with respect to this paper is trying to be realistic about how AI development is probably going to happen and try to inform governance decisions around what we should do as AI development continues I think there are differing opinions on this probably even among the authors on on the paper I mean there's what 20 25 20 26 authors on the paper the paper has a disclaimer listeners that uh not all authors necessarily endorse every claim in the paper that that includes me okay but it's like yeah so we're very very broad group of authors but um I think oh I was trying to forgetting what sorry what was your original question it just flew out of my head like if it's too dangerous to open source is it also too dangerous to make yeah um no so I think there's definitely an argument for this um I guess where I'm coming from is trying to be realistic about where this development process is going and how to inform governments on what to do as AI development continues I mean it may very well be the case that we get to a point where everyone's convinced that we just you know have a big coordinated pause SL stop in AI development but assuming that that's not possible or improbable shall we say I think it's still wise to have a contingency plan how how can we guide AI development to be safe and largely beneficial and reduce the potential for risk I think you know there's also there's also an argument to be made that increasingly capable systems while they pose increasingly severe risks also could provide increasingly great benefits and and potential uh economic potential benefits for helping to to solve other challenges and crises that we face yeah so I think I think it's I think you'd be hard pressed to get a giant coordinated pause so the next the next plan is how do we how do we make AI development happen safely and it is a lot easier to keep people safe if the super dangerous ones are not spread widely um I guess that's that's the very simplistic view um yeah so I think at least for for me I think it just comes from a realism about are we likely to pause AI development before it gets super scary probably not just given how Humanity works and like we developed nuclear weapons probably shouldn't have done that oops well now they exist let's deal with it so I think having a similar plan in line for what do we do with increasingly capable AI systems is important especially given that it might not be that far away like I said we're sort of like with with each new generation of AI That's that's released we all kind of hold our breaths and say oh well what's that what's that one going to do and then we learn from it and um you know we don't we don't know what the next generation of AI systems going to bring and so having systems in place to scan for potential harms potential dangerous capabilities to inform decisions about whether or not these systems should be released and if and how they should be shared that's really important you know we we might not be able to coordinate a pause before the big scary happens and um yeah so I think it's important to discuss this regardless gotcha that's the technical term by the way the big scary makes sense so yeah speaking of the risks of open sourcing AI so in the paper you kind of talk about the offense defense balance yes and basically you say that like Bad actors they can disable misuse safeguards they can introduce new cap dangerous capabilities by fine tuning like open sourcing makes this easier it increases attacker knowledge and in terms of like just the AI technology you basically make a claim that it's tilted towards offense and that attackers can like do they get more knowledge they can disable safeguards they can introduce new dangerous capabilities it's easier to find these problems than it is to fix them and once you fix them it's hard to make sure that everyone has the fixes is that like a fair would you say that's a fair summary of like yeah i' I'd say that's that's pretty fair I think the one thing I'd add yeah I'd say that you know in with software development for example so like offense defense balance is something that's often discussed in terms of open sourcing and and scientific publication especially anytime you have dual use technology or scientific insights that could be used to cause harm you kind of have to address this offense defense balance is the information that's going to be released going to help the Bad actors do the bad things more or less than it's going to help the good actors do the good things SL prevent the Bad actors from doing the bad things and I think with software development it's often in favor of Defense in finding holes and and fixing bugs and rolling out the fixes and making the technology better safer more robust and these are genuine arguments in favor of like why open sourcing AI systems is valuable as well but I think especially with larger more complex models we start veering towards offense balance and I think I I just want to emphasize I think one of the main reasons for this has to do with how difficult the fixes are so in the case of software you get bugs that are and vulnerabilities that are relatively well to find once you find them relatively easy to fix roll out the fix the safety challenges that we're facing with highly capable AI systems are are quite complex we have huge research teams around the world trying to figure them out and it takes you know it's a very res resource intensive process takes a lot of talent vulnerabilities are still easy to find we just well they're just a lot harder to fix so I think this is a main reason why the offense defense balance probably skews more towards off fence and enabling malicious actors because you can still you still find the vulnerabilities you can still you know manipulate the vulnerabilities take advantage of them harder to fix even if you have more people involved um so that's yeah the high level evaluation mostly right I just wanted to push on the the safety the safety issue a little harder gotcha yeah I wonder so yeah one thing this actually brings up for me is you may or may not be familiar so there's AI that plays the board game go yes there's open source training run um and some colleagues of mine have found basically like you can cheaply find um adversarial attacks so like basically dumb goots that play in a way that confuses these like computer AI policies and the these these attacks they're not like generally smart but they they just like push the right buttons on the AI policy yeah and yeah I I guess this is more of a comment than a question but like like there's been enough rounds of like back and forth between like these authors and like the people making this open source goots that it's potentially interesting to just use that as a case study of the offense defense balance um yeah so saying that like given that the system was open source and that people could sort of use it and and query it and then send the information back to the developers that that's yeah yeah and and in fact like that these authors have been working with the um the developers of this software and in fact what's happened is like the de the developments of this software C if people want to Google it like they've seen this paper they're like trying to implement patches like to fix these issues yeah you know the the people finding the adversarial policies of like are basically checking if the the fixes work and like publishing the information about whether they do or not yeah absolutely so I want to be really clear that like this is a huge benefit of Open Source development is getting people like involved in the development process but but also you know using the system finding the bugs finding finding the issues feeding that information to the developers um like this is this is a huge benefit of Open Source development for software and AI systems I think that you know this is specifically why the paper focuses on like the big highly capable Frontier Foundation models is that this gets more difficult the more big complex Cutting Edge the system is like some bugs will still be relatively small well defined and there are you know bug Bounty pro programs um proposals for AI safety Bounty programs as well helping to find these vulnerabilities and give the information back to the developers yeah I think I think it like there are issues though with respect to some of the the larger larger safety issues that are more difficult like sometimes it's difficult to identify the safety problems in the first place more difficult to address the safety problems then there's also the issue of just like rolling out the fixes to the system so software development you know you you fix a bunch of bugs you roll out an update often times in the license it'll say that you're supposed to actually use the updated version there's some some data that came out I can't remember which which organization it was right now I'll have to look it up later but you know there it's actually quite a low uptake rate of of people actually running the upto-date software so first of all like even with just normal software it's hard to guarantee that people are actually going to run the up-to-date version and roll out those fixes which is an issue with with open source because like if you're using something behind API then you just update the system and then everyone's using the updated system if you have an open source system then people actually have to download and run the update version themselves with Foundation models there's actually like a weird incentive structure that changes where people might actually be like de incentivized to to update so with software oftentimes when you have when you have an update it fixes some bugs and it improves system functionality when it comes to safety fixes for foundation models often times it has to do with reducing system functionality like putting on a filter that says say well now you can't produce this class of images now you can't do this kind of function with the system so like it's hard like I don't I don't know if there's good information on how this is actually panned out now like are we seeing lower uptake rates with updates for AI systems I I I don't know but um yeah there might be something weird with incentive structures going on too where if if updates basically equate to reducing system functionality in certain ways people might be less likely to actually take them on board yeah I don't have a good feel I don't have a super good feel but like just I don't know interesting Food For Thought perverse incentive structures yeah the one the one case I don't know I'm still thinking about this Caro case so like so that's that's a case where the attack like does reduce the system functionality and people like are interested in getting the latest version with fixes it also occurs to me that so in fact the structure of this paper like the way they found the attack did not rely on having access to the model weights um it relied on basically being able to query the Goot policy basically to try a bu of things and figure out how to like trick the Gobot policy yeah now it's really helpful if you can have the weights locally just so that you can call the API you know so so that you can like call it a lot yeah but um that was not a case where you needed the actual weights to be shared which so on the one hand that's like a point that sharing the weights is like less valuable than you might think but it also suggests like if you're worried about people finding like these adversarial attacks then just putting the weights behind an API doesn't protect you as much as you think like maybe you need to like rate limit or something yeah I think that's you know that that's a valuable Insight is there are definitely things you can do without weights I mean this is an argument for why like you should be worried anyway but it's also the argument for like you know there's there are lots of arguments for well open sourcing is important because you need it to do Safety Research and getting more people involved in Safety Research will result in safer systems you have more people input into these processes but like you just illustrate a perfect example of how just having query access for example to a system can allow you to do a significant amount of Safety Research in terms of finding vulnerabilities so like yeah query access is one like it's that's can be done completely behind an API but then even if we think about something like interpretability research interpretability research does require much more in-depth access to a system to do but arguably like this is an argument for needing access to smaller systems we we're struggling to do interpretability research on smaller well- defined systems sort of like the like the rate the rate limiting factor on interpretability Research isn't the size of the models people have access to the way I understand it at least like if if we're struggling to do interpretability research on smaller models I feel like having access to the biggest most Cutting Edge Frontier Model is not what needs to happen to drive interpretability research so I I mean yeah I think it depends on this as well um like there's a range of different kinds of AI research not all of it requires Open Access and then some of the kinds that does require open access to the models isn't necessarily the most like helped the most by having the open access and then there are also like this idea of alternative approaches that we talk about in the paper like you can you can help promote AI Safety Research by by providing access to like specific research groups or like there there are other things you can do to give people that access they need to do the Safety Research yeah so yeah I guess I can share my Impressions here um I think yeah so I mean interpretability research is like a broad bucket right it like describes a few different things I think like there are some kinds of things where you know you want to start small and we haven't like progressed that far beyond small right so just like understanding like can we just exhaustively understand how certain neural netw Works start small you know GB but I think like like one one thing you're potentially interested in in in the interpretability context is like how things get different as you get bigger models or like yeah you know do bigger models learn different things or do they learn more like what sorts of what sorts of things start getting represented like can we use interpretability to predict like these shifts there you do want bigger models in terms of like how much can you do with without access to weights I mean there's definitely a lot of interpretability work on these open source models because people apparently really do value having the weights I think like I mean even in the case of um the adversarial policies work I was just talking about like you don't strictly need access to the weights but if you can you know like if you could run the games of go purely on your computer rather than like calling the API like waiting for you know that your request to be sent across the internet and the move to be sent back and doing like doing that a billion times right or I I don't know the actual number but it seems like just practically it's easier to have the um to have the model I also think that there are intermediate things so so one thing the paper talks about and I guess your colleague Toby shlan has talked about is basically structured access of you know yeah giving certain kinds of information available to maybe to certain people or like you know maybe you just say like these types of information are available these types aren't I mean I've had I've heard colleagues say like even if you didn't open source gp4 or gpt3 just like providing like final layer activations or like certain kinds of gradients could be useful which would not like I don't think that would provide all the dangers that or you know all the risks that open sourcing could potentially yeah I think this is a really key Point as well is like trying to get past this open versus closed dichotomy you know just saying that something isn't open source doesn't necessarily mean that it's completely closed and no one can access it yeah so like yeah you said like Toby chevin talks about structured access um I know there a paper we referenced at least when we referenced it it was still forthcoming it might be out now but it was a Toby chevlin and Ben Bucknell were working on it and it was about the potential of of developing research apis so how much how much access can you provide behind API to enable safety research and what kind of access would that need to look like and how could those research apis be be regulated and and who bu so I think like if there's a genuine interest in promoting AI Safety Research and a genuine acknowledgement of the risks of open sourcing we could put a lot of resources into trying to develop and understand ways to get a lot of the benefits to Safety Research that open sourcing would have by alternative means it won't be perfect uh like this by definition that's not completely open but if we take the risks seriously I think it's worth it's definitely worth looking into these alternative model sharing methods and then also into the other kinds of proactive proactive activities we can engage in to help promote Safety Research whether that's committing a certain amount of fund to Safety Research or developing International Safety Research organizations and collaborative efforts I know like one issue that always comes up when talking about like okay well we'll just provide Safety Research access through or will provide privileged downloaded access to certain groups it's like well who gets to decide who has access who gets to do the Safety Research and so I think this points to a need to like have some sort of a multi-stakeholder governance body to to sort of mediate these decisions around who gets access who gets access to do the research whether you're talking about academic Labs or other private Labs you know sort of like you have multi-stakeholder organizations decide how to distribute grants to do Environmental research or like you know you have you have Grant making bodies that distribute Grant funds to different academic groups you could have similar type situation for Distributing access to more highly capable potentially dangerous systems to academic groups research groups Safety Research institutions that meet certain standards and and that can help further This research so like I just I feel like if there's if there's a will to drive Safety Research forward and if access varying degrees of access are needed to allow the Safety Research to happen there are things we can do to make it happen that do not necessarily require open sourcing a system yeah and I think like we said there different kinds of research Safety Research require varying degrees of access it's not like all Safety Research can be done with little access no you need different amounts of access for different kinds of Safety Research but if there's a will there's a way yeah so I want to ask something like a bit more quantitative about that like so some of the benefits of open sourcing can be um gained by sort of halfway measures or you know by like structured access or like pursuing tons of collaborations but as you mentioned like it's not going to be the same as if it were open sourced do you have a sense of like if you want to I guess it's going to depend on like how constrained You Are by safety but how much of the benefits of Open Source do you think you can get with these more limited sharing methods I mean that's that's a good question I think I think you can get quite a bit and I think again it sort of depends what kind of benefit you're talking about so in the paper I think we discuss three different benefits Let's see we talk about accelerating AI research so that's Safety Research and capability research we talk about Distributing influence over AI systems and this sort of ranges everything from like control over AI system like who gets to control the systems uh who gets to make governance decisions about the systems who gets to profit for it kind of like wraps all the democratization themes together under Distributing influence over Ai and then let's see what was the other one that we talked about You' think I've talked about this paper enough in the last three months I'd have it down external model evaluation oh right yeah um yeah so enabling like external oversight and evaluation and I think it depends which one you're talking about so for example if we're talking about maybe okay let's start with external model evaluation I think that this probably benefits the most from open sourcing it depends what you're looking at so for example if you're just sort of looking for like minor bugs and stuff like that like you don't need open source access for that but having more in-depth view to the systems is more important for sort of trying people trying to like help find fixes to the bugs with we've discussed this there are also risks associated with open sourcing if we're talking about accelerating capability research for example which sort of falls under the second category I think you might find that like the benefits of open sourcing here might be somewhat limited um the larger and more highly capable the system gets and I think this largely will just have to do with who has access to the necessary resources to really operate on The Cutting Edge of research and development like open source development it operates sort of like behind the frontier right now largely because of restriction not restrictions but um just the expense of the necessary compute resources and then you talk about Distributing control over AI we've already kind of discussed the distributed effect of open sourcing and model sharing on Distributing control it's sort of like a a second order effect you get more people involved in the development process and then large Labs have more competition and then it distributes influence and control there are probably more like direct ways you can help distribute control and influence over AI besides making a system widely available yeah so to answer your original question then about like how many how much of the benefit of open sourcing can you get through alternative methods I guess it really depends what benefit you're talking about I think for like AI safety progress probably quite quite a bit honestly actually the vast majority of it given that a lot of the Safety Research that's done on these highly capable Cutting Edge models is something that has to happen within well-resourced institutions anyway or like you you need the access to the resources to do that not just the code and the weights but the computational resources and so on yeah so I think I think quite a bit I think I think it's it's less of a can we get close to the same benefits that open sourcing allows it's more of like can we do it in one foul swoop open sourcing like that's the thing it's like open sourcing is the easy the the easy option here it's open and now you get all these benefits from open sourcing the decision to open source or not like the part of the reason it's a hard decision is because achieving these benefits by other means it's just it's harder it's going to take more resources to invest it's more organizational capacity more more thought more cooperation more like it's just it's going to take a lot of infrastructure a lot of effort yeah it's not it's not the One-Stop shop that open sourcing is but I think the idea is that if the risks are high enough if the risks are severe enough it's worth it I think that's where it comes in so I guess it's just it's worth reiterating again and again like this paper is not an anti-open Source paper very Pro open source in the vast majority of cases what we really care about here are like Frontier AI systems that are starting to show the potential for causing really catastrophic harm and in these cases let's not open source and let's pursue some of these other ways of achieving the same benefits of Open Source to to safety and distributing control and model evaluation but you know if you open source away below that threshold the the net benefits are great gotcha so my next question yeah I actually got a bit sidetracked and want to talks it earlier but um so in terms of the offense defense balance um in terms of like the the harms that that you are worried about from open sourcing I sometimes hear the claim that like basically like look AI you know if you open source it it is going to cause more harm but you also enable more people to deal with the harm right so I I think there they're talking about offense defense balance not of like finding fls in AI models but in the underlying like issues that AI might cause right so I guess the idea is something to character it it's something like look if you use your AI to create a pathogen you know I can use my AI to create a broad spectrum antibiotic or something and the hope is that like in these domains where we're worried about AI causing harm look just open sourcing AI is going to enable tons of people to be able to deal with the harm more easily as well as like enabling people to cause harm so yeah I'm wondering yeah what what do you think about the the sort of underlying offense defense balance as opposed to that within AI yeah I get the argument I know I just I'm I'm personally I'm wary about the arms race Dynamic though you know you got to you got to constantly build the stronger technology to keep the slightly less strong technology in check yeah I I mean I guess this comes back to like that very original question you asked about what about just hitting the no more AI button um yeah so like I I guess I I get the argument for that I think there's weird Dynamics yeah I don't know I'm not doing a very good Jam job answering this question I'm just like I'm I'm personally concerned about the race Dynamic here and I think it just sort of comes back to this issue of how how hard is it to fix the issues and vulnerabilities in order to prevent the misuse in the first place like I think I think that should be the goal is preventing the misuse preventing the harm in the first place not saying can we build a bigger stick I know this is also like there's a similar argument that is brought up when people talk about the benefits of producing increasingly capable AI systems and saying like oh well you know we we need to plow ahead and build increasingly capable AI systems because you never know what we'll develop a system that'll help cure cancer or or develop some renewable energy technology that'll help us address climate change or or something like that like this like what what huge problems could AI help us solve in the future and I don't know just I mean this is this is personally me like I I don't know what my other author the other authors on this paper think of this but I I don't know like I kind of feel like if those are the goals if the goals are to solve climate change and cure cancer take the billions upon billions upon billions and billions of dollars that are currently putting into training AI systems and go cure cancer and and develop renewable Technologies and yeah so I don't know those those I struggle with those arguments personally I'd be interested just to hear your thoughts like these are these are not like I have not written about this I have it this is just this is me riffing right now so I'd be interested to hear your thoughts on on this train of thought as well yeah I I mean I think the original question is sort of like unfairly hard to answer just because like it's sort of like asking about the offense defense balance of like any big any catastrophic problem AI might cause and it's like well there are tons of those and it's pretty hard to think about right yeah yeah I mean I do think that so the thing you were saying about like if you wanted to cure cancer maybe step one would not be create like incredibly you know smart AI yeah I I think like I've seen this point I don't know if if you know David Chapman's better without AI no not familiar yeah so so he basically argues like uh we just shouldn't build like big neural Nets and it's going to be terrible um I also Jeffrey heninger at AI impacts I think has said something similar along these lines I mean on the one hand like I do kind of get it just in the sense that like like if I weren't worried about misaligned AI it's kind of like there's this hope that this is like the last invention you need you know yeah you create Ai and now like instead of having to separately solve cancer and climate change and you know whatever just like make it solve solve those things for you right yeah I it's just really hard to kind of look forward and it's like you have to decide now whether or not this technology is that silver bullet and how much investment it's going to take to get to that point yeah I think I think that's right and I think that there's also just yeah I I think your take on this is going to be just driven by your sense of the risk profile of building things that are just significantly smarter than us I guess listen from the fact that I made the like ex risk research podcasts rather than the AI everything's going to be great research podcast you know people can guess my an indication of where you're but um I don't know I think it's a hard question I think like the so so part of my take is like in terms of the offense the underlying offense defense balance I think it becomes more clear when you're worried about like what should I say cogenic risks like B basically like the AI itself like coming up with issues rather than like humans you know using AI to like have nefarious schemes like like once you're worried about like AI doing things on their own where like you're not necessarily in control like there I think it makes sense that like you're probably if you're worried about not being able to control the AIS you're probably not going to like be able to solve the risks that the AIS are creating right yeah yeah like your your management plan for AI shouldn't be to build a slightly more powerful AI to manage your AI yeah well if you if you knew that like you were going to remain in control of the like slightly bigger AI like yeah maybe that's a plan but um no saying that like if if you're worried about loss of control scenarios then the solution shouldn't be well let's let's build another system that's also out of our control but just slightly better aligned to to address the like I I feel like greatest yeah yeah I think my colleague John Wentworth has some saying like uh you know releasing Mothra to contain Godzilla is not going to like increase property values in Tokyo which is a cute little line um yeah it's it's I don't know it's a hard question I think it's hard to say anything very precise on the topic I did want to go back to the offense defense balance so moving back a bit I thing you said was something like look it's probably better to just prevent threats from arising than it is to like like have someone make a pathogen and then like have everyone race to create a like antibiotic or antiviral or whatever so that's one way in which like everyone having like really Advanced AI That's one way that could look in order to deal with threats I think another way does lookit more preven I know it's also more dystopian sounding I guess but one thing that AI is good at is surveillance right yes potentially so you could imagine like look we're just going to open source Ai and what we're going to use the AI for is just like basically surveilling people to make sure the threats don't occur so maybe one version of this is you just like really amp up waste water you know somehow use your AI to just look at the waste water and see if like any new pathogens are arising it could look more like you have a bunch of AIS that can detect if other people are trying to use AI to create super weapons or whatever and like stop them before they do somehow but but then why like I don't okay Wastewater example like yeah that sounds great we should probably do that anyway in terms of like you know surveilling to see how people are using AI systems using AI why why not just have the AI system be behind an API where people can use the systems for a variety of Downstream tasks integrating through this API and then the people who control the API can just see how the system is being used like even even if it can be used for a vast majority of tasks like even if you take even if you were to take all the safety filters off the advantage of the API is still that you can see how it's being used you know that I I don't know I feel like that's yeah that seems like a good argument um all right okay okay so moving I I guess another question I have is related to the frame of the report so in the report you're basically like okay open sourcing has these benefits but it also has like these costs what are ways of like doing things other than open sourcing that basically try and retain most of the benefits while uh getting rid of most of the costs you can imagine a parallel universe universe report where you say like okay open sourcing has these benefits it also has these costs how can we we're still going to open source but we're going to do something different in our open source plan that is going to retain benefits and reduce costs right yeah so like one one example of this is you open source models but you you have some sort of watermarking or you have some sort of like cryptographic back door that can like stop models in their tracks or whatever yeah I'm wondering like yeah why the frame of like alternatives to open sourcing rather than like making open sourcing better very simple I think making open sourcing better is the harder question um like Tech technically more difficult I mean for example say say you have water marking part of the issue with water marking to like identify artificially generated images is making sure the water marks stick how do you make sure that they are IR removable if you are going to open like this is a really complex technical question how do you develop a system that has watermarked images where that Watermark is irremovable if you were to open source the system I'm not saying it's undoable I I personally don't have the technical background to comment very deeply on this I have heard people sort of talking about how possible this would be it's also like it depends how you Watermark right yeah if you have like just a line of inference code that says slap a lot watermark on this thing like okay delete the line of inference code if you're to train the system on images that like only watermarked images well now you have to retrain the entire system to get it to do something else right which is very expensive so again I think it kind of like depends how you do it I was at a meeting last week where people were sort of talking about like are there ways we could like I don't know like build in mechanism into the chips that run the systems that say like if some bit of code is removed or changed in this system then like the chip burns up and won't run the system like like are there ways we could you know it's like I'm not saying this is impossible but you know really interesting technical question really difficult definitely beyond my area of expertise but I think like if this is an approach we can take and say like there are ways to be able to open source a system and get all the benefits of open sourcing by just open sourcing and still miate the risk I think that's great I think it's just a lot more difficult and we do actually talk there's one aspect in which we kind of do take the flip view in the report and I think this is this is where we start talking about uh staged release of models yes so St like you can you can undergo a stage release of a model where you put out a slightly smaller version of a model behind API you study how it's being used you know maybe you take a pause analyze how it was used what the most common Avenues of attack if at all were being used to try and misuse the model and then you release slightly larger model a slightly larger model you just you do this iteratively and you know if you do this process as you get to a stage where it's like hey we've been doing the stage release of this model for however many months and uh no problems look looking good you know there's no mergent capabilities that popped up that are making you worried you didn't have to implement a bunch of safety restrictions to get people to stop doing unsafe things like okay open source you know this is not a this is not a binary yes it has to be completely open or completely closed and I think this is one respect like if you were to take this flip view of how can we open source but do it in the safest way possible just open source slowly take take some time to actually study the impacts and you know it's not like you have to it's not like the only way to sort of get a sense of how the system is going to be used is to just open source it and see what happens you could do a stage release and study what those impacts are again it won't be perfect you never know how it's going to be used 10 years down the road once someone gets access to all the weights and stuff but but it is possible to study and get some some sort of insight and I think one of the nice things about stage release is like if you start doing the stage release process and you realize that you know at each iterative step you are having to put in a bunch of safety filters for example to prevent people from doing really shady stuff that's probably a good indication that it's not ready to be open sourced in its current form because those are safety filters that will just immediately be reversed once open sourced so yeah I think you can get like you can learn a lot from that so I think like that's one way you could open source safely is find ways to actually study what the effects are before you open source because that decision to open source is irreversible and then yeah I think that I think the technical challenge of are there ways we can sort of like have back stops that we can like technically build in like irreversible IR removable filters or water marks or or even just like Hardware challenges that we could Implement I think really interesting technical questions that I don't know enough about but yeah go for it like that'd be a great world yeah if if listeners are interested this this gets into some territory that we that I talked about with Scott arenson earlier this year yeah I I think like the the classic difficulties at least so for watermarking I I read one paper that claims to be able to like bake the water mark into the weights of the model I didn't really to be honest I didn't actually understand how that works I I think it's it has to do with how the model's trained so the way I understand it is like if you have a data set of images that all have a watermark in that data set like you know not not watermark in the sense like like you see on a $1 bill but like like weird pixel stuff that the human eye can't see if all the images in the training data set have that Watermark then all of the images it produce will have that Watermark so the only way to get rid of the water like in that case it's like it's baked into the system because of how it was trained so the only way to get rid of that Watermark would be to retrain the system on images that don't contain the watermark yeah that's one possibility so that's that's going to be a lot rougher for like applying to text models of course like if you want to just train on like the whole internet I think that like yeah there's I I I think I saw something that claimed to work even on cases where the data set was not did not all have the watermark but I really understand how it worked but at any rate the key like kind of the key issue with these sort of water marking methods is as long as there's one model that can basically paraphrase that does not have water marking then you can just like take your watermark thing and basically launder it and get something that if your paraphrasing model is good enough yeah you can create something that's like looks basically similar doesn't have the watermark and then you know then it's sad news yeah and then yeah sorry so I was going to say like there's you know similar in terms of like how doing something with one model allows you to jailbreak another model kind of thing I mean this is what's happened with the adversarial suffixes paper right where you have um you know using using a couple open- Source models one of which was Lama 2 and you know using the weights of those models figuring out a way to basically basically just throw like a random seemingly random string of numbers at a at a large language model and then with that seemingly Rand range of numbers before the prompt basically get the system to do whatever you want except while it was while they figured out how to do that using the weights accessible from L 2 it worked on like all the other large language models so you know Finding finding a way to kind of jailbreak one model and using you know using the weights and access to one model that could that could bring up vulnerabilities and tons of others that aren't open sourced as well so think that's just another roughly related somewhat to what we were just talking about point yeah I guess it brings up this high level thing of like you want whatever governance method for AI you want you want it to be robust to like some small fraction of things not you know breaking the rules like you don't you don't want the small fraction to kind of poison the the rest of the thing which which waterm marking unfortunately has yeah I I guess I wanted to say something brief about back doors as well like so there there is really a way of at least in toy neural networks and you can probably extend it to bigger neural networks you really can introduce a back door that is cryptographically just hard to detect but so partly I mean one problem is like how do you actually use this to prevent AI harm is like not totally obvious I guess the second issue only comes up with like super smart AI but like if you have a file on your computer that's like I implanted a back door in this model the back door is this input yeah then you know you could then it's no longer cryptographically hard to find as long as somebody can break into your computer which hopefully is cryptographically hard but um there are I guess there are security vulnerabilities there so yeah I I wonder if if you want to say a little bit about the the safer ways to get the open source benefits I've given you a chance to talk about the them a little bit but is there anything more you want to say about those um I think I think not not really um I think just the the overarching point is uh you know just as said before like if there's if there's when the risks are high and I think that's key to remember I'm not saying don't open source everything you know when the risks are high it is worth investing in seeing how else we can achieve the benefits of open sourcing yeah basically if if if you're not going to open source because the risks are high then then look into these other options it's really about getting rid of this open versus closed dichotomy so many of the other options have to do with other options for sharing models whether that's structured access behind API even research API access gated download St release and then also more proactive efforts proactive efforts which can actually also be combined with open sourcing they don't have to be seen as an alternative to open sourcing so this is things like redistributing profits towards AI Safety Research or starting AI safety and Bug Bounty programs or I me even like we talked about with the democratization paper thinking about how we can democratize decision-making around AI systems to help distribute influence over AI away from large Labs which is another argument for open sourcing and so yeah I think that this is this is key it's just like there are there are other efforts that can be put in place to achieve many of the same benefits of open sourcing and when the risks are high it's worth really looking into these all right okay so moving on um I want to just talk a little bit more broadly about the field of AI governance research so historically this podcast is mostly focused on technical AI alignment research and I imagine most listeners are more familiar with the technical side than with uh governance efforts oh in which case I apologize for all my technical inaccuracies one of the benefits of having 25 co-authors is that a lot of the technical questions I got to Outsource makes sense yeah it's it's it's good to be interdisciplinary um but so this is kind of a broad question but like how is AI governance going like like what's the state of the field if you can answer that briefly field AI governance um yeah okay I'll try and answer that briefly it's going well in that people are paying attention Okay yeah in in in this respect like the release of chat GPT I think was really great for AI governance because people besides those of us already doing AI governance research are really starting to see this as something valuable and important that needs to be talked about and questions around what role should governments play in regulating AI if at all how do we get this balance between governments and and the developers who should be Reg at with respect to different things you know do all the responsibilities lie on the developers or is it on the deployers and like all these questions suddenly are they're coming to light and there's more general interest in them and so we're seeing things like you know the UK AI Summit is happening next week Global AI Summit looking at AI safety really concerned about catastrophic and existential risks trying to understand what kind of global institutions should be in place to govern AI systems to eval AI systems to audit to to regulate and you know this is bringing in countries from from all over the world I think it's something like 28 different countries are going to be at the UK AI Summit um you have the EU AI act you know where they're it started a while ago looking at narrow AI systems but now is taking on Foundation models and Frontier AI systems and looking at open source regulation and um this has really like over the last year exploded into a global conversation so in that respect AI governance is going well in that people are paying attention it's also very high stress because suddenly everyone's paying attention we have to do something um and so but you know I think I think there's there's really genuine interest in getting this right and that I think that really bodess well so I'm excited to see where this where this next year goes yeah I think you know there's talk about having this Global AI Summit and then making this a recurring series and uh yeah so I think I think it's going well in the sense that people are paying attention and and the wheels are starting to turn and that's that's cool okay I guess related to that what what do you see is like the the most important like open questions in the field in the field of AI governance okay so I think like one one big one is compute governance which my colleague Leonard Heim works on so this is just thinking about how compute is a ever for trying to regulate who is able to develop large models um even like how compute should be distributed so that more people can distribute large models but basically using compute as a lever to understand who has access to and who is able to develop different kinds of systems so I think like that's that's a huge area of research with a lot of growing interest because computes just sort of like it's one of the tangible things that we can actually control the flow of I think that the questions around model sharing and open sourcing is getting a lot of attention right now big open question a lot of debate like I said it's becoming really quite a polarized discussion so it's getting quite hard to cut through but a lot of good groups working on this and I think a lot of interest in like genuinely finding common ground to start working on this like when I get in um I've had a couple situations where I've been in groups or workshops where we get people who are you know very Pro open source and other people who are just like no let's just shut down the whole AI system right now you know like really both sides of the spectrum coming together and we try and find a middle ground on okay where where do we agree is there a point where we agree and very often we can come to a point of agreement around the idea that there may be some AI system some model that poses risks that are too extreme for that model to be responsibly open- sourced and you know that might not sound like that extreme of a statement but when you have people coming from such polarized views to agree on the fact that they're exist a model one day that should not be open sourc that is a starting point you can and you can start the conversation from there and and every group I've been in so far has has got to that point and we can start working on that so I think this model sharing question is a big open question and like lots of technical research needs to be done around benchmarking to decide like when when are capabilities too dangerous also around understanding like what what activities are are actually possible given access to combinations of model compon and that's actually enely and we need more fine grained of what you can actually do given different kinds of model combinations of model components in order not only to have safe standards for model release and really fine grain standard for model release but also to protect the benefits of open sourcing you don't want to just have a blanket don't release anything if you can get a lot of good benefit out of releasing certain model components so I think a lot of technical research has to go into this anyway so yeah second point I think model sharing is a really big point of discussion right now and then with the upcoming UK AI Summit quite a bit of discourse around what international governance structures should look like for AI uh a lot of different proposed models and yeah it'll be interesting to see what comes out of the summit I don't think they're going to agree on anything amazing at the summit it's two days um but you know I think I think come for for me a really great outcome of the summit would be first recognition from everyone that AI systems could pose really extreme risks so just a recognition of the risks and then second a plan going forward a plan for how we can start establishing International systems of governance and sort of like really structure out when are we going to come to what kinds of decisions and how is something how can we start putting something together so yeah I think that those are probably three key open questions and the the international governance structure one is really big right now to Just Right given the upcoming Summit yeah and I guess uh unless I unless we get that editing and transcription for this episode Done unusually quickly listeners the UK AI Summit is almost definitely going to be in your past so I guess listeners are in this interesting position of knowing how how that all Pann out in a way that we don't so so that was that was open questions in the in the field broadly I'm wondering for you personally as a researcher what things are you most interested in looking at next interesting I mean most of my life is kind of taken up with um like follow up on this open source report right now okay so I definitely want to keep looking into questions around model sharing and maybe setting responsible scaling policy responsible model release policy I'm not exactly sure I I think I'm kind of I'm kind of in this place right now where I'm start I'm trying to feel out where the most important work needs to be done and whether like the best place for me to do is like to encourage other people to do certain kinds of work where I don't necessarily have the expertise like we were talking about like needing the more technical Research into what is possible giv access to different kinds of different combinations of model components or like are there are there specific areas of research I could try to help lead in or whether really what needs to be done is just more organizational capacity around these issues so I know I'm I'm personally interested in sort of like keeping up with this model sharing discussion I think there's a lot of interest work that needs to be done here and it's a key thing that's being considered within the discussions around International AI governance right now yeah so sorry I don't have as much of a clearcut answer there but um yeah I'm still sort of sort of Reeling for from having published this report and then everything that's coming off the back of it and just trying to see yeah just trying to feel out where's where's the next most important most impactful step what what work needs to be done so I guess if any of your listeners have have really hot takes on like oh this is what you should next I guess please please tell me helpful how should they tell you if someone's just heard that and they're like I need to I need to tell her now she must know yeah so I mean I I have a I have a website where you can find a lot of my contact information um or you can always find me on LinkedIn I spend far too much time on LinkedIn these days okay and also my my email address happens to be on the open source report so if you download the report my email address is there and and what's the what's the L of your website uh Elizabeth seeker.com all right okay getting back to talking about governance in general I'm wondering so I guess this is an xris Fus podcast how if at all do you think government's research looks different when you're when it's driven by concerns about X risk mitigation versus other concerns you could have about AI governance well that's a good question how does how let's see so the question is like how does the governance research look different um yeah like what kinds of different questions might you focus on or what kinds of different what you know what kinds of different focuses would you have that would be driven by X risk worries rather than by other things I mean so this is something that you know I've I've had to think about a lot in my own research development because I did not come into this area of research from sort of like an X risk background interest I came I came into it um I mean honestly I started in bioethics and then sort of moved from bioethics looking at AI systems and healthare and and have sort of like moved over into the AI governance space um over a very long PHD program and so here I am but I would say like one of the things that I've learned working in the space more interested in long-term X risk impacts of AI and trying to prevent X risks is really paying attention to causal pathways and really trying to be very critical about How likely a potential pathway is to actually lead to a risk I don't know if I'm if I'm explaining this very well but I guess it's starting from a point maybe a better way of saying is like if you have a hypothesis or or like let's say you're worried about the impacts of AI systems on influence operations or impacting political campaigns I find it really helpful to start from the hypothesis of it won't have an impact H and then and really just trying trying to understand how that might be wrong as opposed to kind of trying to start from like oh AI is going to pose this massive threat to I don't know to you know it's going to pose a massive biothreat or it's going to pose a massive threat to political operations or something like that and then almost trying to like prove that conclusion yeah I don't know I I start sort of from the opposite point and then try and think about all the ways in which I could be wrong and I think this is really important to do especially when you're doing X risk research whether it's with respect to AI or or or some other form of X risk because I think I think there are a lot of people that turn off when you start talking about existential risks they think it's too far out there you know it's not it's not really relevant to the important questions that are impacting people today the tangible things that people are already suffering and so I think it's really important to be very very rigorous in your evaluations and um and have a very clear story of impact for why it is that you're doing the research you're doing and focusing on the issues that you're doing at least that's been that's been my experience sort of trying to transition into the space and work on these issues okay another question I have I guess related to my audience so I think my audience a lot of them are technical alignment researchers and maybe they want to there are various things they could do right and maybe they're interested in okay what work could technical alignment people do that would make AI governance better I'm wondering if you thoughts on that question okay technical alignment people AI governance better yeah uh I mean there's a lot of work going in right now especially within the U government like we just set up the ukai task force and sort of like a government institution doing a lot of model EV vals and Alignment research I think like if you have the technical background in alignment research there's you you are very much needed in in the governance space there's a there's very often a disconnect between I mean I'm I am also guilty of this there's a disconnect between the people doing the governance research and the people who have the experience with the technology and like really know the ins and outs of the technology that's being discovered not being discovered that's being developed sorry yep and I think you know if you have the inclination to work in an AI governance space and help bridge that Gap that would be in incredibly valuable and like like I've already said like some some of the more technical question question even around open sourcing are things that you know I was I was very very glad to have colleagues co-authors on the paper who do work with develop have have worked for AI labs and stuff before and um really like knew what they were talking about and could advise and help write some of the more technical aspects of the report yeah so I think like if you have the inclination to work in the space to get involved with governance efforts or or even maybe some of these government institutions that are starting to pop up that that are working on the boundary of AI governance and Technical research that that could be a really valuable place to contribute yeah so I think like those my my two cents off off the top of my brain would be help help bridge that Gap okay great so before we wrap up I'm wondering if there's anything that you wish I'd asked but that I didn't oh that's a good question no I don't I don't think so like I think we've covered a lot of a lot of good stuff yeah I just thank you for having me on really I'd say there's yeah nothing nothing in particular this has been great all right so to to wrap up then if people are interested in following your research um following up on this podcast uh how how should they do that uh yeah I said so I have my website Elizabeth sear.com it sort of outlines my different ongoing research projects has a lot of Publications on it also gov ai's website governance. is a wealth of information all things AI governance from all my great colleagues at gov a and and our Affiliates so really yeah there's new research report putting out being put out almost every week maybe every other week but like really high quality stuff so you can find a lot of my work on the website or my current work and past work on my own website or find me on LinkedIn yeah just happy to talk more all right well thank you very much for being on the podcast great thank you this episode is edited by Jack Garrett and Amber Dawn Ace helped with transcription the opening and closing themes are also by Jack G financial support for this episode was provided by the long-term future fund and light speed grants along with patrons such as Tor barod Lexi mfv and Ben Weinstein Ron to read a transcript of this episode or to learn how to support the podcast yourself you can visit axr p.net finally if you have any feedback about this podcast you can email me at feedback axr [Music] [Laughter] p.net [Music] [Music]