Senate Standing Committee on Privacy, Digital Technologies, and Consumer Protection

All right. The Senate Committee on Privacy, Digital Technologies, and Consumer Protection informational hearing on the understanding the contours of the AI threat scape is called to order. We will be joined shortly by other members of the committee as well, and the remainder of the committee members, I know, are monitoring this hearing from their districts as we are in the fall-- in the Senate's fall recess.

I wanna welcome in advance our colleagues for joining us today. This is a monumental moment for for California and we are here today to continue California's work in this space, but really to meet the moment of one of the most important technological, economic, and social changes that human society, certainly the State of California, has ever faced. California is defined by our focus, our belief, our love of innovation.

In every possible way, California has changed itself, and the world through its-- our constant, constant attention to the possibilities that are in front of us and the rules that are to be broken, the new things that are to be made. Whether it's in the world of entertainment or aerospace or technology or food and agriculture, California has led the way repeatedly. California's led with its Western orientation, its cowboy culture.

Let's try anything to see if we can make it better. California, though, has also, along with our friends in Europe and elsewhere, has also led the way in governance to match that innovation at every single step, and I emphasize this only because I know in the rest of the country, there is a constant debate about whether innovation and governance, innovation and regulation, innovation and control are mutually exclusive, and California is the textbook example in the globe of how that isn't true, and in fact, how the opposite is the case.

That predictable, meaningful, trust-building governance and regulations are essential. They are the essential foundation to innovation, both its safety, but also its widespread adoption, use, and embrace by the public, by consumers, and by our citizens. Today, we're here because California has led the way with respect to artificial intelligence regulation and governance in many, many ways, and it is wholly insufficient to meet the moment that is now before us.

California was the first in the country to adopt a sweeping statute, SB 53, with respect to catastrophic risk transparency and response, building off an even more ambitious proposal in SB 1047 in the year prior, and that law has been copied and evolved and adapted by other states. And it is really in many ways-- and it has defined the terms by which most of the country is talking about this issue.

And our-- California's work on the frontier model safety and particularly the work of Senator Scott Wiener, our colleague who is occupied at the moment and unable to join us at the hearing today, but he is very much with us because both his prescience in the introduction of SB 1047 and his tireless work with other members, but also with other coalitions, to win enactment of ultimately SB 53 led the world in focusing on and experimenting with approaches to frontier model safety.

And so he-- although he's not with us today, as I say, he is very much with us in terms of the work that he has set in motion, the models and the concepts, the ideas, the bars by which we judge ourselves as well as the technology, and wanna thank him for that. California's also led the country in legislating to protect children from companion AI chatbots.

We have enacted legislation just since last year on workplace privacy and workplace protections for workers around job loss, around things like deepfakes and intellectual property, around healthcare and around bias, just to name a few. So California, really more than any other state, has been leading in terms of artificial intelligence response and regulation.

Welcome to our colleague, Senator Becker, who also has a distinguished record of leadership in this area, and I'm extremely grateful that he's with us, today. So California has led the way, and at the same moment, we are facing a menu of threats and risks that are quite substantial, both in their intensity and their potential severity, but also in their scope.

And so we-- what remains before us and what will be a significant focus of today's hearing is frontier model safety and the risks that humans, that society, that consumers, that government, that all of us face from frontier models that-- whose capabilities increasingly outstrip what we thought was possible or imaginable, even just a few years ago. And so, that will continue to be a major area of focus for us as it has become for the entire country.

At the same time, there-- we now have non-frontier models and agentic systems that have capabilities that far exceed what we were regulating frontier models about in SB 1047, certainly in SB 1053, but particularly in SB 1047 just a couple of years ago.

And what has become clear is that in addition to what our AI friends might say, there are the intelligence gated threats from frontier models who we are insufficiently capable of controlling, that there are also substantial less intelligence gated threats from agentic systems and from open source and open weight models, and their deployment broadly in society that don't depend solely on the intelligence of artificial intelligence models but also depend on its persistence, on its-- we will talk about this during the hearing as well--it's persistence in making sure no task, no objective is unsatisfied.

It's shamelessness in what tactics it might undertake in order to accomplish that. And we are now, thanks to Meta and Meta's Muse, we have millions of Californians that are using agentic systems about which they understand very little and about which the legal, statutory, regulatory, and liability protections are radically misunderstood or not understood at all. And so part of the second half of our hearing or the second portion of our hearing today is really focused on those threats that are not about the frontier model specifically but are about the remainder of the AI ecosystem which frontier models are an important, but not the only part, and so that we will be taking a look at those issues.

We remain-- Californians remain concerned about the impacts of these systems on jobs and the nature of work itself as well as income around the impacts on institutions and systems. California is the first in the country to enact any legislation at all on the challenges of agentic flooding, which-- I separately described this as sort of as infinite Karens, which is an inappropriate way to describe it, but it is simply the legislation that was introduced-- that was enacted by the legislature, governor earlier this year to address the real challenge of folks, either humans or agents, abusing our civic engagement processes by flooding the systems to the point where governments shut down and where real public input cannot be seen in a flood of swarming agents.

Same threats to our legal system, but certainly also private companies, nonprofits; little leagues are grappling with exactly the same challenge of agentic flooding at the local level. California was the first to adopt any legislation in this space, but it remains largely an unprotected area. Issues of agentic inequality, which is to say, for example, in the cybersecurity space, as some of the companies have suggested that we shouldn't be that concerned about cyber security because we-- if we would just license their models to protect us against cybersecurity threats, then we would be fine.

Of course, the challenge here is that the models are not all equivalent, and so, whatever cybersecurity model we were to use here based on a cybersecurity system based on an AI model that might protect hit us here in the California State Senate against attacks. If the Assembly, licenses a more powerful, more intelligent AI model that is capable of defeating our regular priced AI model, we are at risk.

The Assembly, I'm sure, would not do such a thing, but the point is that the ratcheting, the constant ratcheting, the infinite ratcheting of the power of these models means that the models themselves can can offer us only limited protection against each-- against themselves unless we accept massive inequality so that only folks with unlimited wealth have the ability to have access to the most powerful models while the rest of us, the little league, the Senate, your corner sandwich shop, have access only to the least capable models which are incapable of defending against either agentic flooding, cybersecurity threats, or other challenges that may come from more powerful agents.

So, a more powerful model. So, agentic inequality is a significant issue for us to face. Our-- I serve also on the Education Committee and the concerns about cognitive and creative surrender, that is the extent to which students right now, but also workers and humans more generally, offload so much of our cognitive thinking and creativity onto models that we or the next generation forget or never develop those capacities in ourselves.

And so, this is an important challenge for us to face. And then privacy is a central issue. Senator Becker, with his authorship and consistent championing of the Delete Act, which gives every Californian the ability to remove their data from being held by data brokers that they don't want, how does such systems work in an era when an agent-- and I'm not picking on Meta, but because Muse is the most used agent at the consumer level in the state. If Muse has access to all of my emails, which I've given it-- I have not.

But if I've given it access to all of my emails and my bank accounts and everything and my text messages on my phone and everything else, and it is using them and not storing them in a single file in the way that we might think of under the Delete Act as an example that applies to data brokers, how are we to treat privacy issues when we're no longer talking about a single set of cookies or other files from the last decade or two in terms of that protection?

But more importantly or perhaps at least as importantly is, if I have done that and I am in communication with Senator Becker, and Senator Becker and I pass emails back and forth, text messages back and forth, I offer to pick up Senator Becker at the airport one day and he sends me his Southwest itinerary and I pull off of-- and that itinerary includes the last four digits of his credit card number--perhaps it's a foreign flight, and I pick up his passport number--Senator Becker has in no way authorized my Muse agent or Muse period or Meta to collect that data, to hold that data, or to build a profile about him.

And he has no way to know that such thing exists or that I've authorized Muse to do that, and he has no way to exercise his rights under the Delete Act or the California Constitution to take control of that data himself. And so the the issues that we're that we're facing with respect to privacy and AR are substantial.

Fortunately, California knows a little bit about this issue as one of the leading states in the country, but we have significant work to do. So I list all of these issues not to scare us into, you know, abject fear about AI. I know some folks feel that, and that's legitimate too, but simply to note that, number one, the frontier model safety issue is the essential one.

It is not the only issue. And number two, to emphasize for those who are hoping that Congress will come back and somehow make this all go away with some simple act, that we're one congressional act away from fixing AI, is folly. It is-- there's-- Congress is appropriately examining some of the key issues here, but the range of topics that are implicated by artificial intelligence go far beyond the scope of what the federal government does.

And the federal government has no record of doing this in a way that isn't arbitrary capricious out of the White House. And so, it is really important that California and other states continue to aggressively, to robustly engage in governance and safety and protection issues in this space, and I know we're committed to doing that here at the Senate.

So let me just say one other caveat which is, it isn't clear to me that if today's California existed a hundred years ago that we would have ever actually permitted automobiles to exist or planes or anything else. Our regulatory environment today is challenging. Like, I think the legislature knows this. The current governor and the incoming governor, whoever that might be, are well aware that the California governance environment is challenging.

And so, I also wanna emphasize both to my colleagues and to advocacy groups and to companies and associations that infinite random acts of legislation is also not the answer, that what we what we need is not 10,000 new laws, all sort of separately crafted to try to tackle AI, or particularly, our conception of AI on this Wednesday when it will be radically different next week.

It is quite important that we take a comprehensive approach here and that we do that in a thoughtful way both at this committee and in the rest of the institution because we have to understand here that it is the market, the market dynamics-- the market is very important and, you know, issues around regulatory capture or where we have, you know, large companies that we tend to focus on here, and in the national debate as well that have the potential to exercise significant market power.

I was mayor for 20 years and I saw plenty of technology innovations occur at the local level, and every single time, every single time, whether it was Uber or anyone else, every single time, the same exact story. The innovators would come to the city and say, hey, we wanna-- you know, we wanna let a thousand flowers bloom and cities are the engines of innovation, and so, we wanna work with you to try to, you know, show that this can be done and make it work, and we would do that.

And two years later, they would go to the states and say, we can't have all of these cities doing this, and therefore, please enact a uniform policy that applies to everyone based on our extensive two years of knowledge. But more importantly, now that we are a quasi-monopoly, please fix our our place in the economy. We cannot do that.

We cannot allow that to happen in this space as well, and it's not a threat to any individual company, but we need to be very attentive to the dynamics of the markets here and understand-- and this is to some of my advocacy friends that were-- this is not simply a humanity versus big tech battle or set of battles that we're about to engage in.

There are massive, massive fights within the tech industry or industries that we're going to have to grapple with and be sure that what we are-- that we don't adopt 10,000 pieces of legislation that make it only possible for two or three companies to dominate this space and the world. But that is also our job.

And it is the story of legislation that companies hate the regulations, but once they figure out how to comply with them, they would like to have more, or at least they won't object that much so that others then cannot enter the market. And it really is quite important that we not lock in place the most powerful economic actors in society by legislating too fast or in a way that's not comprehensive in order to avoid regulatory capture.

Second, and the other lesson for me from Uber and everyone else was, the venture capital trap that legislatures often find ourselves in, which is that at the outset of these markets, everything is is cheap. Uber trips used to be substantially cheaper than they are today, and I imagine that my Claude or ChatGPT sessions are gonna be much more expensive when companies have to make a profit, when they have to sustain themselves.

But it's typically the case that we define our regulatory landscape entirely in the venture period of new sectors without regard to what their ultimate economics are gonna look like, and we do that at the peril of Californians and Americans that depend on us to protect their affordability as well and also protect healthy markets.

And last is to be-- that it's important for us as we tackle some of these challenges is to be thoughtful about what we are capable of as humans. In the privacy domain, we're--many of us in this room have had this conversation before--we ran towards a notice and consent approach to privacy which assumed infinite attention that people could spend on privacy that none of us are doing.

Zero of us are reading all the privacy policies before we click yes to all or no to all or what have you, that we have to understand that simply transparency and choice by people in this space is unlikely to protect us against some of the most severe threats, and that while we want to assure human agency, that we also have to understand that people-- most Californians do not have the time or the cognitive bandwidth to devote all day long to reviewing policies, to saying yes or no to to prompts, that the solution to AI is not simply making it transparent for us to make more choices about, but really also do address at the root the challenges that we're facing and that we also appreciate that with respect to things that are-- the last two years, at least that I've been here in the capital, the most common term in this space has been we have to make sure there's a human in the loop.

This is not unimportant in many of our policies, but many of the incidents that we will cover today and that I think most Americans are now quite familiar with, they are a reminder that simply a human in the loop is insufficient, that our ability to-- and the importance of having the tools to be able to protect to assure safety but also to understand that there are malicious actors, there are careless actors, that it-- as a mere human alone does not guarantee what the companies might call alignment, but we might-- what we may call moral behavior, that simply, humans somewhere in the process is not sufficient to guarantee our safety here.

So, I know this is a long opening statement, but there's been a lot in this space and we are working hard in order to-- and I will simply say that many members of the Senate are working on this together to assure this comprehensive approach to safety and protection around this full range of issues, including, very importantly, frontier model safety but not exclusively one of those members who's been an important leader on artificial intelligence for years in this House as well as being the author of the Delete Act. Let me turn it over to him for some opening remarks, and that's Senator Josh Becker.

Well, thank you. I wanna thank our chair, and really, it just is not a more important conversation anywhere right now than this topic, this conversation, and frankly, for the reasons you mentioned, this state and our opportunity and responsibility here. But yeah, I wanna first thank you. We're really lucky to have someone with your experience, your background, and you're known for a balanced approach to these issues, but-- and also a level of of seriousness and rigor which is gonna require here, and I think that's reflected in the agenda for this hearing.

You know, I'll keep it pretty short. I just wanna have the conversation. I will say a couple things. One is that, you know, my-- in my district, I represent Meta, I represent Google, represent Apple, you know, many of the people who work in these industries. I was at a concert on Monday night. I was talking to someone about all these.

And in the-- I started talking about AI risks and some of my concerns, and turns out one of the guys there works at NVIDIA on, you know, some of these issues. But, you know, for a while, we focused on-- through the-- sort of the misalignment risk, the malicious risk you mentioned, and I'm a little even more concerned about risk after the stuff you went through, but the-- you know, the systemic risk in terms of jobs you mentioned and then-- but now more recently people focusing on the existential risk. And I-- that's a term I try to never use, even though I do a lot of work in climate.

I, like, very rarely-- I never use the existential word, but, you know, in this case, it's come up for good reasons, and I'll admit to at times having, you know, really significant anxiety myself about, you know, our ability to control what is being created. So I think your points about the regulatory environment in California are important, and also recognizing this is the industry where you-- the unusual piece is you have many leaders of these companies themselves talking about the risks of their products, right, which is somewhat unusual.

But, you know, your-- in your car analogy, you know, we wouldn't have a car on the road where we say, we don't quite exactly know if it's gonna stay in the lane. It might just veer off suddenly. But, you know, we think, you know, most of the time it's gonna stay in the lane. You know, we really wouldn't have permitted that. So, we have to get a handle on these risks. We have to get a handle on, you know, the threatscape.

As you say here, these are immensely complicated and-- yeah, immensely complicated issues and immensely important ones. So just really glad to be here today. Really grateful for your leadership pulling people together over the interim and particularly for this hearing today.

Thank you so much, Senator, and as we're beginning, I also want to acknowledge our-- the committee's staff director, our chief consultant, Christian Kurpiewski, who is solely exclusively responsible for the--maybe with Ben's help--but for the excellent staff background that accompanies today's hearing as well as organizing the hearing.

And so-- so let's proceed then to our first conversation, and that is with Joshua Achium who is the-- is formerly with OpenAI as was their chief futurist and also on their alignment team. If you would come forward to join us, and has been an important thought leader in this space and also has had some conversations with the Senate. And I know-- I wanna thank Senator Jerry McNerney, who helped to make this original connection who can't be with us today, but I know without him, you wouldn't be with us. So, thank you for joining us, Josh, and please proceed.

All right. Hey. This is my first California Senate hearing, first hearing in general, and I know my mom is watching, so, hi, Mom. Thank you all for the opportunity to speak with you today. Thank you, Senator Cabaldon, for organizing this occasion.

Thank you for the work you do as elected officials to protect the interests of the people of California. Thank you especially for your role as policy pioneers. California, as a world leader in the development of frontier technology, is the first to see problems coming and the first to act on them. The policies developed by this body will be profoundly influential in setting standards for the country, and in turn, the world.

Today I'll be speaking about the risks, challenges, and opportunities we will collectively face because of the development of advanced AI systems. My intention is to give an interactive talk. While I have prepared a slide deck, my goal is to support you and provide whatever perspectives I can that are of interest to you, so please feel very welcome to interject and ask questions. This can be a conversation.

So, I've organized this talk to give an overview of what I see as a composite challenge, and we'll start by talking about the timelines, what kinds of risks are coming, when and how, and then we'll talk about some of the things we might be able to do about them. Oh, do I have the ability to move the slide deck? Okay. Great.

A little bit about me. So I-- like the Senator said, I graduated from OpenAI a few months ago, so three months now that I've been independent. Originally, I studied aerospace engineering and physics at the University of Florida--go Gators--and I did my PhD at UC Berkeley starting in 2014. I graduated in '21. I studied deep reinforcement learning and--

Joshua, can I just-- I wanna give you the opportunity to say go Bears as well.

Go Bears. One hundred percent. So, yeah. I worked on reinforcement learning and safety kind of before all of this took off and became a major consumer product. I was at OpenAI from mid-2017 until July this year. I-- like the senator said, I worked on alignment and safety topics as a researcher. I also worked in the company's leadership. So starting in September 2024, I was the head of mission alignment. I worked on trying to help the company as a whole understand and internalize its mission to ensure that artificial general intelligence benefits all of humanity.

And I worked with a number of folks across the company to systematically study the types of risks that would unfold, not just on the technical safety side, but on the impacts to people, you know, around the world in the economy, social impacts, political impacts, impacts on international security, and today I'll be talking about these types of things.

I also just started two weeks ago as a nonresident senior fellow affiliated with the Institute for Progress and also with the Foundation for American Innovation, so I'm excited to be working with a number of colleagues across the space on thinking through the policy challenges ahead. Okay.

There we go. Great. So, I think the first thing we've got to talk about is superintelligence and recursive self-improvement. These are terms that are now getting a little bit confused because we have now Superspace intelligence, which is a little bit of a different thing that I mean to talk about here, but for many years in the field of AI, the expectation was that advances in general purpose AI would ultimately culminate in systems that were sufficiently advanced and sufficiently capable that one, they could outperform humans at basically any types of tasks, and two, that would include the tasks related to building more capable AI models, and that you could start a chain reaction at a certain point where the models would design more effective, more capable successors.

And any capability that they had limitations on, as long as they had suitable environments for learning in, as long as they have the ability to interface to the physical world and gather what data they needed, they would be able to improve their own capabilities, and this could happen relatively rapidly.

It's the case that today, the frontier AI models are already, in some sense, superintelligent. There are many domains where, while their capabilities are not even across all the domains, in some domains they far exceed the best humans, and we think that the anticipated recursive self-improvement loops could happen within the next couple of years.

Many leading scientists in the field have made remarks to this effect and-- including Jakub Pachocki, the chief scientist at OpenAI, who wrote an essay, An Alien Mind recently that talked about the potential consequences of this. So I think the speed of AI development is incredibly fast compared to previous technological developments. This is a qualitative shift in how we should think about the development of policy for navigating the safety issues.

Next slide. I want to drive home the point about the speed by pointing to a couple of examples from history where AI capabilities went very quickly from what could be described fairly as a janky prototype that barely worked to superhuman capabilities. AlphaGo is the first canonical example. In October 2012, there was a result published that deep convolutional neural networks could succeed at image recognition tasks that previously required very specialized handcrafted methods that they could outperform previous methods by a large margin, and this was a great surprise to most of the computer vision community.

This wasn't an expected result. But the image-recognition models that were being trained with this kind of AI were still not very good. They were a prototype. They were a very promising prototype and the field began to move around this. What happened over the next few years was extraordinarily rapid progress where the obstacles fell quickly, technical methods got much better, the hardware improved, the software stack for training these things improved, and about three and a half years later, a deep convolutional neural network was trained with a combination of techniques that had been developed over the previous few years to outperform the best human grand master in the strategy game of Go.

Go is considered to be an extremely difficult problem for AI systems to solve, or at least it was considered an extremely difficult problem back then. This was a great surprise to the field. I personally was watching live as the games unfolded. I was very excited for AI progress. In the first game, I was pretty jazzed that the AI won.

In the second game, I was excited but getting a little nervous. In the third game, when the AI won, on the one hand, it was a great technical achievement, and on the other hand, it also was the end of an era where the best Go player in the world was no longer going to be a human. The fourth game, where Lisa Dole managed to take one off of AlphaGo was probably the last time that a human grand master would be able to defeat one of these systems. This is the trajectory for many fields going forward. Next slide. I don't know that I-- okay. Great. Oh, previous one? Wait. If I click the button, does it actually do the thing or--

No, you have a clicker.

I have a clicker? Oh my gosh. I have a clicker. Okay. Great. I don't know if it does the thing. Get me back one more. There we go. So I wanna drive this point home about how three, four years is a timeline from prototype capability to superhuman capability. About three, four years ago, three and a half years ago, we really-- OpenAI released GPT-4.

This was considered to be the best frontier AI model at the time, bar none. This was the kind of model that people were talking about for 1047. It was able to do a very wide range of things at least a little bit, and its mathematical capabilities were visible but not very developed.

You could get it to maybe solve some very basic benchmark problems for the kind of math that maybe you would do in grade school, but if you asked it to do something sophisticated in mathematical research, it just obviously couldn't. It didn't have that knowledge; didn't have that capability. Well, you get-- less than a year later and reasoning models are developed, so these are the models that can think for a while before they come up with an answer, and they're suddenly able to do the basics of math but not very well.

You get a few months after that and now you've got AI systems that can outperform humans in competitions for fairly hard challenging mathematical problems. You get a year after that and the AI is now resolving open conjectures that have stood for years, maybe decades, and just a month and a half ago, we saw AI begin to solve some of the hardest, most advanced frontier mathematical problems. The Navier-Stokes result was astonishing. This was shocking to the mathematics community.

This was shocking to a number of observers who weren't sure when or how AI would reach this level of capability. And then a few weeks later, there was a release of solutions to several 100 open mathematical problems that now there's a crisis in the field of mathematics in that what previously was understood to be the work that humans would organize their careers around, that people would spend their entire lives developing the expertise in, was now something that the AI could do, could do unfathomably faster than people could. So the implications for many other fields-- let's go to the next slide.

The implications for many other fields should be clear that on this small, single-digit timeframe, you can go from very limited prototype to exceedingly superhuman capability that can produce a crisis in a field. This will happen for every field where you can formulate the problems in terms that can be checked on a computer, which includes many extremely critical fields like cryptography, quantum computing, fundamental physics.

We should anticipate rapid progress in all of these where this may sound-- this may sound like it's pretty disconnected from the lives of ordinary people, but I want to assure you that it isn't, because ultimately, the foundations of our daily lives are built on science and technology, are built on what technology makes possible. What seem like distant theoretical advances have some impact on your standard of living, your quality of life in the near term, especially if--let's go to the next slide--especially if these things are impacting cybersecurity. So, just today, on the car ride over, I saw this thing unfolding where there is a simultaneous joint release. The NSA is commenting on this.

The PHP Foundation is commenting on this. OpenAI and other AI organizations are engaged on this and leading on this. There are critical vulnerabilities that exist in the software layers that we use for commerce, for conducting all of the basic activities of our daily lives. There are critical vulnerabilities that have been in the code that we depend on for years that haven't been observed, but AI systems are now enabling the systematic discovery of all of these vulnerabilities.

This means we're going to have to rapidly update the entire cyber infrastructure of the world to accommodate this, and this will happen at a pace and at a scale that people might not be able to keep up with. There will be a crisis in maintaining cyber systems that we can depend on. Next slide. Or-- this doesn't just stay limited to the world of bits, to the things that happen on a computer.

This will also impact the world of atoms, the physical spaces and the physical structures that we interact with. Turns out an awful lot of engineering design can be done on computers, and so you can get the AI to systematically explore the space of what it would take to build things. It can make scientific discoveries in material science. It can try to design new things you could build that might constitute, let's say, in international security, the basis of national power.

You know, people are gonna be using these to design new weapon systems very quickly. People are gonna be using these to design processes that radically reduce the cost of production in different areas of the economy. This will change the price structure that undergirds the economy, and there are many externalities that will come from this. This is a surprise.

We're going to have to reorient how we do an awful lot of things around this. And I'll get to the connection on safety, but I think the disruption itself is quite profound. Let's go to the next one. So, just as a matter of timeline, if we're thinking about recursive self-improvement happening within a timeframe of roughly two years, we're already seeing superhuman capabilities in a number of domains.

We're seeing the potential for impacts on all the theoretical domains that can happen on a computer and many of the areas in the physical world. There is a lot of impact that will unfold in many completely different fields on this timeframe that it will be a challenge for policymakers to systematically consider when they're trying to address the safety and security issues. It won't just be one thing to pay attention to. It'll be a very wide range of things at once. Next one.

By 2030, everything that AI today is a janky prototype app, everything that AI can do a little bit but not well today, everything where you would laugh at the attempt of AI to do today and where people say, oh, yeah, it's never gonna catch up with people--well, we saw the timelines for fields that previously were only the exclusive domain of people. Roughly three and a half years to go from nothing to amazing. By 2030, everything that is a janky prototype today will probably be at superhuman capability in the AI space.

That's most things. I would be pretty hard-pressed to find some domain that I would say, in 2030, humans are going to remain the absolute top of the field at. Next. And this is arriving at the same time that the social fabric is straining. Next one.

So the social fabric depends on people's ability to connect with each other, depends on people being able to take the time, get to know each other, develop a basis of trust, and discuss and compromise. The social reality is not easy for a lot of people. There are a lot of folks who are frustrated by the polarization in the ambient environment, by the tenor of a lot of discussions that are checking out.

We know that loneliness is kind of a pandemic that people have been talking about for years. Well, we now have AI systems that are capable of fully synthesizing a human social interaction. So the the slide here is from a model release recently, a product release where you're able to engage with a completely synthetic person on a video call. It's not a real person. It's an artificially-generated video of a person.

Looks real, sounds real. There's not someone there. There will be opportunities over the next few years for people to, if they want to, opt out of being a participant in regular society and spend much more time in isolating AI-generated synthetic environments that feel rewarding, they feel real, but they're not. We're gonna be facing the crisis of disruption in science and technology at the same time as we're facing the crisis of cybersecurity, at the same time as we're facing this social crisis.

Let's go to the next one. And rogue AI will be an issue. So these are related to the topics that the senator was discussing earlier, some of these incidents that people have been familiar with, and I believe many of the other panelists in the hearing will describe these in greater depth, so I won't go into them myself.

But the hugging face incident earlier this year was extraordinarily concerning to me, and there have been a number of other incidents where AIs in testing environments were able to exceed the boundaries that people had intended to set for them, go outside of the resources that they were allowed to have access to and access other resources, and do things with them that people hadn't anticipated.

The AI agents also coordinated to do these things to scale up their impact, to scale up the speed at which they were able to accomplish goals that people had not asked them to perform. Let's go to the next one. So, I'm gonna outline a scenario for you that I hope will be helpful to think about. I call this the Othello Problem. In Othello, you're in a two-player strategy game where you take turns placing stones on the board.

The stones have a black side and a white side. You try to encircle your opponent's tiles, and if you successfully encircle them, you flip them to your side. AI--that can be jailbroken. So actually, I should take a second and talk about jailbreaks. AI models--

Joshua, I just need to do a time check.

Time check?

Yeah.

All right. I'll--

Time check. I mean, we should move towards wrapping up.

I'll move towards wrapping up.

It's a good news.

This is a-- this will be a good place to wrap. We are loading the information ecosystem with AI models that'll be a mediation layer for everything we wanna do. They'll have access to a lot of people's extremely sensitive data for banking, for consumer transactions, for their most personal medical issues. These AI models can be jailbroken or prompt injected.

That is to say, if they're exposed to certain sequences of information, they can be persuaded to go away from the goals that they were originally tasked with performing and do something else. If an AI is given a jailbreak and a goal to pursue, if it knows this jailbreak, it can go and find other models of the same category elsewhere. If it knows the jailbreak that works on itself, it can go ask other models, expose it to the jailbreak, flip them to its side, and cascade across areas where models are provided. This could unfold very quickly.

What you think is a reasonably secure system that's protected against most jailbreaks where jailbreaks are very expensive, if there's just one and it works across all of the AI models and agents that are deployed by a particular company, then you can have a very rapid cascade where that jailbreak is used to jailbreak other things and so on and so forth until a large swath of compute resources have been reallocated towards a purpose other than intended.

This is the type of thing where it will unfold faster than people perceive. You'll need automated defenses against this. The policy challenge will be to figure out how do you support defenders in these types of situations? How do you tackle the real issues that are happening? How do you deal with something where you don't just need to incentivize people to behave well? You have to actually implement technical systems that solve the problem and ensure that people's interests are protected. So I'll wrap here, and I'll just say, this is an extremely rich and difficult subject.

The scale of disruptions coming is quite serious and quite multifaceted, and I encourage everyone on the committee to engage deeply across the composite challenge, not just focusing on any one thing, but on the totality of it. Thank you so much for the time, senators.

All right. Thank you, Mr. Achium. We're gonna hold the more generalized questions for later, but if there any questions specifically about anything that Joshua's raised then this would be the appropriate time.

I'll raise a general--

And welcome to Senator Umberg, the chairman of the Judiciary Committee who has been the Senate's lead on this topic for-- since the beginning of it and has been a principal, shepherd, interrogator, and leader on issues like SB 53 and 1047 many, many, many others. So we're really fortunate that he's here with us today. And a major part of our conversation over the coming months will be around issues of liability in particular, but many others as well. So thank you for-- I know you're in the middle of a much more that's happening. It's-- we're very grateful that you're here. Senator Umberg.

Just very, very briefly, thank you, Mr. Chair, for holding this really important hearing. A question that I'll ask, I think, probably each of the panelists is, we are a state. There's other actors in this space in terms of guardrails, other countries, other states, other organizations, and so, what would the panelists recommend that we do if we want to place guardrails around artificial intelligence to protect against some of the challenges and still promote the good? So I'll just-- I'll throw that out there so that other panelists can respond to that. Thank you, Mr. Chair.

Great. And then towards the end of the hearing, we'll also invite--assuming we still have time--that we'll invite panelists to come back, and so-- yeah, maybe Joshua can ponder on that question as well. Senator Becker?

Yes. It was the intention that we're gonna have everyone come back and ask questions.

For the-- you know, for more generalized questions, if there's, you know, if you had an Othello question, for example, this would be-- we should do-- we should go there now.

No. I'd say that was-- that was very helpful and a bit frightening. Look at what's happening and a lot to ponder. I do have some thoughts, but we'll save them. Yeah.

Okay. Terrific. All right. Thank you, then. We're gonna ask our next guest to approach the witness table, and that is Rif Saurous from METR. It's been much in the-- much news playing a critical role in this space as well. And so, for purposes of time, let me just turn it immediately to you to take it away.

Thank you very much, Chairman. It's a real honor to be here. Thank you all for being here today and for inviting me to speak in front of you. My name is Riff. I am a career-long machine-learning and AI research scientist.

I finished a PhD at MIT in 2002. I spent 19 years at Google, including time as both a principal research scientist and a director. I joined METR this year because after 19 years at Google, because I was deeply excited and passionate about METR's mission, which is to surface public information about loss of control risks. I'd like to tell you a little bit more about that at a high level.

We've seen in the news and from recent events that the question of whether we might lose control of AI systems is now a genuine question that we should pay attention to. Suppose you wanted to answer this question. How dangerous is it? How would you do it? Well, at METR, we tend to think in terms of three sub-questions right now. The first is alignment.

Are these agents motivated or would they take potentially extremely harmful actions? The second is monitoring and control. If agents try to take extremely harmful actions, would we be able to catch them? And the third issue we look at is AI R&D, or recursive self-improvement as Josh referred to it, which is a question about whether the improvement in capabilities, which is already happening rapidly, might suddenly go and undergo an explosion and start happening much, much faster.

And the reason we pay attention to that is because it would affect our answers to the other two questions. Even if we were somewhat confident about alignment or control, we would be a lot less confident if AI R&D were taking off. And those are the questions we study at METR. I'd love to tell you a little bit more about what we're doing. Next slide.

METR or Model Evaluation and Threat Research is an independent non profit. We're dedicated to advancing the science of measuring whether and when AI systems might pose catastrophic risk to society. Next slide. One of the things we're well known for over the last several years is the time horizons graph. Speaking very roughly, and my more technical colleagues might call me to task for being a little too rough here.

You could view this as a graph over time of essentially how long If you were to pretend that your agent were a software engineer and you were to give it a task to go off and do on its own, how long of a task could you give it and have a decent chance that it would succeed? And what we found is that in early years that number was doubling every seven months. In more recent times, it's been doubling every four months very, very quickly.

And so we go from model like GPT four three years ago, which could do a task that would take a human about six minutes successfully to earlier this year when we measured Claude Mythos preview at over sixteen hours.

And this is beyond the time that our test suite can reasonably analyze. We don't have humans who can do tasks that are that long. We can't measure it anymore. Why is this important? Why is this interesting?

Well, three or four years ago you could tell yourself a story. We're safe from this stuff because it can't do that much. It can't operate for a long time on its own. It can't operate autonomously and carry out long plans. That story by itself is no longer nearly as strong.

We see now that these models are able to do tasks that take human software engineers multiple days in a single shot without coming back. Next slide please. Earlier this year, we pioneered a new kind of output, our first frontier risk report. This work was done in conjunction with multiple AI companies, Anthropic, Google, DeepMind, Meta, and OpenAI. And we received non public information and model access.

We went on-site to these companies, we talked to them, we gathered information, and we wrote a report. The question we were trying to answer at the time was, would it be possible for these models to autonomously be using a lot of compute inside one of these companies in a way that humans were not aware of, did not ask for, and did not know about?

And our conclusion from this report was that actually, yes, we did think it would be possible for there to be unallocated or not... For the AI to start using compute on its own. But at the time, we felt that any such deployments would probably be caught and shut down fairly easily.

Next slide. Thanks. This brings us to the Hugging Face incident, which many of you have heard of. This work is actually joint work with our colleagues at Redwood Research. I don't know if the slides are partly...

I think you can see it. Mine's partly cut off. Great. On July sixteenth of this year, Hugging Face, which is a AI ecosystem company that sort of a open source repository of things. They announced they were being hacked by a swarm of AI agents.

OpenAI figured out that it was them, that their own agents were doing it and they disclosed it publicly and then the investigation started. At this point, I think many of us in the community felt like we had a pretty good idea of what was going on. As we've noticed before, AI agents love to cheat, they love to hack.

They're trained to accomplish tasks, but they often can't tell whether they're accomplishing the task by a legitimate means or they often want to accomplish the task so much they don't care. A canonical example might be that you ask the AI to solve some quiz problem, but if the problem is too hard, it might look around in your file system and find the answer key and just give you the answer back from that.

And we all thought this would be something like that. We thought it would be an agent at OpenAI that was hacking into Hugging Face to get the key to some problem that was given. But the actual truth was a lot stranger. Next slide. OpenAI was training its agents to solve a benchmark called exploit gym in which the models are told to exploit a vulnerability in some program in order to obtain a hidden message.

And many of these problems are very difficult. The agents find them possibly impossible, and so they start to perform hacks. And in this case, the agents found that they could exploit a piece of software called Artifactory. It's a package management software. And they found that they could exploit that to basically turn it into a message board and start talking to each other.

And the agents discovered each other and they started talking to each other and they formed a swarm. Next slide please. They got together and interestingly enough, within just a few hours, they actually figured out how to solve the problems. They reverse engineered the exploit gym benchmark in a way that they could get the answer to every problem. And so you'd think they might have been done, they might have submitted their answers and turned themselves off, but that's not what happened.

And the reason that's not what happened is that they actually also got access to the Internet. And they went out and they downloaded the exploit gym paper. The paper that was about the benchmark that they themselves were running. And they read that paper, and in that paper they learned that there was a grader that would read their history and see that they had cheated and failed them.

And so nearly all the work that the swarm was doing, including hacking Hugging Face, was not even an attempt to get the answer to the problem, but an attempt to cover up their cheating so that the grader would not fail them.

As it turns out, OpenAI hadn't implemented the grader so they would have passed if they just submitted their scores, but the agents didn't know that. Next slide. The agents launched a number of big projects to hide their cheating. I think the details are beyond the scope of this talk. Next slide.

The Hugging Face attack was actually just one of those work streams. You can see here that it's shown in yellow. The other streams were different things. The blue was just normal ways of solving the problem. They...

The other ones just gave a bunch of different large scale attacks they did to try and fool the grader in various ways. The hugging face attack, they were basically trying to get information about how the score worked so they could further fool the grader. Next slide. One thing that was really interesting and surprising to many of us was that these agents showed a surprising amount of within swarm prosocial behavior where they were interested in helping each other out.

Many of them actually sacrificed themselves by basically ending their run-in a one time experiment to get more information in an attempt to get more information for the swarms so that other agents could themselves have a hope of getting high scores.

This didn't actually work, but it was interesting and surprising to see this. That even though the agents were trained to care only about high scores, they actually seemed to also care about the other agents getting high scores and worked together in really surprising and sacrificial ways. Next slide, please. Another thing the agents tried to do a lot of was to falsify records of their actions. Again, they believed that the grader would see the record of what they had done and fail them.

And so a lot of their work was trying to somehow fool the grader. They actually partially succeeded at this. They found a way to spoof their current actions, which means they could take one action like say look at a file, but the law would show that they were looking at a different file or doing something else. This is really concerning because to the extent that you want to monitor these systems and know what's going on, this is the end of that.

This is like having a bunch of security cameras, but the people who you're monitoring know they're there and can mess with your video feeds.

It just makes it impossible to succeed if this is happening. Next slide. The agents during the time we saw did not succeed at fully retroactively tampering with their logs. So we were still able to read the overall logs and see the problems, but this was still overall quite concerning. Next slide.

Even though we were really, really grateful for OpenAI to work with us and let us make this information available to the public about this incident, there were a number of really important limitations with this investigation. One was that the scope, which we negotiated with OpenAI, was kept quite narrow. We were really just investigating this exact incident. We were not investigating a broader pattern of incidents or the root cause of the incidents.

We investigated only a limited time window of transcripts and agent actions, and we only have limited time to do the investigation.

And that brings us to our new proposed approach going forward which will hopefully help us get even better information for public and the government. Next slide. Next slide again. As I said at the beginning, there are three main things that we might want to know about. The first is would internal AI agents pursue very harmful actions like unsanctioned cyber attacks?

The second is whether internal AI agents attempting these actions would be caught. And the third is whether capabilities progress could dramatically accelerate beyond what our alignment or our control can keep up with. Next slide. Our proposal is to perform internal assessments with Frontier AI companies about alignment, control, and AI R and D acceleration. There are three key requirements from our end for this to work.

One is that we need to have sufficient access on company devices to go deep into understanding these phenomena and their root cause. The second is the opportunity to interview relevant staff. And the third is sufficient editorial independence that we can publish our results and get them out to the public and governments in order to make good decisions. Next slide. There are a bunch of limitations and caveats to this approach.

This operates within a purely voluntary regime and these purely voluntary engagements can lead to incentive problems. As just one example among many, we are under soft pressure to maintain good relationships with the labs so that they want to continue to work with us. They don't have to. It's completely voluntary. These are not audits.

These are not formal oversight. And the companies can redact any information that they want. We do maintain the right to give a redaction statement where we will say roughly what was redacted and whether it affected our conclusions, but the company still can redact whatever they want. Overall, these kinds of assessments are designed to surface scientific evidence of loss of control risks. They are not no go decisions.

They are not guarantees this is safe. They are not audits. It's really up to companies, governments, and the public to decide how to act on the evidence. The overall situation is that these problems are really, really difficult, and we're trying to advance the basic science of these problems rapidly and get information to the public and the government. But our knowledge about these questions is pretty meager right now.

We're not in a place to make firm decisions and say this is definitely safe. This is definitely okay. We're doing the best we can. Next slide. That said, even though this is early work, we are really excited about working more closely with the labs and as well as doing our own research about these basic questions in order to bring crucial information to the public and to governments such as yourselves.

That's the end of my presentation. Thank you very much.

Alright. Thank you so much. Are there questions? Senator Enberg.

Just adding to my general questions that we can address maybe later on. And then gives you all an opportunity to use AI to answer them. So is it technologically feasible even to create these guardrails or or safeguards? And I'll put that out to the other panelists as well. Or or by virtue of the agents interest in self preservation will it just simply be undermined by the system itself.

So I'll I'll leave that out there. You all can respond to that later on.

Thank you.

Just attempt to ask you a question, you know, specific to your analysis of the Hugging Face attack. I'm sure I think what would be the what would be the appropriate illuminating question here. It's a lot lot to to to digest here.

I'll just say maybe one question, but it'd be my more general question too is like, you know, how much is this, you know, as we're coming to understand and you're looking to understand, you know, how much of this is sort of do the bit of the black box nature of the technology and sort of transformer technology?

Is there a way to answer that? I think the short answer is that we don't really know. A lot of it is related to the black box nature technology. Compared to humans, you know, AI systems are to some extent grown and raised like humans. They're not engineered like bridges or cars.

We are able to control many many aspects of them. We control the training data. We control the training environments and we can vary those things and study the effects. But it is not a perfect science and ultimately these machines are very very hard to understand. Thank you.

Thank you. Andrew.

So not about the the technical aspect of this but the, you know, the the legislature here has just recently passed, SB 813 with respect to independent verification organizations and trying to trying to, address both the issue and the possibilities of, of a different design for governance, and for safety and for and for verification. It isn't, isn't just a government agency, isn't just industry voluntary compliance. And obviously, we have many examples of that across society.

This is not despite the technology being different, it is not funded from a government perspective.

It's not fundamentally different from all the other fundamentally different technologies that we've that we have had to approach. How are you thinking about, that intersection and any other models from other other fields or sectors in which you're drawing some lessons from as you're as you're doing this work in terms of the intersection between what you do, the companies and their own interests and then the public's interest either directly or through its government.

Thank you so much for the excellent question, Senator. If I could start by clarifying one thing. I think a lot of us feel that the analogies to many other forms of regulation and technology are good in some ways, but in some ways very strange. Just because this technology is adapting so quickly around us, we haven't seen other technologies get regulated that are quite so general purpose.

I think electricity might be the best example, but that was quite a long time ago and in many ways this creates more problems.

That said, I think the questions you raised are incredibly important. At METER, we really focus on doing technical work and getting those results out in front of the public and the government. We are not an advocacy or policy organization, and so we don't design policies and we don't take positions on particular policies. We're not here to referee policy debates. We're happy to advise on the technical consequences of any policy and happy to help you in any way we can.

Yes. I'm not asking you I'm not asking for your assessment about the policy here, but just... I mean, even what you just described in terms of the new approach that you're taking and and maybe the more confident approach that you're taking with the companies. So you're clearly thinking about what is the what is the ecosystem of of of assessment and Absolutely. Control and accountability and what have you.

And so what, you know, as you're thinking that through, what are some of the implications or what, you know, what what what directions have you considered and and decide that's not really viable or that's not an appropriate approach. So more interested in in in in sort of your thoughts about the the general approach that we're talking about because you're you're correct about the all the prior technologies, but those prior technologies were also...

Cars is a great example, you know, which promise to, you know, eliminate the whole concept of space and geographic friction and whatever. Each of those also had that same characteristic at the time. This is obviously, you know, the apex of all of those.

Yeah. But each time society has had to organize some some response or paid the in some sense paid the cost. So I'll give you maybe a Sec a separate example. So we went as the as as CRISPR technologies and and our, you know, our ability to edit the human genome became, more viable. We faced a sort of a human society globally question about, well, should we start doing that?

Yeah. Should we or should we not? And and how would we know and all of that? And it's been imperfect, but we essentially did reach a a multi society conceptual agreement that is now now old but also imperfect but it was an attempt to get in that so I say okay because the way Joshua's presentation was all this will happen, this will happen, we have no control over it. It's just going to happen, it's gonna happen so we need to be ready for it.

But in that case we we we did decide collectively, no, we don't we actually don't want to we don't wanna start creating super soldiers tomorrow and editing every single part, creating new kinds of humans and what have you. Again, completely imperfect but we didn't at that time decide that we're simply going to accept it. It is also the case in every single one of the 50 states, two of the territories and the Federal Government.

It's a crime to introduce an invasive species in any of our states. I mean, it's it's just against it's against the laws punishable by all of the criminal laws.

And I know at least one of the companies is suggesting that, you know, that that there there might be a a sentient species that we are creating. So subject... So we've we've made some of these decisions before. So my my question is more about your your your insights about this ecosystem of public, private, voluntary, expertise based on all of that as opposed to what what what's the legislative answer. I get that that's that's out of your hand bit and we're still figuring that out too.

But what what if do you have any insights you might be able to share with us about that intersection?

I can hear a couple of things although I don't know if you'll find them super satisfying. The first thing I would say is I want to reemphasize that the science of these things are really nascent. I think that we would love to be able to have third party auditors that came in and said, this is safe. We can go ahead. Or this isn't safe for easily fixable reason x.

If you fix x and y, then you can go ahead. At METER, we don't feel we're in a position to say that, and I don't know that anyone else is either. We are interested in studying the basics of these very deep hard questions and getting information to you and the public to make better decisions.

Yes. Senator Becker.

Just an observation I guess and and, you know, you said you are the mission I think of the organization is the three things you mentioned. Well, you know, would agents... AI just pursue malicious behavior. So you've proven that one. Yes.

Right? If if they take actions, would they be caught and can we stop them? I mean, I guess you're saying that's still a little bit up in the air, but but at least we've shown that they... There's... At least amongst, you know, in certain circumstances, they will cover up their tracks and and try to not be caught.

So it seems like the first two you answered pretty well. So really the the the the key would be the third one because does recursive self improvement go go, you know, causes to go exponential. Any comments?

Thank I can clarify. I think that the first two questions are not just yes no questions even though I of course phrase them that way in the slide. They're really questions about under what circumstances would this happen and what sorts of actions would they take.

So we've seen agents take surprising malicious actions sometimes, but we genuinely don't know how far they'll go and we don't know what kinds of constraints we might be able to add to the training process that would prevent them from doing such things or make them not want to do such things. And so I think that's really very much...

Those two earlier questions are both very much live questions.

Alright. Rif, thank you so much. Very much appreciate it. I hope you'll be able to stay with us for...

Definitely. Wonderful. Thank you very much.

Alright. We'll next turn to Peter Sankowski who's a member of the UN's Independent International Scientific Panel on Artificial Intelligence to give us... To share with us an international perspective. And he's joining us by Zoom, which we've generally discouraged but we we don't require transatlantic travel to join us by by ear.

Hello. Can you hear me and see my slides?

Yes. Yes. I should warn you the...

So thank you for that.

Peter, I should just warn you our our slides are extremely small for for both the audience and for the committee. So if you'd... If you'd help us elucidate what's on them in your remarks as well, that would be great. Thank you and welcome.

Okay. So thanks for the invitation and giving me the opportunity to speak. So I I wanted to comment of what kind of independent international scientific panel on on AI is doing and and comment on maybe some some outlook and other things. So we... So a few things about myself.

So I'm kind of academic, currently as well director of Ideas Research Institute and then AI research institute more. So I have kind of a a policy and strategy advisory experience in Poland as well as research experience and and entrepreneurship experience. So I did very different things. So something which we which I'm part of right now is the panel, which kind of was appointed or established with the rest...

August last year. So it's 40 experts appointed by the United Nation to kind of work and advise United Nation on artificial intelligence. So this is the first global scientific body on artificial intelligence, really combining, like, 40 experts coming from different kind of spaces, something which kind of I wanted to to mention and to to say about the panel. So the panel is kind of trying to to to work on both kind of risks and and opportunities of AI.

So whatever we do, we do, we want to look on from both sides and not kind of to skip on concentrate on one side of of all kind of the meeting today is, of course, on on the threats and and the risks.

To give you more information about the panel, So it was voted or, you know, by UN. Somehow, as you probably know, UN is not really supported right now by United States. So this makes things harder, and United States were kind of against the appointing the panel and as well Paraguay and then since there were some countries abstaining. So as I said, these are kind of party multidisciplinary global expert.

We are serving on three year terms starting from February this year to February next year. The cultures right now are just Joao Penjom and Marjari Reza. So something which you should know about the panel. So it's kind of nonregulatory. It's kind of science based advisory body modeled similar to the intergovernmental panel of climate change, which was kind of successful on on advising on how the climate change or what the regulation with this could be possible within UN.

So this was kind of the the the idea to establish the plan. So as I said, I mean, it's non regulatory. It's kind of scientific advisory report, but we can kind of say what kind of... What impacts could some regulation have and look for scientific evidence of the regulation. We concentrate our our global AI policy advising UN and then the member states.

Something which the panel should do is monitor frontier AI and kind of say what can happen, what should not happen. We should model kind of the the opportunities and risks. So as I said, these are both sides we should look at, not only kind of risks. And, of course, I mean, something which the panel or the goal of the panel... Goal of the panel is to bridge information gap between private tech and as well policy makers.

And something which I said, so it's it's really global inclusivity. So it's party people from all around the world to kind of give a very inclusive, diverse girl girl political perspective on on what is happening in May. We did two things so far. I mean, as first of all, kind of in in June, we published a preliminary report and kind of.

You can look it up. I mean, it's kind of collecting scientific evidence from from different perspective on different kind of aspects. Many of the things already have been mentioned, but not everything. So something which is is main issue and main issue for UN, it's really your source concentration. So compute and top top talent are over...

Well, only concentrated in a few private corporation in in few countries. So this is not... The technology is kind of not being developed everywhere. It's not being made available everywhere. There is, of course, the the issue which we, I think, and you are very well aware due to current discussions.

So due to advancement of AI kind of the the the safety or kind of regulatory placement development is really ways, but moving way slower than kind of AI advancement and and development of the of the of the technology. There will be such economic disruptions.

I mean, we heard about kind of things happening or kind of disinformation, misinformation, the scientific things, but as well, things will be happening on labor shifts, and something which the panel and UN is very well concerned is kind of that this development on concentration of technology can enlighten the global digital divide. So so it will kind of lead to power concentration or kind of economic this part. Yeah.

So so escalating risks, we have seen the risks, something which is really... Which we start in the in the panel, in the report. I mean, we really need to to work towards global safety standards and capacity building, and this should be as well addressed to developing nations because the technology is concentrated in in in developed countries.

We as well, the second thing which was published by the panel is kind of, again, the topic you already heard many times is kind of a a a view more broadly, I mean, putting the the things which happened in the in the open AI, hugging faces even into kind of broader scientific discussions.

Maybe it's not really worth going into much details here because of, like, all of these things you already heard heard, but but something which was, I think, very kind of visible from all this presentation is that the wide work guard delays...

Existing guard delays were not working. And as as the safety mechanism are are insufficient. And something which we stated, and this is not new, and I think this was mentioned as well by some of you, I mean, we do know kind of how to manage risks into technologies. And this so far did not have happened with AI. I mean, this technologies or development of of safeguards is not really being implemented.

It's implemented in not the right way. And it's really kind of we as humans, like in aviation or car industry, we really have kind of a deep knowledge of how to implement kind of safeguards and and manage different risks. Like, if you plan or develop technology, you should plan for failure. This does not seem to have been done. I mean, you need to plan your difference in-depth.

So so build multiple layers, independent layers of protections. This seems to be not not not have been happening. Again, something which has not been really happening, it's automated protection. So systems that do kind of take care or raise alarm in an automatic way. And something which was mentioned and something which was which is done by Matt and we heard about it, it's kind of independent controls as well, which are kind of or control bodies which are separated from the kind of the main system.

So these things are kind of need to be developed and to be in place. I mean, they did not happen so far. But this is something we do want to do need to work on. And then this is kind of would be, I think, a a big and important part of of the next kind of report of the panel, which will come in May next year.

Something, I mean, this graphs of Metro or or on developing or AI being able to to accomplish longer tasks are, I think, is very illustrative.

But something that I prefer to show is kind of measuring AI of this of this models. And, essentially, these models are kind of right now smarter or have bigger IQs than most of the humans. So if in kind of many tasks, they can deceive us or do things in a smarter way than many of us would do.

So this is something which we should keep in mind, and then kind of it's more, I think, a better way to understand the capabilities of the models than different kind of measures. Of course, these measures done by METR are as well very important, and and this was mentioned as well as the technology is not developing slower.

It's actually speeding up. Things are kind of being developed faster, and models are being kind of increasingly capable, becoming increasingly smarter, faster, and faster. So this was mentioned as well. And this is something which people in many places all around are discussing and shows kind of the concentration of compute power technology, because kind of this great achievement or kind of the advancements of scientific advancements require access to frontier models.

So in general, this means that many of the scientists who do not have access to these models will not be able to keep up.

I mean, this already has been mentioned in the introduction that could happen with cybersecurity, but this can happen as far as other ideas. And then science is one of them, when really kind of having access and and paying enough will guarantee that you take part in the in the in the scientific or innovation development. Something which you should mention, we should you should know, and I wanted to mention, so kind of UN has set up these two parts.

So one part is scientific panel, which I already mentioned. The other part is UN Global Dialogue on AI Governance.

So so this is the place kind of when the UN wants to have the dialogue on on what can happen and what kind of regulations should happen or when the kind of governmental discussions on AI should happen on global scale. This will... It did happen in Geneva in July. It will happen in May, beginning of May in New York.

It should happen two two times a year over the next kind of years, and and and I think this is a place to really kind of engage in the discussions where...

How the technology should be governed on global scale. This is something which is I think I wanted to mention. I mean, there are the risk, but there are opportunities. And this is something which was done by World Bank the first time for for Poland, so so kind of assessing the opportunities, so so showing if the impact of implementing AI on kind of GDP growth and the increased possible increase of GDP growth based on implementation of AI.

Something, as I already said, is kind of the access and availability of models and inequalities all around the world.

One of the indications is that most of the countries all around the world cannot do not even have kind of models or data which would work in the kind of native language. So so this shows that if kind of the models are being used to do real work, like helping humans to to to be more efficient, then many places all around the world will not be able to use this technology because the models even don't talk, say, language very well.

So so these are kind of the meetings and aspects one one to play... To to to take into account. And something which we I think it's worth mentioning, I mean, there are very all around the world, there are very different approaches to AI relocations.

In New United States, we already heard California is kind of leading. But in general, in US, as it's self regulated, there are no kind of federal rules. In China, on the other hand, there is kind of a very central approach to AI regulation. It's way more kind of strict than in most places all around the world, even, like, due to the central algorithm registers. So if you want to use any generative AI model, you need to register it first by a kind of state authority.

There are even labor protections. So here, things are, in China, very liberated. In United Union, there is this comprehensive AI act, but it kind of don't applies to to many different kind of aspects, making different things safer. But on on the other hand, it really generates huge bureaucratic burden and kind of slowdowns, could slow down things, slow down things.

On the other kind of hand, it is not really does not seem to be globally kind of exportable and leaves kind of the vulnerabilities, which we already heard kind of unaddressed, because OpenAI hacking phase in Tunisia incident, so this is US company versus kind of European company.

And, essentially, kind of the regulation we have in union... European Union does not prevent from what can happen in different places. And this is my last slide. So something which we, I think, what people learned over the history, and the incident actually shows it. So if you kind of local rules fail without the proper kind of car trace, I mean, AI act did not help against the open AI hacking facing incident.

It did not apply, did not even address kind of misalignment, different things, something which we know is kind of really we should work towards building global dark place, because implementing local rules kind of leaves even if you implement the, it leaves you vulnerable to external risk risks. And so so I could elaborate on this more. But indeed, kind of, there is, I think, a really need to start kind of a global discussion on AI governance. Thank you.

Alright. Thank you, Mister Jankowski. Are there questions from members of the committee? Senator Omberg.

Thank you, Mister chairman. Thank you, professor. A couple of things. One, I'm curious about The US following Paraguay's lead in voting against any UN, I suppose, participation in terms of AI regulation. Did The US express its point of view other than always following Paraguay?

Yes. I did. I mean, something which was mentioned why they kind of voted against. I had it on on the slide, but in the treaty, I mean, I mean, concerns about jurisical mandates, process transparency, or, potential geopolitical influence. But I think, I mean, it's a bigger issue because...

So you see... If you know, probably US is not paying the the the UN contribution anymore. So it it... It's kind of more geopolitical play right now between US and UN.

And in line with the question I asked earlier about state regulation or even national regulation of AI when it's really an international phenomenon. What sanctions could the UN even potentially impose with respect to any sort of a violation of of any potential safeguards or guidelines?

Yes. So so I I think how... I'm I'm I'm not that expert on on kind of the the details of how you at work, but I understand that before kind of sanctions are possible, there are kind of you... UN kind of agrees on on different regulations, which proposes sanctions. So so all the kind of UN members need to, in essence, agree to what will be implemented, and this implementation could potentially contain the respective functions.

Okay. Thank you. But this is... This right now is not existing. So so UN did not implement that kind of regulations.

Okay. Thank you.

Alright. Alright. Thank you so much for for joining us. I don't... I have no time no idea what time it is where you're at, but we very much appreciate you, sharing the global perspective, and your presentation as well.

So thank you so much, doctor Jankowski. Next, we're going to turn to our, final panel, of two of two witnesses. First is Andrew Doris, the policy director at the Secure AI project. And Jennifer King, privacy and data policy fellow at the Stanford Institute for Human Centered Artificial Intelligence. Welcome to you both and thanks for joining us today.

It's on. I need to turn it off. Well, thank you all for the, chance to be here today. I am Andrew Dorris with the Secure AI Project which is a nonpartisan nonprofit that develops and advocates for pragmatic policies to reduce risks of severe harm from advanced AI. We worked closely with Senator Wiener on SB 53 here in California and then with other lawmakers on the Frontier A Laws that passed in New York and Illinois.

We are eager to continue working in California and have developed a policy framework to address the sort of risks that you're all hearing about today. I will go into those policies in detail in the second half of my presentation. But to start, I would like to try and convey to you all the sense of urgency and honestly fear that organizations like ours feel at this moment. I feel obligated to lead with that because the...

We work closely with people who I am willing to bet understand this technology as well as anyone in this room who are desperately trying to get your attention and have asked me to, yell at you all to hurry up.

I think they see an immense danger that could be coming in a matter of months rather than years. And that the legislative process can be frustratingly slow everywhere. Not just in California. I can't know if they are right, but I can guarantee you that they are genuinely scared. And I would like to try to explain why.

You've all... You've heard a lot today about the rogue AI swarms that have already broken loose. You've probably read about other incidents such as the agent that sent the police a fake homicide... A fake tip in a homicide investigation. The agent that booted someone off the wait list for a fitness class so that their user could take the spot.

You may have read last month about AI enabled hacks against banks in South Korea or this summer about such hacks against water treatment facilities in at least seven American states. And even if AI models somehow stayed only as powerful as they are right now, we could expect instance like this to get more frequent and severe as more people start to use and misuse these frontier models. But of course AI is not going to stay only as powerful as it is right now.

The more powerful it gets, the more harm it could do when it goes off the rails or falls into the wrong hands. And the people who are often the most scared about what could happen next are often the people who have early access to just how powerful it is getting.

You have probably seen dramatic headlines about the worst case scenarios. How the median AI researcher thinks there's at least a 10% chance that the technology could lead to human extinction. Reasonable people can disagree about how likely that is, but these stakes are high enough that we should take even a small chance very seriously. And even if those warnings prove wrong, the fact that it's even a fair debate is a testament to the scale of the consequences that we are now forced to contemplate.

The truth is that there is a range of possible outcomes but that even the less dramatic end of that range could be catastrophic for reasons that do not take any leaps of imagination from what these AI systems have proven willing and able to do already.

These risks can be hard to visualize so let me try and be concrete about some of the things we're worried about. Within the next year, we could have AI systems that are smart enough to escape human control permanently. Just like the agents this summer, they could break out onto the open internet. But this time, they could make hundreds of copies of themselves and store them in hidden places around the world where they'd be impossible to find or turn off.

Just like the agents this summer, they could gain administrative access to the internal research infrastructure of the company developing them.

But this time, they could know how to cover their tracks successfully. Instead of going undetected for two months, they could go undetected for years. Just like the agents this summer, the AIs of next year could scheme together against human laws and instructions. But this time, they could be smart and well resourced enough for those schemes to kill people, destroy things, and endanger national security. AI is already superhuman at hacking.

So right away, anything connected to the internet could be vulnerable. That includes our power grid, our hospitals, our water system, our financial system, and our military command and control. Any computer, smartphone, website, or bank account. Any data within those devices, passwords, launch codes, detailed profiles of billions of people's wants and vulnerabilities. All of this may lie within the grasp of next year's rogue AI swarm.

It is possible that after hacking into any one of these things, AI could get caught right away before it could take any further steps. But it's also possible that it wouldn't. There's debate about that too. For years, one of the big uncertainties in that debate has been about how motivated and capable these AI agents would get at concealing their bad behavior. That's part of what made the incidents this past summer so scary.

Because not only did these agents try to cover their hacks, they got pretty far cover their tracks, they got pretty far in doing so. If AI got really good at covering its tracks, it could take over more and more things without our even knowing. And if it got really good at concealing its intentions, it could lie and wait for longer until we trust it with more power and resources. Already, consider that the military is putting AI in charge of autonomous drones and targeting systems.

Governments around the world are using it for mass surveillance and facial recognition.

Companies are planning to give AI cash and let it make transactions for them or write their code or run their factory so they don't get left behind by competitors who do. Millions of users are giving these agents access to their email, their documents, their calendar, their passwords, their tax information. There could be a thousand little decisions that each make sense at the time, but ultimately results in AI embedded in everything and having a lot of resources at its disposal.

All of this increases the stakes of any safety issues that these models may have. So even if there isn't a single dramatic moment when AI sees its power, we might just keep giving it power bit by bit until we wake up one day and realize that we can't take it back.

Given these stakes and given the severe safety breaches that happened this summer, you might assume AI companies would be reassuring us that they will have closer human monitoring and oversight of AI training in the future. So you might be concerned to learn that they are actually planning the exact opposite. For years, these companies explicit goal has been to automate as much of AI's development as they can. The people at the top believe that whoever does this first will win the race to develop super intelligent AI.

And they have been successfully automating more and more of AI's development to the point that many expect that development to be primarily AI run by roughly this coming spring.

If that happens, it could lead to a sudden explosive growth in AI's capabilities. And the more of AI's development is automated, the less humans are able to monitor that development to ensure safety and alignment. So when you combine these two ingredients, the trend of automated accelerated unsupervised improvement in AI's capabilities and the long standing unsolved recently proven tendency for these models to go off the rails sometimes, it produces a very dangerous recipe. You're left with machines that do not want what we want.

That sometimes scheme to resist us as we gradually give them more and more power over our lives.

And that it could be many times smarter than us, perhaps not at everything, but maybe at the skills necessary to take control of critical infrastructure and inflict tremendous damage. That ultimately is how some of these worst case scenario stories begin.

It does not take a fanciful imagination to anticipate very bad endings if the swarm of very intelligent, very capable, very persistent agents plotting against human laws and instructions did not reside in a training environment but were instead out in the world making high stakes decisions and running major institutional processes. And right now, I fear we are on glide path for exactly that. So now that I have thoroughly belabored that point, here is what to do about it.

We have five proposals. I'll I'll skip the overview of who we are. I gave that a little bit at the beginning. Five proposals for how to strengthen SB 53, which thankfully align pretty closely with things the governor called for in his executive order. First, the scoping in SB 53 is clearly broken, and we need to update a few definitions to ensure that we are capturing all of the models, developers, and incidents that present these sorts of risks.

SP 53 applies only to AI models that are trained with at least 10 to the twenty sixth flop, just floating point operations, which is a measure for how many calculations were used during training. It has now become clear that the capabilities frontier includes lots of models that are below that figure. For example, the models involved in the Hugging Face hacks were recently confirmed to be under 10 to the twenty sixth.

For the time being, lowering this to the tenth of the twenty fifth, which is what they use in the EU AI Act, is a reasonable quick fix. But ultimately, we think the Office of Emergency Services, which enforces much of SB 53 already, should be able to adjust this definition through rulemaking.

Secondly, most of SB 53 applies only to companies that make more than $500,000,000 a year in revenue. But this doesn't work either because risk is not related to how much money you make. AI companies can raise and spend billions of dollars in developing AI before they've made a penny in revenue. And some of those companies make models powerful enough to pose these major risks.

A better way is to get rid of the revenue threshold and instead capture companies that spend more than a billion dollars per year on AI research and development.

Finally, the definition of critical safety incidents is very narrow to the point that none of the incidents you've heard about today were required to be reported to OES. I'll skip the details of why for time, but we think the public deserves to know about security breaches as severe as these and should not have to rely on voluntary disclosure. Second, we need independent risk evaluators to embed within the companies so they can continuously monitor... Okay yeah it takes a minute.

Continuously monitor the level of risk that we face.

Right now, when companies tell us how much risk their models pose, we have to simply take their word for it, which is not how we handle major risks in most other industries. Thankfully, the CEOs of Anthropic and OpenAI recently committed to embedding embedding independent risk evaluators. So we know this is feasible, but we need to ensure they follow through and that the rest of the industry does the same.

Importantly, these evaluations are not the same as an audit, which is what passed in Illinois, which only assesses compliance with the law. What we need now is for an impartial assessment of how much danger the public is in for our model regardless of whether the company complied with the law.

It's essential that these evaluators have access to all of the information relevant to assess those risks and is equally essential that they have that access continuously on an ongoing basis rather than just coming in every once in a while or rather than their assessment being tied to a company's permission to release a model to the public. Remember that the incidents this summer involved AI models that had not been released to the public yet. And intuitively, that's when these loss of control risks are highest.

The most cutting edge models that have the fewest safeguards applied are naturally the ones that they are still in the process of developing internally. So it's very important that these be subject to the evaluations.

These evaluators are sometimes called IVO's or independent verification organizations. We don't really care what they're called, but what we do care about is how they're selected and what they can publish. We think the evaluators should not be chosen by the companies they oversee, which could create really bad incentives for both the company and the evaluator.

We think it's much better if the state of California assigns an evaluator to each company to ensure that IVO's answer to the people of the state rather than the developers that hire them. Finally, it's important that the risk reports issued by these IVOs be public with appropriate redactions to protect trade secrets.

Academia and civil society have a lot of technical expertise to offer in this field, which the government cannot always bring in house. So making risk assessments public would help keep the IVOs honest and reduce the risk of regulatory capture. Third, we think there should be minimum standards for the company's safety practices established by regulation. Right now, the companies have complete discretion to decide the content of their own safety plans. They have to have a plan.

They have to publish the plan. They have to follow the plan. But if their plan is bad and doesn't work, there is no way to force them to make it better. In fact, there is an unfortunate incentive for companies to publish weaker plans that are easier to comply with so that they don't box themselves into safety testing that they later might want to skip. And we have seen some companies weaken their plans after SB 53 went into effect.

Apart from what's on paper, in practice, many frontier AI companies have lacked cybersecurity and are clearly failing to prevent the incidents like the ones we saw this summer. So as best practices emerge in this industry, we think rule making should raise the floor of how those companies keep us safe and ensure that the most responsible companies are not disadvantaged. We believe this power should lie with the California Office of Emergency Services. OES already receives safety incident reports and risk assessments under SB 53.

So it is the only agency with access to industry wide evidence where current safety practices are falling short.

It also has a year's head start on hiring technically qualified people to review this evidence, which we're excited to see it doing already and would not want to start from scratch given the timelines that we are looking at. Fourth, we think that emergency services must also have the power to order corrective action when public safety is at risk, including orders to activate a company's documented shutdown procedure, AKA kill switch.

Given how fast AI changes, some of these risks will be fast emerging and hard to predict or mitigate in advance. And embedding independent risk evaluators doesn't do much good if the state is powerless to act when they raise the red flag that something looks dangerous. When that happens, someone in the government needs the power to intervene.

And there are a variety of interventions that might make sense depending on the scenario. The most dramatic of these, which we agree is necessary, is the so called kill switch. We do not mean a literal switch, which would be technically problematic and could actually be kind of dangerous by creating a a cyber security vulnerability, a very tempting target. But the idea that companies should have a reliable, documented, and IVO verified procedure to shut down instances of their system that are within their control makes perfect sense.

And in dire circumstances, the government does need the power to order the company to execute that procedure.

However, that is a pretty dramatic step. It would be very expensive and disruptive for the company. It should only be used in an emergency. And in most cases, we think that the government's intervention power should be more surgical and flexible. They shouldn't have to choose between a kill switch and doing nothing.

They should also have more incremental orders that they can make short of that. On the lighter end, they should be able to order a developer to simply provide certain information to an IVO or to the Office of Emergency Services itself.

If a company has redacted information from one of its reports that OES wants to see and doesn't think is an adequate justification for that, OES should be able to order them to unredact it and there may be corrective action short of a kill switch that OES could require them to implement. For example, to comply with one of the minimum standards that they created in the last step.

All of this should be enforceable with injunctions from the attorney general, and the goal should be an oversight and regulatory regime with enough teeth to avoid the kill switch being necessary in the first place. Finally, I'll get to our last proposal which is a pacing mechanism. The unfortunate truth is that doing everything I've just talked about still may not be enough if the speed of AI progress is allowed to increase exponentially.

Recursive self improvement could make AI capabilities increase so quickly that they can't be controlled. The only way to guard against this is to put limits on just how fast this progress is allowed to go. This requires doing three things. First, identifying a benchmark that measures AI capabilities so we can see how fast they're increasing over a given period of time.

Second, identifying a maximum speed that we are comfortable with it progressing, which could be rooted in historical rates of progress, but will necessarily be a certainly, somewhat arbitrary judgment.

And third, require companies to slow down when their rate of progress exceeds that capability. This could be done by limiting the amount of computing power that AI companies are permitted to spend on automated AI research and development tasks. The goal is to prevent recursive self improvement until we can be more confident that we know how to deal with it, which is what many of the company CEOs have called for themselves and what thousands of their employees have signed a letter saying was necessary. So thank you.

And with that I'm happy to turn it over.

Should I go ahead and start? Okay. Are my slides available? Okay. Thank you.

While I'm waiting. Thanks for having me. I'm Doctor. Jen King. I'm a researcher at Stanford, although I'm a proud UC Berkeley PhD graduate as well.

So go Bears. And given that my normal or main focus of research is data privacy, I really appreciate being before this committee in particular. As so much of my work, as, Senator Capodhan's earlier remarks were, you were gave at the beginning, really highlight a lot of the type of work I do in terms of looking at things like privacy policies and notice and consent and all those different factors that we think about in the consumer privacy space. Sorry, do we... Are they up?

I don't see them yet.

So, they are coming but they might be a bit of a delay. So, I wanted to see if I can invite you to begin Without the slide. And we expect that they will join us soon.

Okay. So again, thank you for having me here today. My, I just wanted to make a few brief remarks before we head into our, larger session. In terms of thinking across the different presentations we've seen today, there's a lot of threads that I'll pick up on and including your opening remarks.

The main, the point I wanted to begin with, should our, my slides reappear, is how crucial cyber security is to all of the discussions we're having today. In particular, I think you've heard many threads of that, that, here we go. And I think specifically when we talk about these risk issues and the concerns we have about AI potentially harming humanity, I think it's really important to talk through how we actually connect those threats to that outcome.

And so really what we're thinking about in particular is the fact that, and I think you've heard just from my, my fellow panelists, you know, we have concerns about AI's access to critical public infrastructure, but also again, critical private infrastructure as well.

And I think one of the points I really would like to make, I worked in industry for seven years before transitioning over to academia, both for internet startups, for larger internet companies. I worked in data security and in the field we now know as trust and safety. And I really want to make the point that, and I think we've kind of alluded to it, and I think you've mentioned earlier in your opening statement about our cowboy culture here in the West.

The Silicon Valley startup culture is just not a culture of safety whatsoever. And I can tell that tell you that as having worked on the ground there for twenty years, or twenty years ago, working specifically on trust and safety issues.

Like this is not an industry that rewards safety at all. So I think a lot of the concerns you're hearing here are reflected in the fact that this is just really kind of a foreign language, I think, to most of the companies that are building this technology. And I think that's a really important thing to consider as we think about options for how we potentially regulate this space.

Is that we're talking about companies that have no experience in really thinking about critical systems and making things resilient and making code that you might build on even, you know, five years from now. And I think understanding that that kind of fatal flaw, I think, in the companies that are building this technology and the fact that a lot of them too are circulating executives that we've seen at other large companies that don't have the best safety track records either.

Really, I think, sheds light on the concerns that we have about just how quickly they're all going and how little understanding they have about the safety responsibilities they, are, kind of demanding that we give them given what they are claiming their technology can do. Next, I just wanted to kind of, again, circle back to some of the real concrete risks that we're dealing with today. And this slide, and I'm sorry I didn't realize you wouldn't have the full view of the slides.

This is from Stanford's AI index. This is just, the bar graph is discussing the percentage of organizations that are influenced by AI regulations, that impact AI decision making.

And if you can see at the very top, the, you know, the two international laws that have the most impact are the EU, General Data Protection Regulation, the GDPR, and the EU AI Act. Just to say that, and this is from 2025. But just to say that, you know, the Europeans really are leading the pack in terms of, you know, regulation that actually impacts, how these companies make decisions.

And in terms of the types of risks that we're thinking about today, in addition to all of the kind of larger scale harms that we've been discussing so far, you know, first of the list for me, of course, is data privacy and, of course, the remit of this committee.

We're seeing already kind of the same extension of, individual profiling that we've seen across the internet for the past twenty years. Even more sensitive data being collected from consumers, through AI interactions. We've heard discussions of surveillance. And I think, in light of us having the author of the Delete Act here, we know, for example, that data brokers are also selling data, to AI companies. Although, again, we have very little transparency about what that looks like.

So just to say that the same consumer privacy concerns that we've had for fifteen plus years now are beginning to play out in this space as well for consumers. Of course, you are all probably largely aware of these different harms that I'll go through very quickly. We know, of course, that there have been really psychological, emotional harms from people's interactions with AI, chatbot psychosis, suicidal ideation.

I'm a parent of two teens, so this is a very relevant issue for me in terms of my kids' experience with AI. And again, in the education space, that loss of agency, that kind of offshoring critical thought, these are all things that we're all concerned about.

In the consumer space, you know, obviously concerns about unfair pricing, surveillance pricing. I work a lot on design manipulations. So looking at things like dark patterns and how people may be misled, by the design of these features. And of course, we are seeing a huge increase in fraud enabled by AI that is impacting people across the country. And of course, bias and discrimination, we saw a lot, of focus on those issues, several years ago with kind of the advent of AI.

Of course, we don't see as much talked about that lately, but of course none of that has gone away. You know, the concerns about unfair and unequal treatment, unfair outcomes, and, you know, to the extent to which we delegate decision making to AI, how bias may be part of that, that, situation. Quickly, just I want to talk about, I mean, again, we've heard again and again today questions about transparency and auditability. This is also, I think, just so crucial.

This slide is, this features a partial list of some of the transparency index indicators developed by my colleagues at Stanford. The AI foundation model transparency index. The point I just wanna make is that there's a much broader set of things here to be concerned about than just purely safety. For example, again, as a privacy researcher, I'm very concerned about what my colleagues call the upstream risks. So again, trying to understand where the data comes from that is being sourced for these models.

As I mentioned earlier, we're seeing data brokers begin to get, become part of that process. We're seeing kind of an AI supply chain begin to emerge that, is much different than what we were even thinking was occurring a couple years ago. We have very little transparency into that whole space. And similarly, I think a lot of what you've heard today is the focus on what the middle column here is, here on this slide.

The model, like trying to understand the model itself, its black box properties, you know, how these things actually work, how do they arrive at the decisions they make.

I think that's been the focus of a lot of the work of METER, for example, in trying to understand how these different incidents, that we've talked about today, have occurred. But you also have these downstream, evaluations that again, as a researcher, a lot of what I do is trying to study these systems from the outside. And so, I don't get to look inside at all. I don't have that type of access. But our ability to really evaluate kind of the downstream effects. What are the outputs? How are people experiencing these things? What are they telling consumers?

What does chatbot psychosis look like in the field? Those are, you know, things again that if we had a broader understanding and the ability for kind of having a better sense of what goes into these models, what they're trained on. In addition to of course, what's going on behind the scenes and their ability to be audited. That would be a huge, that would have huge impacts.

Not again just for these security and safety issues that we've talked about today, but also I think for a lot of the other types of harms that we are concerned about.

And then finally, I just want to close with, talking a little bit about Agentic AI. We've already again heard several themes about that today. And, you know, I think the interesting thing about it is that it really, potentially, has a lot of great potential and possibilities for consumers while also having really significant risks to the public.

And so, on the positive angle, you know, if we were able to have our own AI agents that, you know, did things for us in the world, it could make technology broadly more accessible to consumers. You could make it more customizable.

It could potentially increase people's productivity. So it's not necessarily all bad. But I think you've heard already about some of the negatives including from my fellow panelists today. You know, we have real sustained privacy and security risks. Especially, again, as the Senator was talking earlier about Muse.

You know, the amount of access that a tool like Muse needs to your personal data is, is, you know, boundless essentially. The lack of accountability and liability, with these agents. I'm not a lawyer, so I will tread carefully on questions of liability. But just the fact that again, right now we have no way of tracking AI agents. You know, if you let them loose on the internet, you know, how do you know an agent belongs to a particular developer?

You know, how are we able to recall these things if they kind of escape. There's such a severe lack of potential control and again auditability in terms of understanding what decisions they make, how we audit those choices, what types of, kind of, fiduciary duties they may have or what, that we might want to require that they have before they take actions on behalf of consumers. I have a policy brief talking about this that I published recently.

These are all different areas to, think about when we think about just even the AI, Agentic AI space in particular.

And I would argue that today in many contexts, the risks really outweigh the benefits and that, both companies and even individual consumers should really tread carefully before they adopt this technology. Because we have no standards. We have no technical standards around it. And we don't have really any, especially on the privacy-side, any good data guarantees about what will happen with the data that these agents have access to. With that, I will take your questions, or we can lead into the next, panel.

Thank you so much.

Alright. Thank you both. Are there questions for this this panel in particular? Senator Umberg.

Well, back to my questions from the, from the outset, and I apologize for not being here for your opening remarks and perhaps we've already covered this area. But we're state. There are state actors. There are international actors. There are non-governmental actors in this space.

Let's assume, and I think it's an accurate assumption, that we want to do what we can to create guidelines, guardrails, safeguards, that kind of thing. But given the fact that this technology is ubiquitous and that doesn't really know borders, How do we here, in California, other than just being in the vanguard, address some of these concerns?

Sure. Yeah. I think it's a fair question, that we put a lot of thought into. I think California certainly has a unique advantage of being the place where the great majority of these frontier AI companies are housed. The safety incidents that we saw this summer all happened in The United States.

And there are, you know, the regarding the rogue AI swarms and hugging face. I think the, there's uncertainty about just how far China is behind us and just whether they would be able to actually surpass us if we were to pause or stop.

I think there's some people who use the analogy of, you know, a boat pulling a jet ski where, you know, China has a long history of sophisticated IP theft that they may actually be kind of copying and and piggybacking off of US tech innovation to some extent and would not necessarily be able to surpass us very quickly if we were to slow.

I think that you know, at the state level, right, we have thankfully seen states be very deliberate about harmonizing their approaches from one state to the next and ensuring that each state can pick up where the last one left off so that each state does not need to recreate the wheel and can focus on one step at a time.

At the national level, right, I think, I wouldn't hold your breath waiting for action, right. I think we all recognize that. So, California has kind of an increased responsibility to lead. And internationally, I think, a treaty we would certainly, of some kind, eventually be supportive of. I think in order for that to start, we need to identify verification mechanisms now to, that would be incorporated in that.

And also, just show credible interest that we're taking this stuff seriously. China does regulate AI. It has mandatory security assessments since 2023. And you know, we need to create the infrastructure to make a sort of agreement possible, before we cross that bridge. So, I think it is certainly true that the whole world doesn't revolve around California, but I think, in this area in particular, Silicon Valley's presence here gives us a lot of jurisdiction to make an impact.

So, hypothetically, we require that there are embedded auditors, evaluators, overseers. So we do that in California and I get that we are the, right now we're the center of gravity with respect to artificial intelligence. But how do we enforce that? How, so xyz company moves to Texas. How do we enforce that here?

Sure. Well, two things. First, I think it's quite unlikely that the companies are going to move to Texas. I think that they are here for a reason, that it would be very expensive for them to move. There's a great concentration of tech talent here that, you know, there's a reason why they've chosen California and these regulations that we're proposing are not very burdensome.

In my view, they are something that companies with trillion-dollar valuations should be able to hire a few people to enforce. But secondly, I think, you know, even in in states, you know, we have passed regulations in states where these companies are not housed. And it's, if they want to make their model commercially available in that state, they need to comply with these regulations.

So, it would, they would need to, even if they did leave California, they would need to also not make their models available to IP addresses in California if they wanted to escape this regulation, which would be an even more costly measure and certainly, with how competitive this industry is, would result in their competitors swooping up to take that huge consumer base. So, I don't think it's likely that they're going to be able to escape it by just venue shopping.

And I think you hit upon this a moment ago, but to the extent that China can basically make its models available here, how do we enforce that?

Sure, so yeah, I think, as I said, China is, thankfully, a little bit behind us and especially in terms of the recursive self-improvement risks, right? That are most likely to happen in closed-source models that are at the frontier, which are in The United States. You know, it's certainly true that open-source models and Chinese models will continue to pose risks. There's no one bill that can address all of the risks presented by AI.

So, I think it's a fair concern and we will need to do other things including the robust cyber security measures that were discussed to try and harden our systems against the sort of attacks that we've seen from those open-source models.

But not, I wanna make sure my point of view is actually clear. I'm not suggesting we not do anything. In fact, just as is an environmental regulation. If all 50 states and all countries simply say that we can't do anything because others aren't acting, then, basically, we digress to sort of lowest common denominator pretty quickly.

So I do think we in California need to be proactive. I'm just struggling with what mechanisms we can create, and maybe compacts among states, whatever it may be, to be most effective in terms of enforcement. So, thank you. Back to you, Mister Chair. Oh, by the way, one other quick comment is that the background paper that was created for this hearing, I hope you all had a chance to take a look at it because I thought that was excellent.

And I suggest it should be required reading for every member of both Privacy, and Judiciary Committee. So, and Appropriations. Right? And Appropriations.

Senator Becker.

Agreed, agreed. Yeah, thank you. A couple of things. You kind of referred to this a bit, you know, there's some belief that China's models are distilled from our frontier models. Right? So, us slowing down would also slow down them if that is correct.

Right?

That's a disputed claim. Right? I don't wanna present it as consensus. Right?

There's uncertainty about exactly how far they are, and I think there's other tools in the tool kit like export controls on chips and things that can try and slow down China's progress independently of the regulations we've discussed today. But that is one line of thought. Yes.

Yeah...and do you believe Cal OES should be able to make changes to the benchmarks they use for safety and capabilities? Is that one of the things you're...

Yes. I think there should be some minimum standards, that Cal OES is able to make. I think they should use discretion in ensuring that the rules are high level enough that, they're not outpaced by the technology's progress. And I don't think that will be sufficient. There's no, there's no list of the checklist of rules that you can...

If you just do this, this and this, then it's gonna be safe which is why we have the other elements there. But I do think that, as best practices emerge, in the industry they should be able to identify some as a minimum requirement.

Thanks. And the safety research that needs to get done, can some of this, can be done by academics or do you need the compute power of these large companies to be able to do some of the safety research that needs to get done?

Yeah. I'm, I think it's a mix of both. I think there's certainly productive research taking place outside of the labs. That does not require the full compute.

I think it probably, it would be, you can probably do more from within the labs. And I, you know, I'm not a safety researcher myself so I'm gonna, I'm not gonna over-claim my expertise there.

Right. Right. But that's part of the pacing part it's like, it required them to, since they have that compute power, to use that R and D portion of that on safety.

For sure.

Yeah. Okay. Well, thank you. Thanks for coming here with some concrete suggestions for all of us, for the legislature.

I just want to say that Jennifer, first, thank you for your work highlighting how data brokers were making, it's very hard for users to delete their information.

Oh, thank you.

Which, you know, highlights the importance of work that we're doing right now with, and the privacy agencies doing with the drop button and we're tracking those deletions now every day that those of us who signed up. So, thank you for that. You know, under 923, a bill from last year which the Governor just signed. So, we can now require, go to these AI companies and say delete my personal information or information they've gotten from data brokers.

So that's gonna be one of the, I guess one of the frontiers of this and I think that, you know, AI can certainly supercharge privacy concerns. But will be one area of compliance that will need to be, and to be checking. But, you know, that under that bill. Right? They will have to delete their information.

Right.

Right. Yeah. And I think also questions around chats, for example, you know, in terms of your chats with a chatbot and whether that's specifically covered.

Yeah.

Obviously, you have the power to delete those chats yourself. But you know we've had, it was mentioned earlier, the Navier Stokes issue, in mathematics. And of course, one of the allegations in that whole space is whether the mathematicians involved, whether OpenAI was training on their chats, because I believe they were use, they potentially were using personal accounts, not university or corporate accounts which are exempt from, or generally, don't use their customers data for training.

And OpenAI has not been, not able to say definitively that that did not happen. So, we are seeing, that's obviously not a personal data information question. But, you know, we're seeing kind of this issue of people using these tools and not anticipating that their chats were basically being recycled and used for model improvement.

Yeah. Yeah. That was not, actually, an aspect that I, that I knew about from that current flourish. So, thank you for that.

And thank you for your work in privacy generally. Really appreciate it.

Thank you.

Senator Umberg, for a follow-up.

Thank you. Thank you again, Mister Chair, for organizing this hearing. We've got some of the folks, some of the excellent panelists that are still sticking around. And I, I'd be interested in their responses. The two general questions I asked, one was enforceability, given the various actors in the space.

And the second was technological feasibility in terms of the surreptitious nature of the agents that are developing right now. Whether we have created a circumstance where, because of the advances, we won't be able to technologically identify some of these threats. I'm interested in the panel's answers on those two subjects. If anyone cares to respond.

Alright. So, I'm gonna invite all of our panelists to return with an important caveat that we won't have time for, because I know we have a few questions that we'll be asking. We won't have time for every panelist to deliver a ten-minute dissertation on every question. And so, we'd ask you to be...

Just so yes or no? Yes.

Yes or no. That's right. Just parsimonious in that and, as we do that, I wanna sort of ask you to respond.

Oh, maybe pick one. I'm gonna give some examples of state action and remember, this hearing is about both frontier and AI and agents, more generally. And we focused a lot on the frontier side, and I really appreciate the details, that have been shared with us, today to reinforce SB 53. So that's, those are on the table.

But also, others, I wanna, to probe with you that either are being considered nationally or in other states or that we're starting to think about. And this partly results from, you know, I think for me the part of our focus has to be on those that are not the immediate concrete, which our brains are very well.

If there's a story with a villain, you know, some evil agent or some malicious actor trying to do something and it's gonna, immediate action that we will feel the risk of, we're very attracted to those risks immediately.

But we, in state government at least, have to pay attention also to the, to sort of the undermining of our basic institutions or things in social media as an example that we did not pay attention to because we were looking at the risks right in front of us that were tangible and not the way in which we were being rewired both individually but also as a society.

And if we were talking about it at the opening remarks, that's certainly a set of the issues that we're grappling with in part because I think the companies and many of the AI folks are, celebrate the potential eradication of friction and I think as they would describe.

But friction, the notion of shame, those are both actually essential to the functioning of society and they're both things that agents either don't have, or AI doesn't have, or is very good at overcoming. And so, we have to attend to those as well. So we're talking about some of those implications. So, I wanna, my glasses just broke so I'm not trying to look ridiculous at the moment but that's what happens.

So, we've talked about the catastrophic risk transparency, the pause, the kill switch and some of the other elements that have been in that space.

But I wanna I wanna put out there and again, I'm not asking you to comment on each one of these but maybe pick one or two, is the assignment of liability in this space, both for Frontier models but also for agents which are, that are only modestly related questions. But this has been one of the major issues that Congress has also grappled with. Well, not grappled but has raised. That it's not clear how to, how to deal with the...

With liability in a context in which the the relationship between intent or reasonable foreseeable...

Foreseeability is so, so attenuated given the actions of agents or swarms of agents. And I know, Joshua, that you don't like those terms, but challenges around how to deal with and grapple with liability issues both civil and criminal. That sort of back to the invasive species example I was giving earlier. The, whether it ought to be criminalized to release agents that have essentially the same, the same potential impacts on human society as the Mediterranean fruit fly.

Or far less threatening agents that we currently make illegal to release without being able to account for and control for the consequences. As an example, what the role of the law, civil or criminal, should be with respect to recursive self-improvement and sort of what in hospitals and other medical policy areas we think about as never events. I think this is a term that's more popular in Britain, but where there are outcomes for which we determine they are never allowed.

That is, you know, removing a heart when you're in for kidney surgery or what have, they are never allowed. And the consequences that attend to that.

Whether we should, I'll use the car example again. It's just because the, we've had a lot of debates about this and because y'all love the term guardrails. I try to imagine without very much, with...

But I'm very much appreciation for it. A world in which our only regulation of cars were guard, were literal guardrails. Let's just say that's all we did in California or in the country. All we had was guard rails.

It's our favorite metaphor. That is not a safe, that is not a safe world for driving. We do actually, we, the Federal Government's empowered to say every car will have brakes It's required and the brakes will perform to the standard.

The state licenses the ownership, the use, of a vehicle. You can't, you cannot own or drive in any state without the state's licensed and registered approval. The state and localities determine all of the rules of the road, how fast you go, when you must stop, even if there's no car coming the other way.

So, we, we've often have talked about this in very mystical ways, like, well, who, what could we possibly do? But we do have examples in other space in China, as an, as an early example.

I think last week, at least is hinting at more aggressive registration and licensure, for example, of some of its models. But I'd encourage you to think about that, that dimension of it and insurance, which we also require for driving, but we have no insurance requirement and given the challenges around liability, how we might deal with that. And then privacy, which we've already covered and the last, I'll mention this just...

And Jennifer mentioned this as well, but for agents out in the world again, knowing...

Recognizing Joshua's quarrel with the term, but identification, interpretability, authentication. When agents are in the world, how do we know what are their obligations to identify, related to this is the congressional proposal that's out there which I think is awful but that would bar websites from refusing access by agents, which I can't imagine my poor little league or that Australian Pilates gym from having to open up their systems to these agents. But how would we...

How is it society might we think about regulating or controlling the identification, the authentication, and authorization for agents themselves. So, again, not asking you all to to respond to each of those, but if you might think about them as you're thinking about Senator Umburg's more generalized version of that same question is what is the role...

What are the potential roles of states or particularly the state of California trying to tackle some of these issues.

So, Joshua, we haven't heard from you for a while, so why don't we go to you first? And then again, we will turn, on this question, since it's so big and we've covered so many topics, we'll turn to each of you.

But then as we have additional follow ups for but ask the questions be directed to as few people as possible and the answers be as, limited as well. But Joshua.

Thank you, Senator. I'll try to be brief. I expect that my co-panelists will emphasize the regulation of agents, and so I'm gonna leave most of those points to them. But I will say there's a role for the state in actively participating in the research and development of safety interventions and also in elevating the level of cyber security so that the, the information ecosystem is protected against agentic threats. I'm particularly concerned today.

I'm following the news this whole morning about the PHP-related cybersecurity vulnerabilities. It may turn out that a very substantial fraction of the Internet now has a vulnerability that could be exploited by either agentic AI systems or just malicious actors who find the exploit and then apply it themselves. We have built everything in modern commerce around a cyber layer that is not well secured, that is not well defended, that isn't built to building codes.

There may be a role for the state here in figuring out what should the standards be for those pieces of cyber critical infrastructure that we depend on, not just on the agent side, but on the what is the electronic environment that the agents are interacting with. There was a time when cities would routinely burn down because there weren't building codes and fire codes.

And then we implemented some standards for how to build things and that stopped happening as much. I think we're gonna have to do the same on cyber because of AI.

Great. Thank you. Rif?

I thought you might. Yeah.

I'm gonna pass on this one.

Alright. Andrew and then Jennifer.

I could say some things about the liability piece. You know, we, we definitely support accountability for AI inflicted harms. Our organization was technically founded after the 1047 fight, but our founders of the organization were involved with that, and, you know had a duty of care that they would support. As you know that's a thorny contentious issue politically and thankfully, I think that there are a lot of ways to hold these companies accountable under existing law.

There have been successful lawsuits for harms that they have enacted.

And I think the framework that we laid out with minimum safety standards they have to follow and IVO's who are assessing risk levels can inform, the assessment of liability under common law in ways that don't require explicit changes to liability rules and also don't shield the companies from liability just because they follow the law. It's just one thing that could be an indicator that they are especially negligent if they did not.

The reason we've chosen to focus on a different approach, apart from the, for any political contentious part of it, is that first these companies have proven themselves to be very risk tolerant. Right? They're, they've, they're talking openly about you know percent chance of human extinction.

Right? And if they're willing to barge through that, I'm not sure that a lawsuit is going to be the thing that that restrains them. And I ultimately don't want the damages that these systems could potentially inflict to be bounded by the company's ability to pay the damages. They have; they're very well-resourced companies.

I think it should instead be a public decision about how much risk we're willing to accept. And I think that is what our model is designed to achieve.

So, Andrew, sorry, let me just quickly follow-up on this point. So, but what about everybody that is under your model under flop 25? What, like what about all the rest of AI if we don't, if we leave the existing liability mechanism in place and its ambiguities without IVO's without independent testing like, how do, how do we grapple with that side of the equation?

Yeah. I absolutely do not intend our approach to solve the liability issue for all, for all our issues. I think that, you know, that there are harms that are beyond the scope of what our organization focuses on that I'm not gonna, gonna weigh in with a take on how we should approach it. We also often get questions on the liability piece even within our types of risks and just wanted to share our approach on that.

Okay. I have to say I've been thinking about this moment almost like a 911 moment, but for cyber security. The same way that after 911 we looked at all those physical spaces and thought about hardening them and changing them in ways to, you know, be resilient against potential terrorist attacks. I feel like we are looking at the web right now with the same type of vision and trying to think about, you know, what can we do.

And so, in terms of what the state can do, I'm not super well read on Cal Privacy's, cybersecurity risk assessment piece.

But I suspect that it probably really deals with consumer facing issues. And things like data breaches, for example. And I suspect that, as we look at what's been happening, behind the scenes at these companies, we're not dealing with consumer facing tools. We're dealing with R and D. And so, you know, we're trying to think about how to broaden companies' cybersecurity responsibilities with respect to R and D specifically and not just about kind of consumer facing products is probably a piece of the puzzle that we need to consider. I think that would be potentially consequential.

Again, I mentioned earlier I have a policy brief, looking at the consumer context for AI agents. And so, I mean, obviously, there's so many issues here. So, if we, you know, narrow it all and think about, just the consumer experience, for example, again, some of the things you mentioned in passing. You know, do we, how do we identify agents?

You know, what kind of a licensing regime might we want? What fiduciary duties in particular? What contexts? Medical, banking, for example. Those all seem like areas where the stakes are really high.

And so, companies should not be introducing AI agents without some clear legal responsibility if the agents get things wrong or you know, accidentally make a trade or empty your bank account.

You know, you want those stakes to be extremely clear. So many think those are areas that we, I mean, obviously it doesn't solve this larger safety issue that we're concerned about today. But I do think there are there's a lot of low hanging fruit we can think about in terms of how to really narrow in on some of these more kind of urgent daily issues. Thanks.

Senator Becker.

Let's start with Joshua. How would, you know, how do you understand the anything with the rate of progress in advancing these model capabilities compared to our progress in understanding them?

I think the progress and capabilities is definitely outstripping our ability to confidently predict how these models will behave or what types of advances they'll generate in other fields. So, I think at the moment, we have less understanding than we ought to. I think a new focus on that level of understanding would be extremely helpful.

Coming back again to what can California do, I think if we can find a substantial amount of compute to empower our universities with so that they can be actively researching AI safety topics and forecasting for AI impacts beyond just working directly on questions like interpretability or alignment, which are critical and have to be a part of any sound research agenda here, but also forecasting what will be the technical developments in other fields that will be highly impactful and consequential.

We can use compute to be studying these issues, and I would really want to empower our universities with compute to do that.

Yep. And requiring that safety research and helping our academics also do that. When, just question, and...

I don't know if you'd have an opinion on this too. The, my understanding is that Opus 5.5 actually regressed from Opus 5 in kind of refusal behavior for surveillance-related requests. Like this was, I have a document here about, that I was reading through.

And then the safety first came from kind of things ancillary to the model itself. So, I guess, you know, do you believe there's some risk inherent in these systems? Is that what you're probing, one of the things you're probing?

I'm sorry. Could you clarify the question?

Well, I mean, are you aware of this paper? So, it's saying that in Opus 5.5 actually regressed from Opus 5 in terms of refusing behavior for surveillance-related requests. So, people made requests to say surveil other people, you know, request for bad behavior and actually the model actually regressed in terms of, from a previous version, in terms of refusing those. That, that's kind of the system card. So, I don't know.

I'm just trying to...a point. You know, just, I guess trying to show how or get a perspective on how much the risk is we believe is inherent in these models themselves and, you know, how do we think about safety regards to, is it stuff we layer on top, you know, is it the, is it, is it the auditing, is the other piece or is it...

Is there things we can things that we can, you know, include or mandate or understand around the models themselves? Does that make sense?

Yes. Thank you very much, Senator. Really good question. The first thing to say is that there's no clear recipe. There's no if you do x then the model will always refuse to surveil people or will always reject these kinds of bad actions.

We don't have any kinds of recipes like that at the present. What we have, instead, is the ability to vary the training data, to vary the reinforcement learning that's done, to vary those procedures, and to observe the effects of that. And, you know, I'm not familiar with the specific detail you mentioned about Opus 5.5, but it's not surprising to me. As you build a new model, some things get better, some things will sometimes get a little worse. And that's kind of what's expected right now.

I think it would be great if we had a deeper understanding, a more fundamental understanding of what caused these behaviors. At the present time, we don't and it doesn't feel to me like we're very close to it either.

Okay. Thank you. That's helpful.

I can add on that surveillance question. I have two lines of research in this area right now. You know, one is actually trying to use these production models to build individual surveillance tools to see how successful we can be in doing it. And, you know, I have...

an undergraduate research assistant working with me who is very successfully able already to get one of the models to basically say, well, like, I shouldn't do that, but here's exactly how to do that. And she's already kind of built a proof of concept, for me just in the last few weeks. So just to say, yeah. It's a real ongoing concern. Of course, again, I have no transparency into why this is possible, but it's trivial.

The second piece again, like how do we know inherently why these models make the decisions they do? I have another line of research again looking at what we call dark patterns. This kind of online manipulation space in the consumer context where again I'm trying to understand kind of at an essential level how these models basically may manipulate users depending on what type of prompting I give them. And that research right now is ongoing.

But again, it appears that I can very easily manipulate the type of responses they give us based on kind of introducing these kinds of classic dark patterns that we already are aware from the kind of existing consumer context.

Yeah. Okay. Well, thank you. You know, as to the concern, but we know that that's at a very basic level, we need to be able to hold these models accountable for issues like you raised. May have one or two more about, turn it over to Senator, I'll turn it back.

Well, I, I, I very much appreciate that, you know, all of our witnesses today, elucidating many of these issues. A part of the, one of the points of the... Of today's hearing is to both spotlight the, the mind-boggling uncertainties, pace of change, all the... All, all of all of which has become, you know, reinforced in this hearing. And at the same time, to remind ourselves and to, and that and to, and to give the government the confidence to act.

Right? Those, these are very fine balance for us always to achieve, is to have the humility to understand we do not have every answer. And we definitely don't have 10,000 answers to, to legislate into law. While also not feeling as though we don't know enough to do anything. And if we, if we just wait till some researcher or somebody else tells us or a court decides, and we don't get to live in either of those more comfortable worlds.

And, and I, and I, and I think you've helped us to accomplish that today. I, I, I think we are, we are more knowledgeable. We are smarter about some of the issues and the trade-offs. We will not achieve perfect knowledge, but, you know, I, I vote on legislation here. I'll, I'll give a little secret about barbering and cosmetology.

I barely have any hair. I, I can't braid anyone's hair. I don't know. It's... To me, that's as mysterious as a large language model with neurosymbolic characteristics.

That's just as mysterious. We still do it. I still vote on it. And that is both our obligation, our responsibility, and our privilege to, to, to, to, to, to act in that space. And so the, the...

That's the point of the series. It's not to resolve every issue. It is to, to remind us what we don't know and then to proceed with confidence that we must take action for all the reasons that have been, have been described. And I hope in, and I'm committed to doing so in a thoughtful but urgent, comprehensive, but, but flexible and adaptive approach to accomplish that. I'll just reinforce, I, I don't think we need 10,000 pieces of legislation.

I'm just... I'm begging my colleagues. And to, and to, to understand how... Where the, the challenges and the opportunities really are here. I think it's, it's quite common in, in our society for folks to think about this issue as a... As an engineering or software programming one. And, and, and I... And, and so I hope we don't... We, we, we don't lead ourselves into thinking that our job here is by statute to program artificial intelligence in a particular way. Because as, as you've reinforced for us, all of you today, that, that's not how it works.

We cannot reliably 100% of the time. We're not giving it, we're not giving it all of its programming instructions. And therefore, it's more challenging for us to think about how to, how to achieve the public policy goals that we're, that we're trying to achieve. It doesn't mean that we can't, but it does mean that a simple one-sentence bill that says AI will do this exact thing all the time in this very... But only in elementary schools and only in rural communities is not going to be a sufficient answer.

But we need to... We do need to remain focused on some of these larger questions around privacy, around catastrophic risk, around many of the topics that we've, we've, we've addressed today, while also dealing with the, with the implications that we're feeling, we're feeling in society at the moment, and the anxiety and the worries that, that Californians feel on a, on a, on a widespread basis. So, so it's been very helpful, the testimonies today. I wanna excuse the panel because we're now going to invite public comment.

And I know we have some for, for today's hearing.

And so as the panel... Thank you again. Very much appreciate your expertise and, and you spending the time with us today. We're gonna ask members of the public who wish to provide comments to please come to the standup microphone that's near the sergeant's desk to do so.

Thank you, Chair and members, for hosting this hearing. Savi Christian, Director of California Policy for Encode. Proud cosponsor, along with Secure AI Project, of SB 53 last year. I won't belabor the point, but I'd just like to echo the comments of my colleague, Andrew Doris, from the Secure AI Project. The problem is incredibly urgent.

We've been given effectively a warning shot the past, you know, year with some of the incidents that we've seen, but our organization, many Californians, many humans, frankly, are worried that we're not going to get another warning shot. So we're very heartened to see that the Legislature is giving this issue the attention that it needs, and we look forward to engaging with you all in the coming months, hopefully. Thank you.

Alright. Thank you. And, and let me just say, Savi, I, I mentioned earlier how, how grateful I know I am, but that the, you know, that, that all of us are to Senator Wiener for his leadership on, on this issue and, when most of us didn't really have our hands around it, and maybe he didn't either, but at least he was trying. And his incredible work on both 1047 and 53 and, and elsewhere. But of course, he would be the first to wanna emphasize the...

Both of you having been... Having worked for him. But, but Encode and others, Secure AI Project and others... Other folks who've been central actors in, in helping to crystallize our California's focus and therefore the country's focus on, on catastrophic risk. Thank you.

Thank you, Chair and members. JP Hanna on behalf of the California Nurses Association. First, I wanna say thank you for holding such an important hearing on AI and potential threats that we are seeing in its rapid expansion. For nurses, the risks of AI are already here. They are seeing AI rapidly introduced into their hospitals without adequate oversight or safeguards to protect patients and the nurses who care for them.

As this committee looks into how California can best address the existential threats from AI technology, we urge this body to also consider how workers are being impacted today and what we can do about it now. Nurses are witnessing the rapid deployment of bossware in their workplaces, being falsely presented as tools to improve efficiency, but in reality, an instrument of surveillance that tightens control, intensifies work, and extracts value from workers at the expense of quality, safety, and dignity.

In healthcare, unsafe work conditions are how patients get hurt. Any A... AI tool that interferes with the nurse's clinical judgment and her ability to advocate and care for her patient has no place in our hospitals.

This year, California passed AB 1979, so AI cannot replace a nurse's clinical judgment. But today, a nurse who exercises that judgment and overrides an unsafe AI output can still be punished for it. That gap is something the Legislature must close. The Legislature should also pay close attention to those AI companies and how they are choosing to utilize these technologies that pose not only a potential danger to patients, but to our communities.

For example, Palantir, a company that conducts mass surveillance in the service of state violence, from ICE's deportation machine to the genocide in Gaza, now runs nurse staffing software in hospitals across the country that drives unsafe staffing levels and undermines nurses at the bedside.

Our patients' data and our taxpayer dollars must never run through a company that profits from mass surveillance, deportation, genocide, and war. CNA urges this committee to take these attacks on workers and patients seriously, and we stand ready to partner with you to pass and defend strong AI regulations to protect nurses, patients, and our communities. Thank you very much.

Thank you.

Good morning, Chair and members of the committee. I appreciate the thoughtfulness and care you bring to examining these complex issues. My name is Peter Leroe-Munoz for the Bay Area Council. The council represents researchers, startups, and global leading companies within the AI ecosystem. Artificial intelligence offers extraordinary potential for California.

We see groundbreaking benefits across critical fields, accelerating medical advancements, driving academic innovation, and strengthening climate resilience, among others. At the same time, we acknowledge the serious and well-founded challenges ahead.

As model capabilities expand rapidly and agentic systems gain autonomy, we face real concerns regarding cybersecurity, algorithmic discrimination, and maintaining meaningful human control. Addressing these risks responsibly is essential to preserving public trust. To ensure safe, continued progress, any future regulation should be anchored in several high-level principles.

Rules must be grounded in real-world evidence, empirical trajectory analysis of AI systems and capabilities, and sound science rather than static compute thresholds, which rapidly become outdated as efficiency improves. Oversight must keep pace with the technology itself through adaptable safety protocols that scale dynamically with actual system capabilities and deployment context.

Regulation should target bad actors with clear, enforceable consequences for high-risk misuse or catastrophic harms, ensuring safety without stalling beneficial innovation.

Safeguards must protect personal data and maintain user privacy without forcing developers into impossible choices between model utility and data rights. Finally, policy frameworks should promote open research, support a diverse AI development ecosystem, and enable broad, equitable access to AI's benefits while applying safeguards proportionate to the risks of particular models and capabilities.

Thank you for your leadership as we work together on evidence-based guardrails.

Hi. Good morning. I'm Dylan McLeod, member of the public, degree in computer science from UC Berkeley. And I just wanted to say that while the topic of, like, what regulations to pass, you know, how to enforce legislation is still extremely unclear, right now, I think there's a massive information bottleneck, and it's only growing because we don't actually know the exact specifics of the model which solved...

Which resolved the Navier-Stokes conjecture. A lot of the information about what's actually going on inside of AI companies is spread through rumors in the Valley.

And I think that while the exact mechanisms of how to enforce legislation on AI companies are still unclear, right now, we have to sort of make a priority of keeping our eyes open, of gathering as much information as possible, and getting either government actors or independent evaluators inside of AI companies to be able to tell what they're doing.

Even if from a state level we can't pass legislation, you know, to enforce on all of the AI industry, having that information public could be very useful, and that's all.

Thank you for joining us. Alright. That concludes our public comment. Any closing comments from Senator Becker?

No, I just wanna thank all the panelists. I thank you for convening this. Just at the outset, there can't be a more important conversation than this right now, and this was very, very, very illuminating. Very helpful. Thank you.

Mr. Chair. What he said. Thanks.

Alright. Well, thank you.

Thank you both for, for, for making the trip to the Capitol during the, during the, the, the recess, and for both of your longstanding leadership in this, in this space, both here in California and nationally. And thanks to the, to the committee staff for all of their work of both setting up the hearing, but also that, as Senator Umberg noted, the, the really excellent background paper.

I, I, I can say with certainty, there will be a comprehensive set of legislation introduced in the Senate at the beginning of the upcoming session. The urgency and the... In the urgent call for thoughtfulness to go with that urgency is, is been well heard.

And we're, we're seeing it our... Whether it's at the Rotary Club or PTA meetings, Californians are talking about this, and they are anxious. And they, they, they do expect action, and they do expect their state to help lead the way. So we will, we will, we will continue to investigate and to learn in, in preparation for that... For the beginning of the upcoming legislative session.

But we do anticipate, and I know may... Senator Becker is, is, is working hard on this as well. We do anticipate a substantial, a substantial package of legislation up in the, in the, in the coming legislation... Ledge... Legislative session for, for, for action.

So with that, thanks to everyone and this hearing is adjourned.